Building OpenWrt ON OpenWrt

The intention in the past hadn't been to make OpenWrt self hosting. However with only a very few additions it can be. This is still experimental and requires workarounds for bugs in perl, llvm, and OpenWrt itself. However it is eminently doable.

Why Build OpenWrt on OpenWrt?

1. Becoming self-hosting is an important milestone for any OS
2. It makes building OpenWrt more accessible for those with less resources.
3. Only building on a mono-culture (x86-based full blown Linux systems) lets a lot of bugs slide under the radar. Building on a different system can (and did) expose a lot of issues that would inevitably crop up later
4. For the educational value and bragging rights of knowing that your device's OS was built on your device.

This has so far been tested on armv7l and aarch64 devices, but will likely work on most if not all supported OpenWrt architectures. However the device's hardware does need to meet some minimum requirements:

• 1GHz or faster CPU, multi-core is an asset.
• 512MiB of physical RAM per core you intend to use in the build for 32-bit devices. For 64-bit devices you need double this.¹
• 2GiB of swap space. This should be on a physical hard drive, or a high quality external device (meaning not the device's internal eMMC).²
• 1GiB of storage space on your root device for all the OpenWrt packages you will need.
• ~26GiB of storage space for the build

Note¹: The initial toolchain build (especially of llvm-bpf) is quite RAM-hungry. The given memory requirements should be considered a minimum if you are building your own llvm-bpf. If you are able to use a pre-built llvm-bpf, then you can get away with less RAM-per-used-core

Note²: Your swap space (and to a lesser extent, the storage space for the build itself) should be on a physical hard drive, or on a high quality solid state (USB, SSD) storage device. Certain steps in the process can be somewhat heavy on the swap. With a large, A good quality storage device will handle the wear better. Also, the larger your storage device is the more the wear will be spread.

First of all you need a whole lot of OpenWrt's own packages:

• The development tools themselves:
  opkg install pkg-config make gcc diffutils autoconf automake check git git-http patch libtool-bin
• Miscelaneous tools used in the build system:
  opkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip bzip2 flock wget-ssl
• Perl³ and Python:
  opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b python3
• A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there:
  opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq
• A few libraries:
  opkg install libncurses-dev zlib-dev musl-fts libzstd
ln -s libncursesw.a /usr/lib/libncurses.a
• Some good-to-have packages to make life easier (instructions below will assume you have these):
  opkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr xxd

Note³: The test systems this was performed on had quite a bit of perl pre-installed, so there may be more perl packages required.

Before we can go further, we have our first bug workaround:

OpenWrt Bug: OpenWrt doesn't put execute permissions on the scripts for the automake, autoconf, and libtool packages. This prevents a lot of build systems (including OpenWrt's own) from working correctly. To fix:
chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL
chmod +x /usr/share/autoconf/Autom4te/*
chmod +x /usr/share/libtool/build-aux/*

OpenWrt is almost but not quire fully self hosting. We do need a few dev libraries and one tool that OpenWrt doesn't natively support, and we can do most of that now. Three of them are just to support GNU LIBC extensions that musl doesn't have. Two of them are included in OpenWrt, just as a non-dev libraries. However, by default OpenWrt strips them (using sstrip) in such a way as you can't actually compile against them. Switch to your ~/devel folder (or wherever you're going to use as yoour base build directory) and do the following:

• libargp:
  

  git clone https://github.com/xhebox/libuargp.git
  cd libuargp
  make
  make prefix=/usr install
  ln -s libargp.so /usr/lib/libargp.so.0
  cd ..

• libfts:
  We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one:
  

  git clone https://github.com/void-linux/musl-fts.git
  cd musl-fts
  ./bootstrap.sh
  ./configure --prefix=/usr
  make
  make install
  cd ..

• libobstack:
  

  git clone https://github.com/void-linux/musl-obstack.git
  cd musl-obstack
  ./bootstrap.sh
  ./configure --prefix=/usr
  make
  make install
  cd ..

• libzstd:
  Like musl-fts above, OpenWrt does support this library, but by default ships a lobotomized version that can't be built against. We install the OpenWrt version above so it won't be later installed and overwrite the proper version we're making below:
  

  wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
  tar -xvzf zstd-1.5.2.tar.gz
  cd zstd-1.5.2/lib
  make
  make prefix=/usr install
  cd ..

The single tool needed in the build system that OpenWrt doesn't support is rev, a little-known and littler-used Unix tool. It's part of util-linux, which does provide a lot of packages for OpenWrt, but rev isn't one of them. It is, however, built as part of the process below, so it'll be be installed later.

This is based on the build system instructions, with a bunch of deviations to fix various bugs along the way:

Make sure you're in your devel directory from before. The following assume you are building SNAPSHOT, but can be (with appropriate changes) be used for release versions as well.

• Get OpenWrt master and associated packages:
  

  git clone https://git.openwrt.org/openwrt/openwrt.git
  cd openwrt
  git checkout master
  git pull
  ./scripts/feeds update -a
  ./scripts/feeds install -a

• Get the config file for your architecture and save it as .config. For example, for a device based on the mt7622 chipset (such as the BPI-R64) it is this:
  

  wget https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/config.buildinfo -O .config

• Configure your build:
  

  make defconfig
  make menuconfig

  Things to change in the config: set it up to build for only your platform (since by default it will want to build for all platforms using a given chipset). Also read the next step below about LLVM and configure whether or not you're going to use a pre-built one. Finally make any other tweaks you normally do, then save.

• EITHER get a pre-made LLVM (preferable)...
  Compiling LLVM is the single most time and resouce-intensive step in the whole process. It can easily take a full day by itself. It is recommended to check the resources below to see if there is a premade LLVM for your platform that is the right version. If there is one, you can get it and untar it into the OpenWrt working directory. If not, you will have to...
• ...OR Build LLVM
  BE CAREFUL HERE! This is memory intensive, and a multi-core build on a system with marginal memory can push the system so far into memory-debt swapping that it can damage solid-state storage. A single-core build will take a long time, but is safer. This is going to take a long time and a drawback of the OpenWrt build system is that any time the build of any one package is interrupted, it starts that package from the beginning the nest time. You don't want compilation to stop on a random net blip, so use nohup. And start getting into the habit of using V=sc and logging everything, because this whole process is still experimental:
  

  nohup nice -10 make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &

  Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us.