Differences

This shows you the differences between two versions of the page.

--- docs:guide-developer:toolchain:building_openwrt_on_openwrt [2023/02/09 19:24] – Format bug fix va1der
+++ docs:guide-developer:toolchain:building_openwrt_on_openwrt [2024/11/26 19:21] (current) – small language tweaks after attemping on my r4s palebloodsky
@@ Line 1: / Line 1: @@
 ====== Building OpenWrt ON OpenWrt ======
-<WRAP important>Under Construction - until this tag is removed, it is better to follow [[https://forum.openwrt.org/t/building-openwrt-on-openwrt-a-howto/146213|the instructions on the forum]].</WRAP>
+While the intention in the past hasn't been to make OpenWrt self hosting, with a few additions it can be.  This is still experimental and requires workarounds for bugs in perl, llvm, and OpenWrt itself.  This is a work in progress, but building OpenWrt //inside of// OpenWrt is possible.
-While the intention in the past hadn't been to make OpenWrt self hosting, with only a very few additions it can be.  This is still experimental and requires workarounds for bugs in perl, llvm, and OpenWrt itself.  The following is experimental, and a work in progress.  But building OpenWrt inside of OpenWrt is possible.
 ===== Purpose =====
- Why Build OpenWrt on OpenWrt?
+Why Build OpenWrt on OpenWrt?
-  - Becoming self-hosting is an important milestone for any OS
+  - Becoming self-hosting is an important milestone for any OS.
   - It makes building OpenWrt more accessible for those with less resources.
-  - Only building on a mono-culture (x86-based full blown Linux systems) lets a lot of bugs slide under the radar. Building on a different system can (and did) expose a lot of issues that would inevitably crop up later
+  - Only building on a monoculture (x86-based full blown Linux systems) lets a lot of bugs slip under the radar. Building on a different system can (and did) expose issues that would inevitably crop up later.
   - For the educational value and bragging rights of knowing that your device's OS was built on your device.
@@ Line 16: / Line 14: @@
 ===== Device Requirements =====
-This has so far been tested on armv7l and aarch64 devices, but will likely work on most if not all supported OpenWrt architectures.  However the device's hardware does need to meet some minimum requirements:
+So far this has been tested on armv7l and aarch64 devices, but will likely work on most if not all OpenWrt supported architectures.  However the device's hardware does need to meet some minimum requirements:
   * 1GHz or faster CPU, multi-core is an asset.
   * 512MiB of physical RAM per core you intend to use in the build for 32-bit devices.  For 64-bit devices you need double this.<sup>1</sup>
-  * 2GiB of swap space.  This should be on a physical hard drive, or a high quality external device (meaning not the device's internal eMMC).<sup>2</sup>
+  * 2GiB of swap space.  This should be on a physical hard drive, or high quality external device (meaning not the device's internal eMMC).<sup>2</sup>
-  * 1GiB of storage space on your root device for all the OpenWrt packages you will need.
+  * 1GiB of storage space on your root device for all the packages you will need.
-  * ~26GiB of storage space for the build
+  * ~26GiB of storage space for the build.
-Note<sup>1</sup>: The initial toolchain build (especially of llvm-bpf) is quite RAM-hungry.  The given memory requirements should be considered a minimum if you are building your own llvm-bpf.  If you are able to use a pre-built llvm-bpf, then you can get away with less RAM-per-used-core
+Note<sup>1</sup>: The initial toolchain build (especially of llvm-bpf) is quite RAM-hungry.  The given memory requirements should be considered a minimum if you are building your own llvm-bpf.  If you are able to use a pre-built llvm-bpf, then you can get away with less RAM-per-core.
-Note<sup>2</sup>: Your swap space (and to a lesser extent, the storage space for the build itself) should be on a physical hard drive, or on a high quality solid state (USB, SSD) storage device. Certain steps in the process can be somewhat heavy on the swap. With a large, A good quality storage device will handle the wear better. Also, the larger your storage device is the more the wear will be spread.
+Note<sup>2</sup>: Your swap space (and to a lesser extent, the storage space for the build itself) should be on a hard drive or high quality solid state storage device. It should not under any circumstances be on the device's internal eMMC storage. This is to avoid NAND flash wear as this process is write heavy especially with swapping. Also, the larger your storage device is the more the wear will be spread.
 ===== Setting Up The Development Environment =====
@@ Line 31: / Line 29: @@
 ==== OpenWrt Packages ====
-First of all you need a whole lot of OpenWrt's own packages:
+First of all you need a lot of OpenWrt packages:
   * The development tools themselves: \\ ''opkg install pkg-config make gcc diffutils autoconf automake check git git-http patch libtool-bin''
   * Miscelaneous tools used in the build system: \\ ''opkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip bzip2 flock wget-ssl''
-  * Perl<sup>3</sup> and Python: \\ ''opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b python3''
+  * Perl<sup>3</sup> and Python: \\ ''opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b perlbase-ipc perlbase-module perlbase-extutils python3''
-  * A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there: \\ ''opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq''
+  * A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there: \\ ''opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq coreutils-join''
   * A few libraries: \\ ''opkg install libncurses-dev zlib-dev musl-fts libzstd'' \\ ''ln -s libncursesw.a /usr/lib/libncurses.a''
   * Some good-to-have packages to make life easier (instructions below will assume you have these): \\ ''opkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr xxd''
@@ Line 41: / Line 39: @@
 Note<sup>3</sup>: The test systems this was performed on had quite a bit of perl pre-installed, so there may be more perl packages required.
-Before we can go further, we have our first bug workaround:
+Before we can go further, we have our first workaround:
 **OpenWrt Bug**: OpenWrt doesn't put execute permissions on the scripts for the ''automake'', ''autoconf'', and ''libtool'' packages. This prevents a lot of build systems (including OpenWrt's own) from working correctly. To fix: \\ ''chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL'' \\ ''chmod +x /usr/share/autoconf/Autom4te/*'' \\ ''chmod +x /usr/share/libtool/build-aux/*''
-==== Non-OpenWrt Libraries and Tools ====
+==== Libraries and Tools ====
-OpenWrt is almost but not quire fully self hosting.  We do need a few dev libraries and one tool that OpenWrt doesn't natively support, and we can do most of that now. Three of them are just to support GNU LIBC extensions that musl doesn't have.  Two of them //are// included in OpenWrt, just as a non-dev libraries.  However, by default OpenWrt strips them (using ''sstrip'') in such a way as you can't actually compile against them.  Switch to your ~/devel folder (or wherever you're going to use as yoour base build directory) and do the following:
+A few dev libraries are needed. Two of them //are// are included in OpenWrt, but by default OpenWrt strips them (using ''sstrip'') so that you can't actually compile against them.  Switch to your ~/devel folder (or wherever you're going to use as your base build directory) and do the following:
 === Libraries ===
-  * [[https://github.com/xhebox/libuargp|libargp]]: \\ <code bash>
+  * [[https://github.com/xhebox/libuargp|libargp]]: A GNU GLIBC extension used by elflibs\\ <code bash>
 git clone https://github.com/xhebox/libuargp.git
 cd libuargp
@@ Line 59: / Line 57: @@
 cd ..
 </code>
-  * [[https://github.com/void-linux/musl-fts|libfts]]: \\ We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one: \\ <code bash>
+  * [[https://github.com/void-linux/musl-fts|libfts]]: Another GNU GLIBC extension needed for elflib. \\ We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one: \\ <code bash>
 git clone https://github.com/void-linux/musl-fts.git
 cd musl-fts
@@ Line 67: / Line 65: @@
 make install
 cd ..</code>
-  * [[https://github.com/void-linux/musl-obstack|libobstack]]: \\ <code bash>
+  * [[https://github.com/void-linux/musl-obstack|libobstack]]: The final GNU GLIBC extension needed for elflib \\ <code bash>
 git clone https://github.com/void-linux/musl-obstack.git
 cd musl-obstack
@@ Line 75: / Line 73: @@
 make install
 cd ..</code>
-  * [[https://github.com/facebook/zstd/|libzstd]]: \\ Like musl-fts above, OpenWrt does support this library, but by default ships a lobotomized version that can't be built against.  We install the OpenWrt version above so it won't be later installed and overwrite the proper version we're making below: \\ <code bash>
+  * [[https://github.com/facebook/zstd/|libzstd]]: A compression library now needed by llvm-bpf. \\ Like musl-fts above, OpenWrt does support this library, but by default ships a lobotomized version that can't be built against.  We install the OpenWrt version above so it won't be later installed and overwrite the proper version we're making below: \\ <code bash>
 wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
 tar -xvzf zstd-1.5.2.tar.gz
@@ Line 82: / Line 80: @@
 make prefix=/usr install
 cd ..</code>
+  * libdl, librt, libresolv, and libpthread: \\ These are libraries for functions which are actually included in musl libc directly.  However, since some tools try and link explicitly against these libs, you will run into "file not found" errors.  To get around that, just make stub libraries for them: \\ <code bash>
+cat <<EOF > /usr/lib/libdl.a
+!<arch>
+EOF
+cp -a /usr/lib/libdl.a /usr/lib/librt.a
+cp -a /usr/lib/libdl.a /usr/lib/libpthread.a
+cp -a /usr/lib/libdl.a /usr/lib/libresolv.a
+</code>
 === Tools ===
-The single tool needed in the build system that OpenWrt doesn't support is ''rev'', a little-known and littler-used Unix tool.  It's part of  util-linux, which does provide a lot of packages for OpenWrt, but rev isn't one of them.  It is, however, built as part of the process below, so it'll be be installed later.
+The single tool needed in the build system that OpenWrt doesn't support is ''rev'', a little-known and littler-used Unix tool.  It's part of util-linux, which does provide a lot of packages for OpenWrt, but rev isn't one of them.  It is, however, built as part of the process below, so it'll be be installed later.
 ===== Getting and Building OpenWrt =====
@@ Line 112: / Line 118: @@
   - **If you are using pre-made llvm from step 4, then skip all of this step** (read the NOTE at the end though). \\ If you were unable or didn't want a pre-built llvm-bpf, then now is when you will build it.  But first... \\ **llvm-bpf bugs:**  There are two bugs in llvm's build system.  One that [[https://discourse.llvm.org/t/build-of-14-06-fails-on-armv7-due-to-libatomic-not-being-linked/67091|causes a built failure due to libatomic not being linked]], and one that [[https://discourse.llvm.org/t/build-of-15-0-7-fails-due-to-libstdc-not-being-linked-against/68132|causes a build failure when gcc is used instead of g++ at the linking stage]]. We will copy patches for both into the OpenWrt build system: \\ <code bash>
 mkdir tools/llvm-bpf/patches
-wget https://excelcia.ca/~public/openwrt/patches/llvm-bpf/901-fix-fuzzer-linking.patch -O tools/llvm-bpf/patches/901-fix-fuzzer-linking.patch
+wget https://va1der.ca/~public/openwrt/patches/llvm-bpf/901-fix-fuzzer-linking.patch -O tools/llvm-bpf/patches/901-fix-fuzzer-linking.patch
-wget https://excelcia.ca/~public/openwrt/patches/llvm-bpf/902-use-libatomic-as-required.patch -O tools/llvm-bpf/patches/902-use-libatomic-as-required.patch</code> Once those are copied, you can build llvm. **Caution!** This is memory intensive, and a multi-core build on a system with marginal memory can push the system so far into memory-debt swapping that it can damage solid-state storage. A single-core build will take longer, but is safer. This is going to take a long time. Note that a drawback of OpenWrt's build system is that any time the build of any one package is interrupted, it starts that package from the beginning the next time. You don't want compilation to stop on a random net blip after ten hours, so use ''nohup''. And start getting into the habit of using ''V=sc'' and logging everything, because this whole process is still experimental: \\ <code bash>
+wget https://va1der.ca/~public/openwrt/patches/llvm-bpf/902-use-libatomic-as-required.patch -O tools/llvm-bpf/patches/902-use-libatomic-as-required.patch</code> After musl-1.2.4 there were changes, such as lseek64 missing, and replaced with lseek, so llvm fails to build. To fix this, you need to make a minor change to llvm-bpf's Makefile, just under include $(INCLUDE_DIR)/cmake.mk add following "HOST_CFLAGS+= -D_LARGEFILE64_SOURCE", so it will look a bit like this: <code>
+...
+CMAKE_SOURCE_SUBDIR := llvm
+include $(INCLUDE_DIR)/host-build.mk
+include $(INCLUDE_DIR)/cmake.mk
+HOST_CFLAGS+= -D_LARGEFILE64_SOURCE
+LLVM_BPF_PREFIX = llvm-bpf-$(PKG_VERSION).$(HOST_OS)-$(HOST_ARCH)
+...
+</code> Once those patches are copied and Makefile is fixed, you can build llvm. **Caution!** This is memory intensive, and a multi-core build on a system with marginal memory can push the system so far into memory-debt swapping that it can damage solid-state storage. A single-core build will take longer, but is safer. This is going to take a long time. Note that a drawback of OpenWrt's build system is that any time the build of any one package is interrupted, it starts that package from the beginning the next time. You don't want compilation to stop on a random net blip after ten hours, so use ''nohup''. And start getting into the habit of using ''V=sc'' and logging everything, because this whole process is still experimental: \\ <code bash>
 nohup nice make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &</code> Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us. \\ \\ **NOTE:** Once llvm-bpf is out of the way, **if** you have multiple cores **and** enough RAM per core, then you should be ok to use multi-core compiling. Always keep a periodic eye on your build, though and htop is your friend. For all examples throughout these instructions, ''-j 1'' will be supplied to make for pasted commands for safety.  But feel free to increase the core count at your discretion.
   - Now build the rest of the toolchain:  \\ <code bash>
@@ Line 122: / Line 139: @@
 cp $(find . -wholename \*ipkg-install/usr/bin/rev) /usr/bin/rev</code>
   - **perl bugs**: Here we fix three bugs in perl. One which [[https://github.com/openwrt/openwrt/issues/11591|causes a problem with perl on musl]], one where recent compilers cause a seg fault, and [[https://github.com/Perl/perl5/issues/20606|one in Perl's Configure script]]. Fort these we need two patches. The first patch we will apply directly to OpenWrt's perl makefile: \\ <code bash>
-( cd feeds/packages/lang/perl && wget https://excelcia.ca/~public/openwrt/patches/perl/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )</code> That fixes the first two bugs. This next fixes perl's configure bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it:<code bash>
+( cd feeds/packages/lang/perl && wget https://va1der.ca/~public/openwrt/patches/perl/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )</code> That fixes the first two bugs. This next fixes perl's configure bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it:<code bash>
-wget https://excelcia.ca/~public/openwrt/patches/perl/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch</code>
+wget https://va1der.ca/~public/openwrt/patches/perl/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch</code>
   - **DECISION POINT**: perl bug - Test to see if your the OpenWrt device you are building on is affected by one of the above bugs. Some are, and if it is, you need to build perl right now with the above patch to fix your existing system.  This is because the host system's perl is used when OpenWrt builds openssl. Download and run the script: \\ <code bash>
-wget https://excelcia.ca/~public/openwrt/patches/perl/test_index.pl -O ~/test_index.pl
+wget https://va1der.ca/~public/openwrt/patches/perl/test_index.perl -O ~/test_index.pl
 chmod +x ~/test_index.pl
 ~/test_index.pl</code> This will tell you if your perl is affected by the index() bug. If it is then build perl with the above bug patch above and copy libperl.so over top of of your system's buggy one: \\ <code bash>
@@ Line 133: / Line 150: @@
   - That was the last bug fix - you can now proceed with the rest of the build.  This is likely to take a while (not as long as LLVM though). Continue to use ''nohup'', tell make to give verbose output with ''V=sc'', and continue to redirect that output to a log: \\ <code bash>
 nohup nice make -j 1 V=sc world > world01.out &</code>
+===== Resources =====
+==== llvm-bpf ====
+As noted above, it's beneficial to use a prebuilt llvm if possible.  The following are available:
+=== armv7l (ARM 32 little endian) ===
+  * [[https://va1der.ca/~public/openwrt/llvm/llvm-bpf-14.0.6.Linux-armv7l.tar.xz|llvm-bpf-14.0.6.Linux-armv7l.tar.xz]]
+  * [[https://va1der.ca/~public/openwrt/llvm/llvm-bpf-15.0.7.Linux-armv7l.tar.xz|llvm-bpf-15.0.7.Linux-armv7l.tar.xz]]
+=== aarch64 (ARM 64) ===
+  * [[https://va1der.ca/~public/openwrt/llvm/llvm-bpf-14.0.6.Linux-aarch64.tar.xz|llvm-bpf-14.0.6.Linux-aarch64.tar.xz]]
+  * [[https://va1der.ca/~public/openwrt/llvm/llvm-bpf-15.0.7.Linux-aarch64.tar.xz|llvm-bpf-15.0.7.Linux-aarch64.tar.xz]]
+=== AMD64 (aka x86_64) ===
+  * [[https://downloads.openwrt.org/releases/22.03.3/targets/mediatek/mt7622/llvm-bpf-13.0.0.Linux-x86_64.tar.xz|llvm-bpf-13.0.0.Linux-x86_64.tar.xz]]
+  * [[https://downloads.openwrt.org/snapshots/targets/mediatek/filogic/llvm-bpf-15.0.7.Linux-x86_64.tar.xz|llvm-bpf-15.0.7.Linux-x86_64.tar.xz]]
 ===== On Errors =====
-The above procedure has been tested, but this is still very much an experimental process and Your Mileage May Vary™. On the first successful attempt output was up to log file world13.out on two different platforms.  The majority of errors were simple ones, caused by missing tools or inadequate ones where the basic busybox version wasn't good enough.  There still may be missing packages in the above procedure, since it was tested on existing systems with preset installation packages that you may or may not have.  There may be other subtle issues on other architectures. When htop tells you compilation has stopped, look at the log. Jump to the end, then (if you were doing multi-core builds) search backwards for "Error" (often with capital E) to find what exactly went wrong. Most of the time it's a missing tool. Sometimes the issue has been subtle problems with multi-core building.  The timing of and staging of different parts of a build when doing a multi-core build changes on different architectures, and deficiencies in dependency settings may not show up in normal x86 builds that do on other archtectures.  If you, for example, run into issues of symbols not being found on linking, sometimes simply redoing the build, or building that package in a single-thread build resolves the issue.  Feel free to post your results [[https://forum.openwrt.org/t/building-openwrt-on-openwrt-a-howto/146213|in the forum thread for this]].
+The above procedure has been tested, but this is still very much an experimental process and Your Mileage May Vary™. On the first successful attempt output was up to log file world13.out on two different platforms.  The majority of errors were simple ones, caused by missing tools or inadequate ones where the basic busybox version wasn't good enough.  There still may be missing packages in the above procedure, since it was tested on existing systems with preset installation packages that you may or may not have.  There may be other subtle issues on other architectures. When htop tells you compilation has stopped, look at the log. Jump to the end, then (if you were doing multi-core builds) search backwards for "Error" (often with capital E) to find what exactly went wrong. Most of the time it's a missing tool. Sometimes the issue has been subtle problems with multi-core building.  The timing of and staging of different parts of a build when doing a multi-core build changes on different architectures, and deficiencies in dependency settings may not show up in normal x86 builds that do on other archtectures.  If you, for example, run into issues of symbols not being found on linking, sometimes simply redoing the build, or building that package in a single-thread build resolves the issue.  Feel free to post your results in the [[https://forum.openwrt.org/t/building-openwrt-on-openwrt-a-howto/146213|forum thread for this]].
-===== SUMMARY =====
+===== Summary =====
 There were quite a number of bugs that were exposed while developing this procedure. Bugs in OpenWrt, Perl, LLVM, and other places. However, one shouldn't consider the presence of bugs the deciding factor on whether a system is, in general, self-hosting. OpenWrt needs one small utility (rev), and three bandaid libs to cover for GNU extensions to GLIBC. That makes OpenWrt about 99.9% self-hosting ready. This is a great achievement, especially for an embedded OS where self-hosting wasn't even the goal, and more especially considering the enormous amount of complex code and a build system that uses so many different build tools. The OpenWrt devs are to be greatly commended for an excellent embedded OS!