Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
| docs:guide-user:additional-software:extroot_configuration [2022/11/07 04:11] – [1. Preparation] remove the mkfs as its done later crass | docs:guide-user:additional-software:extroot_configuration [2024/03/07 09:46] – kmod-usb-storage often is required as well. zpe | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Extroot configuration ====== | ====== Extroot configuration ====== | ||
| - | {{section> | + | {{section> |
| ===== Introduction ===== | ===== Introduction ===== | ||
| This guide describes how to configure OpenWrt to use a storage device (USB or SATA or SD card or whatever) to expand your root filesystem, to install freely all the packages you need. | This guide describes how to configure OpenWrt to use a storage device (USB or SATA or SD card or whatever) to expand your root filesystem, to install freely all the packages you need. | ||
| - | In most supported devices OpenWrt splits the internal storage into '' | + | In most supported devices OpenWrt splits the internal storage into '' |
| ^ Partition ^ Mount point ^ Compression ^ Writable | | ^ Partition ^ Mount point ^ Compression ^ Writable | | ||
| | '' | | '' | ||
| - | | '' | + | | '' |
| | '' | | '' | ||
| - | This way OpenWrt fits even in tiny amounts of internal storage (as low as 4 MiB), but still allows to write settings and install some packages in the writable partition without changing all linux programs used. | + | This way OpenWrt fits even in tiny amounts of internal storage (as low as 4 MiB), but still allows to write settings and install some packages in the writable partition without changing all Linux programs used. |
| Extroot works by setting another overlay partition in the external storage device, and during boot this new overlay partition will be mounted over the internal storage' | Extroot works by setting another overlay partition in the external storage device, and during boot this new overlay partition will be mounted over the internal storage' | ||
| - | This approach allows easy fallback in case the external storage device is removed, as your device will still have its own overlay partition and thus will load all configuration from there. | + | This approach |
| Which means that it will behave exactly the same as just before you set up extroot. | Which means that it will behave exactly the same as just before you set up extroot. | ||
| - | **Note** | + | Note that OpenWrt is known to [[flyspray> |
| + | You can work around the issue by using ''/'' | ||
| - | This configuration will not be able to be used on devices that do not have the ''/ | ||
| ===== Instructions ===== | ===== Instructions ===== | ||
| - | The following instructions assume that you already have access to a shell on your OpenWRT device. Most if not all of these commands can be done via the web interface, however that is emphatically not recommended. Usually the shell is accessed via [[: | + | The following instructions assume that you already have access to a shell on your OpenWRT device. |
| + | Most if not all of these commands can be done via the web interface, however that is emphatically not recommended. | ||
| + | Usually the shell is accessed via [[: | ||
| ==== 1. Preparation ==== | ==== 1. Preparation ==== | ||
| Devices with 8 MiB flash or more should have enough space to install the required packages, otherwise create a [[docs: | Devices with 8 MiB flash or more should have enough space to install the required packages, otherwise create a [[docs: | ||
| - | Remove all packages you have installed to add secondary functionality, | + | Remove all packages you have installed to add secondary functionality, |
| + | (If you do not have a record of what these are, try removing ' | ||
| + | Leave only those needed to access the internet and needed to access the extroot filesystem. | ||
| After you make the extroot you will have all the space you need to install secondary packages. | After you make the extroot you will have all the space you need to install secondary packages. | ||
| - | The extroot can be anything that '' | + | You may not need to make a custom image: try the OEM image first (OpenWRT GL.inet for a GL.inet mango). |
| - | The following assumes that you will be creating your extroot as an EXT4 filesystem on your OpenWRT device with a connected USB flash drive. The process is similar for other kinds of devices. | + | |
| - | This will install the required packages and create the extroot | + | The extroot |
| - | < | + | Currently '' |
| + | It must a [[commit>? | ||
| + | For most, this filesystem | ||
| + | However, it could also be on an SD-Card or a SATA drive connected via e-sata or even a network block device (assuming its set up early enough). | ||
| + | If you're using a USB connected device follow | ||
| + | |||
| + | The following assumes that you will be creating your extroot | ||
| + | The process is similar for other kinds of devices. | ||
| + | |||
| + | Installing these packages requires a sensible amount of extra filespace. If you completely fill the filesystem by installing | ||
| + | you will probably have to re-flash the entire system. So if you think you may already be close to filling the filesystem, | ||
| + | remove some installed packages first. Good candidates for removal are ntfs3 and ntfs3-utils: | ||
| + | you have extroot installed. | ||
| + | |||
| + | Install the required packages. | ||
| + | |||
| + | < | ||
| opkg update | opkg update | ||
| - | opkg install block-mount kmod-fs-ext4 e2fsprogs parted | + | opkg install block-mount kmod-fs-ext4 e2fsprogs parted |
| - | parted | + | |
| </ | </ | ||
| - | ==== 2. Configuring rootfs_data ==== | + | Identify |
| - | Configure ''/ | + | |
| <code bash> | <code bash> | ||
| - | DEVICE=" | + | ls -l /sys/block |
| - | uci -q delete fstab.rwm | + | |
| - | uci set fstab.rwm=" | + | |
| - | uci set fstab.rwm.device=" | + | |
| - | uci set fstab.rwm.target=" | + | |
| - | uci commit fstab | + | |
| </ | </ | ||
| - | Or, you can identify | + | ==== 2. Partitioning and formatting ==== |
| + | Partition and format | ||
| <code bash> | <code bash> | ||
| - | grep -e rootfs_data | + | DISK="/dev/sda" |
| + | parted -s ${DISK} -- mklabel gpt mkpart extroot 2048s -2048s | ||
| + | DEVICE=" | ||
| + | mkfs.ext4 -L extroot ${DEVICE} | ||
| </ | </ | ||
| - | If your '' | + | This will wipe all data on the disk, so do not run these commands blindly. |
| - | + | ||
| - | The ''/ | + | |
| ==== 3. Configuring extroot ==== | ==== 3. Configuring extroot ==== | ||
| - | See what partitions you have using the following command: | + | Configure |
| <code bash> | <code bash> | ||
| - | block info | + | eval $(block info ${DEVICE} | grep -o -e ' |
| + | eval $(block info | grep -o -e ' | ||
| + | uci -q delete fstab.extroot | ||
| + | uci set fstab.extroot=" | ||
| + | uci set fstab.extroot.uuid=" | ||
| + | uci set fstab.extroot.target=" | ||
| + | uci commit fstab | ||
| </ | </ | ||
| - | You will see similar output: | + | ==== 4. Transferring data ==== |
| + | Transfer the content of the current overlay to the external drive. | ||
| <code bash> | <code bash> | ||
| - | /dev/ | + | mount ${DEVICE} |
| - | / | + | tar -C ${MOUNT} |
| - | /dev/sda1: UUID=" | + | |
| </ | </ | ||
| - | Here '' | + | ==== 5. Configuring rootfs_data |
| + | Configure a mount entry for the the original overlay. | ||
| <code bash> | <code bash> | ||
| - | DEVICE="/ | + | DEVICE=" |
| - | mkfs.ext4 ${DEVICE} | + | uci -q delete fstab.rwm |
| + | uci set fstab.rwm=" | ||
| + | uci set fstab.rwm.device=" | ||
| + | uci set fstab.rwm.target="/ | ||
| + | uci commit fstab | ||
| </ | </ | ||
| - | Now we configure | + | This will allow you to access |
| - | <code bash> | + | ==== 6. Apply changes |
| - | eval $(block info ${DEVICE} | grep -o -e " | + | Reboot |
| - | uci -q delete fstab.overlay | + | |
| - | uci set fstab.overlay=" | + | |
| - | uci set fstab.overlay.uuid=" | + | |
| - | uci set fstab.overlay.target="/ | + | |
| - | uci commit fstab | + | |
| - | </ | + | |
| - | + | ||
| - | ==== 4. Transferring data ==== | + | |
| - | We now transfer the content of the current overlay to the external drive and reboot | + | |
| <code bash> | <code bash> | ||
| - | mkdir -p /tmp/cproot | ||
| - | mount --bind /overlay /tmp/cproot | ||
| - | mount ${DEVICE} /mnt | ||
| - | tar -C /tmp/cproot -cvf - . | tar -C /mnt -xf - | ||
| - | umount /tmp/cproot /mnt | ||
| reboot | reboot | ||
| </ | </ | ||
| Line 140: | Line 152: | ||
| * Do not use vfat (FAT/ | * Do not use vfat (FAT/ | ||
| - | * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase '' | + | * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase '' |
| <code bash> | <code bash> | ||
| Line 189: | Line 201: | ||
| <code bash> | <code bash> | ||
| # Create swap file | # Create swap file | ||
| - | dd if=/ | + | DIR=" |
| - | mkswap | + | dd if=/ |
| + | mkswap | ||
| # Enable swap file | # Enable swap file | ||
| uci -q delete fstab.swap | uci -q delete fstab.swap | ||
| uci set fstab.swap=" | uci set fstab.swap=" | ||
| - | uci set fstab.swap.device=" | + | uci set fstab.swap.device=" |
| uci commit fstab | uci commit fstab | ||
| - | / | + | service |
| # Verify swap status | # Verify swap status | ||
| Line 266: | Line 279: | ||
| ==== Remote file system ==== | ==== Remote file system ==== | ||
| Follow: [[https:// | Follow: [[https:// | ||
| + | |||
| + | ==== LUKS encrypted extroot ==== | ||
| + | You may wish to have your extroot filesystem in a LUKS encrypted container. | ||
| + | As of OpenWrt 22.03.2, this [[https:// | ||
| + | OpenWrt does not have an official way to open encrypted LUKS volumes before the extroot check happens during the normal boot path. | ||
| + | So at the time of extroot check time, the extroot filesystem will not be visible and the boot process will continue as if there is not extroot. | ||
| + | Below are two different methods for getting the system to run on an encrypted extroot. | ||
| + | The first method is preferable because there are less side-effects and is a cleaner approach. | ||
| + | |||
| + | Before doing any of the below, you'll need to create the LUKS container in which to put your extroot filesystem. | ||
| + | Follow the [[docs: | ||
| + | You will need enough space on your '' | ||
| + | Once, you have your LUKS container follow the [[# | ||
| + | |||
| + | === PREINIT === | ||
| + | In the PREINIT phase of boot, '' | ||
| + | Using a stock OpenWRT firmware, there will be no extroot config on the ROM device. | ||
| + | To check the '' | ||
| + | If found, it will search in order ''/ | ||
| + | This is why the '' | ||
| + | |||
| + | Because '' | ||
| + | This is done by creating a script to replace the '' | ||
| + | The script below does this while relying on the '' | ||
| + | So run '' | ||
| + | To install the block script, move the binary at ''/ | ||
| + | |||
| + | The '' | ||
| + | So if this file does not exist, the extroot will not be configured. | ||
| + | This allows one to easily disable setting up the crypto disk early on, thus effectively disabling extroot. | ||
| + | Once on the extroot, to access this file you'll need to mount the '' | ||
| + | |||
| + | Once the '' | ||
| + | |||
| + | <code bash block> | ||
| + | #!/bin/sh | ||
| + | |||
| + | # Prereqs: | ||
| + | # * packages: | ||
| + | # * block-mount | ||
| + | # * cryptsetup | ||
| + | # * move /sbin/block to / | ||
| + | # * install decrypt script to / | ||
| + | # | ||
| + | # This script should be placed at / | ||
| + | # or /sbin/block if already on the overlayfs and be set with execute | ||
| + | # permission. | ||
| + | # It is expected that the extroot is on a device that the kernel names as | ||
| + | # sd* or mmcblk*, otherwise modify appropriately. | ||
| + | |||
| + | # Set to 1 to enable debug logs | ||
| + | export DEBUG= | ||
| + | |||
| + | SDIR=${0%/ | ||
| + | BLOCK=" | ||
| + | LD_LIBRARY_PATH=${LD_LIBRARY_PATH: | ||
| + | LD_LIBRARY_PATH=" | ||
| + | PATH=$PATH: | ||
| + | |||
| + | block() { | ||
| + | ( exec -a ${0} ${BLOCK} " | ||
| + | } | ||
| + | |||
| + | if [ " | ||
| + | exec block " | ||
| + | fi | ||
| + | |||
| + | get_jiffies() { | ||
| + | head -n3 / | ||
| + | } | ||
| + | |||
| + | if [ -z " | ||
| + | TIME=$(get_jiffies) | ||
| + | export BLOCK_LOG="/ | ||
| + | exec 2>" | ||
| + | set -x | ||
| + | fi | ||
| + | |||
| + | if [ ! -x " | ||
| + | echo " | ||
| + | return 1 | ||
| + | fi | ||
| + | |||
| + | if [ " | ||
| + | # We are being called to setup the extroot, so make sure crypto block | ||
| + | # devices are all setup. | ||
| + | |||
| + | # Hotplug runs too late, create device nodes for /dev/sd*, if there are any | ||
| + | for SYSDEVPATH in / | ||
| + | [ ! -f " | ||
| + | [ -e "/ | ||
| + | MAJMIN=$(cat " | ||
| + | mknod / | ||
| + | done | ||
| + | |||
| + | # Load modules needed for cryptsetup | ||
| + | KVER=$(uname -r) | ||
| + | insmod ${SDIR}/ | ||
| + | insmod ${SDIR}/ | ||
| + | insmod ${SDIR}/ | ||
| + | insmod ${SDIR}/ | ||
| + | |||
| + | # FIXME: Why does block info only show ubi devices? | ||
| + | # block info | cut -d: -f1 | | ||
| + | # Do this hack instead, only check scsi and mmc devices | ||
| + | find /dev -type b | grep -E "/ | ||
| + | while read DEVPATH; do | ||
| + | cryptsetup --disable-locks isLuks $DEVPATH || continue | ||
| + | export ACTION=add DEVNAME=" | ||
| + | # Assume this script is located in $OVERLAY/ | ||
| + | ALTROOT=" | ||
| + | done | ||
| + | fi | ||
| + | |||
| + | block " | ||
| + | </ | ||
| + | |||
| + | === / | ||
| + | There is another way to work around the current limitations. | ||
| + | **However, it should only be used if the above method does not work for your setup, it is more prone to breaking or having strange side effects**. | ||
| + | The basic idea is that extroot will be setup as in the [[# | ||
| + | This will be expected. | ||
| + | Modifications to ''/ | ||
| + | |||
| + | So at this point your uci fstab configuration should have a mount section with target ''/ | ||
| + | I use the '' | ||
| + | Here's a relevant snippet of script that illustrates what needs to be put into ''/ | ||
| + | Currently this script will not work for LUKS volumes being opened with a password. | ||
| + | The volume must be opened with a keyfile (stdin is not properly setup in ''/ | ||
| + | In the script below, the key is stored at ''/ | ||
| + | Check your threat model to see if this works for you. | ||
| + | |||
| + | <code bash> | ||
| + | # Only setup the encrypted extroot if / | ||
| + | # This makes it easier disable the encrypted extroot from failsafe mode. | ||
| + | mkdir -p /mnt/tmp | ||
| + | if [ -e / | ||
| + | # Setup crypt device which contains the extroot | ||
| + | cryptsetup open -d / | ||
| + | umount /overlay | ||
| + | |||
| + | # /tmp will get overridden by another tmpfs by mount_root, but we need the | ||
| + | # initial one because it contains the ubus named socket. | ||
| + | mount --bind /tmp /mnt/tmp | ||
| + | |||
| + | # Re-run mount_root now that we have a block device that it will recognize | ||
| + | # as an extroot. This sleep is needed, otherwise procd seems to freak out | ||
| + | # and the watchdog timer doesn' | ||
| + | sleep 5 | ||
| + | PREINIT=1 mount_root | ||
| + | |||
| + | # Free the new tmpfs just created by mount_root. Since it will never be used, | ||
| + | # its just wasting memory. | ||
| + | umount -l /tmp | ||
| + | |||
| + | # Put the original tmpfs back to where it was in the VFS, primarily so that | ||
| + | # programs can find the ubus socket. | ||
| + | mount --bind / | ||
| + | |||
| + | # Need to re-run this too for some reason, otherwise some other mounts are not | ||
| + | # mounted after mount_root, eg. /rwm. | ||
| + | block mount | ||
| + | |||
| + | # Reload rpcd to register rpc objects on the extroot | ||
| + | service rpcd reload | ||
| + | fi | ||
| + | </ | ||
| + | |||
| + | **NOTE:** Since this method is essentially redoing some of the boot process, it does take longer. | ||
| + | On my device, its about 20-30 seconds longer for the web interface to be available. | ||
| + | Logging in via SSH is not delayed though. | ||
| ==== System upgrade ==== | ==== System upgrade ==== | ||
| Line 321: | Line 505: | ||
| ==== Automated upgrade ==== | ==== Automated upgrade ==== | ||
| + | Set up [[docs: | ||
| + | Packages required by Extroot should be saved in the '' | ||
| + | |||
| <code bash> | <code bash> | ||
| cat << " | cat << " | ||
| - | if uci -q get fstab.overlay | + | if uci -q get fstab.extroot |
| && [ ! -e / | && [ ! -e / | ||
| - | && lock -n / | + | && [ -e / |
| - | && [ -e /etc/opkg-restore-init ] | + | && lock -n / |
| + | then | ||
| + | UUID=" | ||
| + | DIR=" | ||
| + | DEV=" | ||
| + | if touch /etc/extroot-restore | ||
| + | && grep -q -e " | ||
| + | && mount " | ||
| then | then | ||
| - | UUID=" | ||
| - | OVRL=" | ||
| - | mount " | ||
| BAK=" | BAK=" | ||
| mv -f /mnt/etc /mnt/upper " | mv -f /mnt/etc /mnt/upper " | ||
| - | touch / | + | cp -f -a " |
| - | if grep -q -e " | + | umount " |
| - | then cp -f -a /overlay/. /mnt | + | |
| fi | fi | ||
| - | umount " | ||
| lock -u / | lock -u / | ||
| reboot | reboot | ||