| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
| docs:guide-user:additional-software:extroot_configuration [2023/10/05 20:31] – [Introduction] vgaetera | docs:guide-user:additional-software:extroot_configuration [2024/03/07 09:46] – kmod-usb-storage often is required as well. zpe |
|---|
| | ''overlay'' | ''/'' | Unmodified files | Yes | | | ''overlay'' | ''/'' | Unmodified files | Yes | |
| |
| This way OpenWrt fits even in tiny amounts of internal storage (as low as 4 MiB), but still allows to write settings and install some packages in the writable partition without changing all linux programs used. | This way OpenWrt fits even in tiny amounts of internal storage (as low as 4 MiB), but still allows to write settings and install some packages in the writable partition without changing all Linux programs used. |
| Extroot works by setting another overlay partition in the external storage device, and during boot this new overlay partition will be mounted over the internal storage's overlay partition. | Extroot works by setting another overlay partition in the external storage device, and during boot this new overlay partition will be mounted over the internal storage's overlay partition. |
| This approach allows easy fallback in case the external storage device is removed, as your device will still have its own overlay partition and thus will load all configuration from there. | This approach also allows easy fallback in case the external storage device is removed, as your device will still have its own overlay partition and thus will load all configuration from there. |
| Which means that it will behave exactly the same as just before you set up extroot. | Which means that it will behave exactly the same as just before you set up extroot. |
| |
| Note that Extroot is not possible to use on devices that do not have overlay partition on ''/proc/mtd'' or on ROMs that do not have overlay partition at all. | Note that OpenWrt is known to [[flyspray>2231|ignore]] the fstab configuration on devices without overlay partition in ''/proc/mtd''. |
| In the first case OpenWrt will not want to read the configuration of ''/etc/config/fstab'', see [[flyspray>2231|FS#2231]], while in the latter case you can work around it by mounting the external/additional disk directly to ''/''. | You can work around the issue by using ''/'' for the mount point on ROMs without overlay partition at all. |
| |
| ===== Instructions ===== | ===== Instructions ===== |
| The following instructions assume that you already have access to a shell on your OpenWRT device. Most if not all of these commands can be done via the web interface, however that is emphatically not recommended. Usually the shell is accessed via [[:docs:guide-quick-start:sshadministration|ssh]] or [[:docs:techref:hardware:port.serial|serial console]]. | The following instructions assume that you already have access to a shell on your OpenWRT device. |
| | Most if not all of these commands can be done via the web interface, however that is emphatically not recommended. |
| | Usually the shell is accessed via [[:docs:guide-quick-start:sshadministration|ssh]] or [[:docs:techref:hardware:port.serial|serial console]]. |
| ==== 1. Preparation ==== | ==== 1. Preparation ==== |
| Devices with 8 MiB flash or more should have enough space to install the required packages, otherwise create a [[docs:guide-user:additional-software:extroot_configuration#custom_image|custom image]]. | Devices with 8 MiB flash or more should have enough space to install the required packages, otherwise create a [[docs:guide-user:additional-software:extroot_configuration#custom_image|custom image]]. |
| Remove all packages you have installed to add secondary functionality, as they are only wasting space now. Leave only those needed to access the internet and needed to access the extroot filesystem. \\ | Remove all packages you have installed to add secondary functionality, as they are only wasting space now. |
| | (If you do not have a record of what these are, try removing 'ntfs' as that may free up enough space.) |
| | Leave only those needed to access the internet and needed to access the extroot filesystem. |
| After you make the extroot you will have all the space you need to install secondary packages. | After you make the extroot you will have all the space you need to install secondary packages. |
| |
| The extroot can be anything that ''block'' can mount. Currently ''block'' creates some restrictions on what extroot can be. It must a [[commit>?p=project/fstools.git;a=blob;f=block.c;hb=HEAD#l1554|filesystem of type]]: ext2/3/4, f2fs, btrfs, ntfs, or ubifs (note that it can not be a FAT16/32 filesystem). For most, this filesystem will be a on USB storage device. However, it could also be on an SD-Card or a SATA drive connected via e-sata or even a network block device (assuming its set up early enough). If you're using a USB connected device follow the [[docs:guide-user:storage:usb-installing| USB installation guide]] to set up USB storage in OpenWrt. | You may not need to make a custom image: try the OEM image first (OpenWRT GL.inet for a GL.inet mango). |
| The following assumes that you will be creating your extroot as an EXT4 filesystem on your OpenWRT device with a connected USB flash drive. The process is similar for other kinds of devices. | |
| |
| This will install the required packages and create the extroot filesystem. **Note: This will wipe all data on your USB stick.** It is further assumed that the USB flash shows up as block device /dev/sda. Please **DO NOT** run these commands blindly, verify that they are really what you want to do. | The extroot can be anything that ''block'' can mount. |
| <code> | Currently ''block'' creates some restrictions on what extroot can be. |
| opkg update | It must a [[commit>?p=project/fstools.git;a=blob;f=block.c;hb=HEAD#l1554|filesystem of type]]: ext2/3/4, f2fs, btrfs, ntfs, or ubifs (note that it can not be a FAT16/32 filesystem). |
| opkg install block-mount kmod-fs-ext4 e2fsprogs parted | For most, this filesystem will be a on USB storage device. |
| parted -s /dev/sda -- mklabel gpt mkpart extroot 2048s -2048s | However, it could also be on an SD-Card or a SATA drive connected via e-sata or even a network block device (assuming its set up early enough). |
| </code> | If you're using a USB connected device follow the [[docs:guide-user:storage:usb-installing| USB installation guide]] to set up USB storage in OpenWrt. |
| |
| ==== 2. Configuring rootfs_data / ubifs ==== | The following assumes that you will be creating your extroot as an EXT4 filesystem on your OpenWRT device with a connected USB flash drive. |
| Configure ''/etc/config/fstab'' to mount the ''rootfs_data'' or ''ubifs'' in another directory in case you need to access the original root overlay to change your extroot settings: | The process is similar for other kinds of devices. |
| |
| <code bash> | Installing these packages requires a sensible amount of extra filespace. If you completely fill the filesystem by installing these, |
| DEVICE="$(block info | sed -n -e '/MOUNT="\S*\/overlay"/s/:\s.*$//p')" | you will probably have to re-flash the entire system. So if you think you may already be close to filling the filesystem, |
| uci -q delete fstab.rwm | remove some installed packages first. Good candidates for removal are ntfs3 and ntfs3-utils: you can re-install them later after |
| uci set fstab.rwm="mount" | you have extroot installed. |
| uci set fstab.rwm.device="${DEVICE}" | |
| uci set fstab.rwm.target="/rwm" | |
| uci commit fstab | |
| </code> | |
| |
| The ''/rwm'' mount will not mount via ''block'' until you've already successfully booted into your extroot configuration. This is because ''block'' has a restriction to only mount from devices that are not currently mounted. And ''/rwm'' should already be mounted at ''/overlay''. Once booted into your extroot, you can edit ''/rwm/upper/etc/config/fstab'' to change your extroot configuration (or temporarily disable it) should you ever need to. | Install the required packages. |
| | |
| ==== 3. Configuring extroot ==== | |
| See what partitions you have using the following command: | |
| |
| <code bash> | <code bash> |
| block info | opkg update |
| | opkg install block-mount kmod-fs-ext4 e2fsprogs parted kmod-usb-storage |
| </code> | </code> |
| |
| You will see similar output: | Identify the name of the USB disk. |
| |
| <code bash> | <code bash> |
| /dev/mtdblock2: UUID="9fd43c61-c3f2c38f-13440ce7-53f0d42d" VERSION="4.0" MOUNT="/rom" TYPE="squashfs" | ls -l /sys/block |
| /dev/mtdblock3: MOUNT="/overlay" TYPE="jffs2" | |
| /dev/sda1: UUID="fdacc9f1-0e0e-45ab-acee-9cb9cc8d7d49" VERSION="1.4" TYPE="ext4" | |
| </code> | </code> |
| |
| Here ''mtdblock'' are the devices in internal flash memory, and ''/dev/sda1'' is the partition on a USB flash drive that we have already formatted to ext4 like this: | ==== 2. Partitioning and formatting ==== |
| | Partition and format the USB disk. |
| |
| <code bash> | <code bash> |
| DEVICE="/dev/sda1" | DISK="/dev/sda" |
| | parted -s ${DISK} -- mklabel gpt mkpart extroot 2048s -2048s |
| | DEVICE="${DISK}1" |
| mkfs.ext4 -L extroot ${DEVICE} | mkfs.ext4 -L extroot ${DEVICE} |
| </code> | </code> |
| |
| Now we configure the selected partition as new overlay via fstab UCI subsystem: | This will wipe all data on the disk, so do not run these commands blindly. |
| | |
| | ==== 3. Configuring extroot ==== |
| | Configure the extroot mount entry. |
| |
| <code bash> | <code bash> |
| eval $(block info | grep -o -e 'MOUNT="\S*/overlay"') | |
| eval $(block info ${DEVICE} | grep -o -e 'UUID="\S*"') | eval $(block info ${DEVICE} | grep -o -e 'UUID="\S*"') |
| | eval $(block info | grep -o -e 'MOUNT="\S*/overlay"') |
| uci -q delete fstab.extroot | uci -q delete fstab.extroot |
| uci set fstab.extroot="mount" | uci set fstab.extroot="mount" |
| |
| ==== 4. Transferring data ==== | ==== 4. Transferring data ==== |
| We now transfer the content of the current overlay to the external drive and reboot the device to apply changes: | Transfer the content of the current overlay to the external drive. |
| |
| <code bash> | <code bash> |
| mount ${DEVICE} /mnt | mount ${DEVICE} /mnt |
| tar -C ${MOUNT} -cvf - . | tar -C /mnt -xf - | tar -C ${MOUNT} -cvf - . | tar -C /mnt -xf - |
| | </code> |
| | |
| | ==== 5. Configuring rootfs_data / ubifs ==== |
| | Configure a mount entry for the the original overlay. |
| | |
| | <code bash> |
| | DEVICE="$(block info | sed -n -e '/MOUNT="\S*\/overlay"/s/:\s.*$//p')" |
| | uci -q delete fstab.rwm |
| | uci set fstab.rwm="mount" |
| | uci set fstab.rwm.device="${DEVICE}" |
| | uci set fstab.rwm.target="/rwm" |
| | uci commit fstab |
| | </code> |
| | |
| | This will allow you to access the ''rootfs_data'' / ''ubifs'' partition and customize the extroot configuration ''/rwm/upper/etc/config/fstab''. |
| | |
| | ==== 6. Apply changes ==== |
| | Reboot the device to apply the changes. |
| | |
| | <code bash> |
| reboot | reboot |
| </code> | </code> |
| |
| * Do not use vfat (FAT/FAT32); it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using ''mkfs.ext4 /dev/sda1'' as per the example). | * Do not use vfat (FAT/FAT32); it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using ''mkfs.ext4 /dev/sda1'' as per the example). |
| * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase ''config global / option delay_root''. On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot. | * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase ''config global / option delay_root''. On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot. |
| |
| <code bash> | <code bash> |
| <code bash> | <code bash> |
| # Create swap file | # Create swap file |
| dd if=/dev/zero of=/overlay/swap bs=1M count=100 | DIR="$(uci -q get fstab.extroot.target)" |
| mkswap /overlay/swap | dd if=/dev/zero of=${DIR}/swap bs=1M count=100 |
| | mkswap ${DIR}/swap |
| |
| # Enable swap file | # Enable swap file |
| uci -q delete fstab.swap | uci -q delete fstab.swap |
| uci set fstab.swap="swap" | uci set fstab.swap="swap" |
| uci set fstab.swap.device="/overlay/swap" | uci set fstab.swap.device="${DIR}/swap" |
| uci commit fstab | uci commit fstab |
| /etc/init.d/fstab boot | service fstab boot |
| |
| # Verify swap status | # Verify swap status |
| Follow: [[https://forum.openwrt.org/viewtopic.php?id=32812|Fstab not mounting cifs at boot or through CLI]] | Follow: [[https://forum.openwrt.org/viewtopic.php?id=32812|Fstab not mounting cifs at boot or through CLI]] |
| |
| ==== LUKS Encrypted extroot ==== | ==== LUKS encrypted extroot ==== |
| You may wish to have your extroot filesystem in a LUKS encrypted container. As of OpenWRT 22.03.2, this [[https://forum.openwrt.org/t/extroot-encryption/133230/6|isn't well supported]]. OpenWRT does not have an official way to open encrypted LUKS volumes before the extroot check happens during the normal boot path. So at the time of extroot check time, the extroot filesystem will not be visible and the boot process will continue as if there is not extroot. Below are two different methods for getting the system to run on an encrypted extroot. The first method is preferable because there are less side-effects and is a cleaner approach. | You may wish to have your extroot filesystem in a LUKS encrypted container. |
| | As of OpenWrt 22.03.2, this [[https://forum.openwrt.org/t/extroot-encryption/133230/6|isn't well supported]]. |
| | OpenWrt does not have an official way to open encrypted LUKS volumes before the extroot check happens during the normal boot path. |
| | So at the time of extroot check time, the extroot filesystem will not be visible and the boot process will continue as if there is not extroot. |
| | Below are two different methods for getting the system to run on an encrypted extroot. |
| | The first method is preferable because there are less side-effects and is a cleaner approach. |
| |
| Before doing any of the below, you'll need to create the LUKS container in which to put your extroot filesystem. Follow the [[:docs:guide-user:storage:disk.encryption|disk encryption]] documentation to get a LUKS container setup on your device. You will need enough space on your ''rootfs_data'' to install ''cryptsetup'' and its dependencies. Once, you have your LUKS container follow the [[#instructions]] above for creating the extroot filesystem on the unlocked LUKS device, including copying the ''rootfs_data'' files from ''/overlay'' to the newly created extroot. | Before doing any of the below, you'll need to create the LUKS container in which to put your extroot filesystem. |
| | Follow the [[docs:guide-user:storage:disk.encryption|disk encryption]] documentation to get a LUKS container setup on your device. |
| | You will need enough space on your ''rootfs_data'' to install ''cryptsetup'' and its dependencies. |
| | Once, you have your LUKS container follow the [[#instructions]] above for creating the extroot filesystem on the unlocked LUKS device, including copying the ''rootfs_data'' files from ''/overlay'' to the newly created extroot. |
| |
| === PREINIT === | === PREINIT === |
| In the PREINIT phase of boot, ''mount_root'' will be run, which will check for an extroot config on first the ROM device and then on the [[https://github.com/openwrt/fstools/blob/master/libfstools/overlay.c#L439|''rootfs_data'' device]] (see [[docs:techref:flash.layout|The OpenWrt Flash Layout]] for more on the flash layout). Using a stock OpenWRT firmware, there will be no extroot config on the ROM device. To check the ''rootfs_data'' filesystem, ''mount_root'' will [[https://github.com/openwrt/fstools/blob/master/libfstools/overlay.c#L431|mount the filesystem to a temporary location]] (''/tmp/overlay''), check the file ''/tmp/overlay/etc/config/fstab'', which is the same fstab configured above, for a configured extroot. If found, it will search in order ''/tmp/overlay/upper/sbin/block'', ''/tmp/overlay/sbin/block'', and ''/sbin/block'' for the presence of the block binary, and if it finds one will [[https://github.com/openwrt/fstools/blob/master/libfstools/extroot.c#L69|run that binary]] with the ''extroot'' argument. This is why the ''block-mount'' package must be installed for extroot to work. | In the PREINIT phase of boot, ''mount_root'' will be run, which will check for an extroot config on first the ROM device and then on the [[https://github.com/openwrt/fstools/blob/master/libfstools/overlay.c#L439|''rootfs_data'' device]] (see [[docs:techref:flash.layout|The OpenWrt Flash Layout]] for more on the flash layout). |
| | Using a stock OpenWRT firmware, there will be no extroot config on the ROM device. |
| | To check the ''rootfs_data'' filesystem, ''mount_root'' will [[https://github.com/openwrt/fstools/blob/master/libfstools/overlay.c#L431|mount the filesystem to a temporary location]] (''/tmp/overlay''), check the file ''/tmp/overlay/etc/config/fstab'', which is the same fstab configured above, for a configured extroot. |
| | If found, it will search in order ''/tmp/overlay/upper/sbin/block'', ''/tmp/overlay/sbin/block'', and ''/sbin/block'' for the presence of the block binary, and if it finds one will [[https://github.com/openwrt/fstools/blob/master/libfstools/extroot.c#L69|run that binary]] with the ''extroot'' argument. |
| | This is why the ''block-mount'' package must be installed for extroot to work. |
| |
| Because ''mount_root'' which is running from the ROM in the PREINIT phase attempts to run ''block'' from the overlay filesystem, we can use this as a way to inject commands/code into the PREINIT phase before the extroot check takes place. This is done by creating a script to replace the ''block'' binary which will be in charge of setting up the encrypted device, getting the extroot filesystem visible to the system, and then running the real ''block''. The script below does this while relying on the ''decrypt.sh'' script from the [[docs:guide-user:storage:disk.encryption|disk encryption tutorial]]. So run ''install-decrypt.sh'' from there first. To install the block script, move the binary at ''/sbin/block'' to ''/sbin/block.bin'', and move the ''block'' script to ''/sbin/block'' and make sure it has the executable permission bit set. | Because ''mount_root'' which is running from the ROM in the PREINIT phase attempts to run ''block'' from the overlay filesystem, we can use this as a way to inject commands/code into the PREINIT phase before the extroot check takes place. |
| | This is done by creating a script to replace the ''block'' binary which will be in charge of setting up the encrypted device, getting the extroot filesystem visible to the system, and then running the real ''block''. |
| | The script below does this while relying on the ''decrypt.sh'' script from the [[docs:guide-user:storage:disk.encryption|disk encryption tutorial]]. |
| | So run ''install-decrypt.sh'' from there first. |
| | To install the block script, move the binary at ''/sbin/block'' to ''/sbin/block.bin'', and move the ''block'' script to ''/sbin/block'' and make sure it has the executable permission bit set. |
| |
| The ''block'' script checks for the existence of a special path, ''/.use_crypt_extroot'' on the mounted overlayfs or ''/upper/.use_crypt_extroot'' on the ''/overlay'' filesystem, to determine if it should use do its magic. So if this file does not exist, the extroot will not be configured. This allows one to easily disable setting up the crypto disk early on, thus effectively disabling extroot. Once on the extroot, to access this file you'll need to mount the ''rootfs_data'', or you could turn off the device, remove the extroot block device, and boot without it. | The ''block'' script checks for the existence of a special path, ''/.use_crypt_extroot'' on the mounted overlayfs or ''/upper/.use_crypt_extroot'' on the ''/overlay'' filesystem, to determine if it should use do its magic. |
| | So if this file does not exist, the extroot will not be configured. |
| | This allows one to easily disable setting up the crypto disk early on, thus effectively disabling extroot. |
| | Once on the extroot, to access this file you'll need to mount the ''rootfs_data'', or you could turn off the device, remove the extroot block device, and boot without it. |
| |
| Once the ''block'' script has been properly installed, ''decrypt.sh'' has been installed, ''/etc/crypttab'' has been setup, the extroot has been configured, and ''/.use_crypt_extroot'' exists, then you are ready to reboot and enter your encrypted extroot. | Once the ''block'' script has been properly installed, ''decrypt.sh'' has been installed, ''/etc/crypttab'' has been setup, the extroot has been configured, and ''/.use_crypt_extroot'' exists, then you are ready to reboot and enter your encrypted extroot. |
| |
| === /etc/rc.local === | === /etc/rc.local === |
| There is another way to work around the current limitations. **However, it should only be used if the above method does not work for your setup, it is more prone to breaking or having strange side effects**. The basic idea is that extroot will be setup as in the [[#instructions]] section above, which will fail to load during the normal boot path because the extroot filesystem will not be found. This will be expected. Modifications to ''/etc/rc.local'' will unlock the LUKS volume at the end of the boot process when we have more control of the system and then we'll run ''mount_root'' again and this time it will find the extroot filesystem and switch root into it. | There is another way to work around the current limitations. |
| | **However, it should only be used if the above method does not work for your setup, it is more prone to breaking or having strange side effects**. |
| | The basic idea is that extroot will be setup as in the [[#instructions]] section above, which will fail to load during the normal boot path because the extroot filesystem will not be found. |
| | This will be expected. |
| | Modifications to ''/etc/rc.local'' will unlock the LUKS volume at the end of the boot process when we have more control of the system and then we'll run ''mount_root'' again and this time it will find the extroot filesystem and switch root into it. |
| |
| So at this point your uci fstab configuration should have a mount section with target ''/overlay''. I use the ''uuid'' option instead of the ''device'' option so I don't need to keep the ''/etc/rc.local'' synchronized with ''/etc/config/fstab''. Here's a relevant snippet of script that illustrates what needs to be put into ''/etc/rc.local''. Currently this script will not work for LUKS volumes being opened with a password. The volume must be opened with a keyfile (stdin is not properly setup in ''/etc/rc.local'' so cryptsetup will fail when trying to get a password). In the script below, the key is stored at ''/root/extroot.key''. Check your threat model to see if this works for you. | So at this point your uci fstab configuration should have a mount section with target ''/overlay''. |
| | I use the ''uuid'' option instead of the ''device'' option so I don't need to keep the ''/etc/rc.local'' synchronized with ''/etc/config/fstab''. |
| | Here's a relevant snippet of script that illustrates what needs to be put into ''/etc/rc.local''. |
| | Currently this script will not work for LUKS volumes being opened with a password. |
| | The volume must be opened with a keyfile (stdin is not properly setup in ''/etc/rc.local'' so cryptsetup will fail when trying to get a password). |
| | In the script below, the key is stored at ''/root/extroot.key''. |
| | Check your threat model to see if this works for you. |
| |
| <code bash> | <code bash> |
| |
| # Reload rpcd to register rpc objects on the extroot | # Reload rpcd to register rpc objects on the extroot |
| /etc/init.d/rpcd reload | service rpcd reload |
| fi | fi |
| </code> | </code> |
| |
| **NOTE:** Since this method is essentially redoing some of the boot process, it does take longer. On my device, its about 20-30 seconds longer for the web interface to be available. Logging in via SSH is not delayed though. | **NOTE:** Since this method is essentially redoing some of the boot process, it does take longer. |
| | On my device, its about 20-30 seconds longer for the web interface to be available. |
| | Logging in via SSH is not delayed though. |
| |
| ==== System upgrade ==== | ==== System upgrade ==== |