| Both sides previous revision Previous revision Next revision | Previous revision |
| docs:techref:sysupgrade [2022/02/02 00:44] – [Options] Mostly rewrite the the section. Describe in greater detail what files are preserved and convert ad-hoc question/answer into Q/A style. dansan | docs:techref:sysupgrade [2022/09/25 18:36] (current) – [Sysupgrade – Technical Reference] url typo fix wifime |
|---|
| | This information is based upon v21.02, last updated for [[https://github.com/openwrt/openwrt/tree/6d266ef158|commit 6d266ef158 on 2022-02-10]]. |
| | |
| ====== Sysupgrade – Technical Reference ====== | ====== Sysupgrade – Technical Reference ====== |
| In contrast to ''opkg'', ''mtd'' and others, ''sysupgrade'' is merely a shell script: ''[[https://github.com/openwrt/openwrt/blob/master/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]'' intended to facilitate easy updates. | In contrast to ''opkg'', ''mtd'' and others, ''sysupgrade'' is merely a shell script: ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]'' intended to facilitate easy updates. |
| |
| ===== Usage ===== | ===== Usage ===== |
| |
| ===== Options ===== | ===== Options ===== |
| sysupgrade supports the following options ([[https://github.com/openwrt/openwrt/blob/fdbdbe8eaaa6aa3acacdcb3ae1308b2a2055fc39/package/base-files/files/sbin/sysupgrade|fdbdbe8]] Feb 1, 2021): | sysupgrade supports the following options |
| |
| <code> | <code> |
| ===== How It Works ===== | ===== How It Works ===== |
| |
| The ''sysupgrade'' process works roughly like this, starting from the execution of ''/sbin/sysupgrade'': | The ''sysupgrade'' process starts with the execution of ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]''. The below list describes its behavior. |
| |
| - Locate the file to upgrade from on the filesystem, or if the second option to sysupgrade starts with http://, download the firmware file using ''wget''. | - Parse command line and validate no mutually exclusive options passed. |
| - Do some minor validation of various things, and grab whatever config files the end user wants to be restored and packs them up into a tarball. | - At this point, sysupgrade calls ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade#L219|include /lib/upgrade]]'' -- a function in ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/functions.sh#L293|/lib/functions.sh]]'' that will source all ''*.sh'' files in the given directory. |
| - Send a message, via ''ubus'', to ''procd'', to initiate the upgrade. | - NOTE: An optional, platform-specific ''/lib/upgrade/platform.sh'' can override behavior, so for a full understanding, you should examine this file. (See ''target/linux/<arch>/<sub-arch>/base-files/lib/upgrade/platform.sh'' in your source tree.) |
| - ''procd'' does some stuff which I haven't finished completely understanding just yet, but it looks like firmware verification to make sure we don't upgrade to a bad firmware file. | - The optional functions ''platform_copy_config'' and ''platform_do_upgrade'' are at the end of this list. |
| | - Create list of files to preserve and store them in ''/tmp/sysupgrade.tgz'' (unless ''-f'' was supplied). |
| | - If the image supplied is an http or https URL, ''wget'' is run to retrieve it. |
| | - The firmware image is saved (if a URL) or copied to ''/tmp/sysupgrade.img''. |
| | - Validates the firmware image, and that the root ''compatible'' node in the device tree file matches the value in ''/sys/firmware/devicetree/base/compatible'' from the existing firmware's device tree. (May be overridden with ''-F''). |
| | - Copies ''[[commit>?p=project/procd.git;a=blob;f=upgraded/upgraded.c;h=db98701d8ab64da4c78f105a6182ae0db9edb062;hb=2cfc26f8456a4d5ba3836c914a742f3d00bad781|/sbin/upgraded]]'' into ''/tmp'' |
| | - Builds a [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=system.c;h=83aea423ec6aaceedca54e42aea18ce90d7ddfa1;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l627|json message]] and sends a message, via ''ubus'', to ''procd'', to initiate the upgrade. Among other things, the message specifies: |
| | - path: ''/tmp/sysupgrade.img'', |
| | - backup: ''/tmp/sysupgrade.tgz'' if any files are being preserved, unset otherwise, |
| | - force: if ''-F'' was supplied, |
| | - and command: ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/upgrade/do_stage2|/lib/upgrade/do_stage2]]''. |
| | - The ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=system.c;h=83aea423ec6aaceedca54e42aea18ce90d7ddfa1;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l651|sysupgrade]]'' function in ''procd'' will unpack and validate the message, then validate the firmware image and such. |
| - Notably, ''procd'' does not terminate any services here. | - Notably, ''procd'' does not terminate any services here. |
| - ''procd'' replaces itself (via ''execvp'') with the program ''/sbin/upgraded''. Service management is no longer possible. | - **This is where things get funky!** If all is well, we call [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=sysupgrade.c;h=fc588b0248353137d4b81fce130d2d35d8dfa710;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l28|sysupgrade_exec_upgraded]], which further parses the original json and then passes control to ''[[commit>?p=project/procd.git;a=blob;f=upgraded/upgraded.c;h=db98701d8ab64da4c78f105a6182ae0db9edb062;hb=2cfc26f8456a4d5ba3836c914a742f3d00bad781#l66|/sbin/upgraded]]'' via ''[[https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html|execvp]]''. |
| - ''/sbin/upgraded'' now acts as PID1. It executes the shell script ''/lib/upgrade/stage2'' with parameters. The remaining sequence runs from this shell script. | - Note that at boot time, the kernel passes control to ''/sbin/init'' as PID=1, which in turn ''exec''s ''/sbin/procd''. Thus, this results in ''/tmp/upgraded'' becoming the new PID=1 ("init") process. |
| | - At this point, service management is no longer possible. |
| | - ''/sbin/upgraded'' executes the command passed (''/lib/upgrade/stage2'') with parameters. The remaining sequence runs from this shell script. |
| | - ''/bin/sh /lib/upgrade/stage2'' is run via fork/exec, so is not PID1. |
| - Terminate (''SIGKILL'') all ''telnet'', ''ash'', and ''dropbear'' processes. | - Terminate (''SIGKILL'') all ''telnet'', ''ash'', and ''dropbear'' processes. |
| - Loop over all remaining processes, sending them the TERM signal. | - Loop over all remaining processes, sending them the TERM signal. |
| - Change root into the new RAM filesystem. | - Change root into the new RAM filesystem. |
| - Remount ''/overlay'' read-only, and lazy-unmount it. | - Remount ''/overlay'' read-only, and lazy-unmount it. |
| - Write the upgraded firmware to disk. (This can include platform/device-specific upgrade steps in ''platform_do_upgrade'', handling ''mtd'' or partitioning details.) | - Write the upgraded firmware to disk. If ''platform_do_upgrade'' is defined then it is run. Otherwise, this is done via [[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/upgrade/common.sh#L301|default_do_upgrade]], using ''mtd'' to flash the firmware. |
| - Write the saved configuration (if any) to disk. (This is a platform-specific step, via ''platform_copy_config''.) | - If any files are being preserved, the tarball is passed via ''mtd'''s ''-j'' option. This causes mtd to [[https://github.com/openwrt/openwrt/blob/6d266ef158/package/system/mtd/src/jffs2.c#L163|write a raw jffs2 inode (with id = 1) and file data]], resulting in the tarball to appearing as ''/sysupgrade.tgz'' once the jffs2 file system is mounted. |
| | - A consequence is that the mtd sources are intimately tied to the jffs2 implementation and can break if changes are made to jffs2 in the kernel (it should probably be using the uapi header from the kernel ''make headers_install'' instead of copying it into its source tree). |
| | - After reboot, this file will be extracted by ''/lib/preinit/80_mount_root'' (''/sbin/init'' during [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=initd/preinit.c;h=46411aa413a2a65614cfc765d3d6a42dee200532;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l157|preinit]] --> ''/bin/sh [[https://github.com/openwrt/openwrt/blob/fdbdbe8eaaa6aa3acacdcb3ae1308b2a2055fc39/package/base-files/files/etc/preinit|/etc/preinit]]'' --> ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/preinit/80_mount_root#L20|/lib/preinit/80_mount_root]]'') |
| | - and finally deleted by ''/etc/rc.d/S95done'' (''/sbin/procd'' --> at ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=state.c;h=fb81248fd7e7bc7b588a82a5e667424af188cbab;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l159|STATE_INIT]]'' --> ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=inittab.c;h=45118f4f1eaf05867e98040562131dc7df8e233e;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l273|procd_inittab_run("sysinit");]]'' --> ''/etc/inittab'') |
| | - If a ''platform_copy_config'' is implemented, it is run at this point. |
| - Unmount any remaining filesystems. | - Unmount any remaining filesystems. |
| - Reboot. | - Reboot. |
| * Easily broken by open files on storage devices (e.g., for swap, or explicitly opened), as these can prevent unmounting ''/overlay''. | * Easily broken by open files on storage devices (e.g., for swap, or explicitly opened), as these can prevent unmounting ''/overlay''. |
| * Does not handle errors (e.g., I/O) in writing the disk image, potentially leaving a partially-upgraded system upon reboot. | * Does not handle errors (e.g., I/O) in writing the disk image, potentially leaving a partially-upgraded system upon reboot. |
| | * Various error conditions are ignored in ''mtd'', which can also result in preserved files not being written to the new jffs2 file system or and leaving the file system corrupted, but without reporting an error. |
| |
| Many of these deficiencies are historical artifacts, remaining simply because no one has fixed them. | Many of these deficiencies are historical artifacts, remaining simply because no one has fixed them. |
| |
| Thanks to Michael Jones for writing most of this down on the mailing list [[http://lists.openwrt.org/pipermail/openwrt-devel/2020-May/029074.html|[OpenWrt-Devel] Sysupgrade and Failed to kill all processes]]. | Thanks to Michael Jones for writing most of this down on the mailing list [[http://lists.openwrt.org/pipermail/openwrt-devel/2020-May/029074.html|[OpenWrt-Devel] Sysupgrade and Failed to kill all processes]]. |
| | |
| | TODO: Explain how this process works during failsafe mode. |