Differences

This shows you the differences between two versions of the page.

--- docs:techref:sysupgrade [2019/04/06 03:51] – list backup configurations vgaetera
+++ docs:techref:sysupgrade [2022/09/25 18:36] (current) – [Sysupgrade – Technical Reference] url typo fix wifime
@@ Line 1: / Line 1: @@
+This information is based upon v21.02, last updated for [[https://github.com/openwrt/openwrt/tree/6d266ef158|commit 6d266ef158 on 2022-02-10]].
 ====== Sysupgrade – Technical Reference ======
-In contrast to ''opkg'', ''mtd'' and others, ''sysupgrade'' is merely a shell script: ''[[https://dev.openwrt.org/browser/trunk/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]'' intended to facilitate easy updates.
+In contrast to ''opkg'', ''mtd'' and others, ''sysupgrade'' is merely a shell script: ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]'' intended to facilitate easy updates.
 ===== Usage =====
@@ Line 6: / Line 8: @@
 ===== Options =====
-sysupgrade supports the following options (r34975):
+sysupgrade supports the following options
 <code>
 Usage: /sbin/sysupgrade [<upgrade-option>...] <image file or URL>
-       /sbin/sysupgrade [-q] [-i] <backup-command> <file>
+       /sbin/sysupgrade [-q] [-i] [-c] [-u] [-o] [-k] <backup-command> <file>
 upgrade-option:
-        -d <delay>   add a delay before rebooting
         -f <config>  restore configuration from .tar.gz (file or url)
         -i           interactive mode
         -c           attempt to preserve all changed files in /etc/
+        -o           attempt to preserve all changed files in /, except those
+                     from packages but including changed confs.
+        -u           skip from backup files that are equal to those in /rom
         -n           do not save configuration over reflash
         -p           do not attempt to restore the partition table after flash.
+        -k           include in backup a list of current installed packages at
+                     /etc/backup/installed_packages.txt
         -T | --test
                      Verify image and config .tar.gz but do not actually flash.
@@ Line 41: / Line 47: @@
 </code>
+WARNING: Preserving files across sysupgrades [[this>docs/techref/preinit_mount#mount_root_filesystem|can be fatal]] (see 'NOTE: ...') on systems with weak cpu and exceptionally large rootfs_data partitions.
 Files to be preserved depend on the following:
@@ Line 46: / Line 54: @@
   * ''/lib/upgrade/keep.d/*'' - system configurations provided by specific packages preserved by default.
   * ''opkg list-changed-conffiles'' - list of files derived by package manager.
+  * ''-o'' will cause the entire ''/overlay'' directory to be saved (with the ''-u'' caveat below).
+  * ''-n'' will cause //NO// files will be saved and all configuration settings will be initialized from default values.
+  * ''-u'' will prevent preservation of any file that has not been changed since the last sysupgrade.  This prevents the need for programs to migrate an old configuration and reduces time needed for sysupgrade.
+  * ''-f'' will //COMPLETELY OVERRIDE// all behaviour described above.  Instead, the exact files provided in the .tar.gz file will be extracted into ''/overlay/upper'' after the sysupgrade.
+**Q:** Does this mean, I make an archive.tar.gz of /etc and /root for example and sysupgrade -f archive.tar.gz will flash the router and afterwards restores the configs from this archive?
+**A:** That's what is says: 'restore configuration from .tar.gz (file or url)'. Anything archived in the tgz will be written to /overlay after the flash. This way you can hand-pick the files that will be the system after new firmware boot.
+===== How It Works =====
+The ''sysupgrade'' process starts with the execution of ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade|/sbin/sysupgrade]]''.  The below list describes its behavior.
+  - Parse command line and validate no mutually exclusive options passed.
+  - At this point, sysupgrade calls ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/sbin/sysupgrade#L219|include /lib/upgrade]]'' -- a function in ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/functions.sh#L293|/lib/functions.sh]]'' that will source all ''*.sh'' files in the given directory.
+    - NOTE: An optional, platform-specific ''/lib/upgrade/platform.sh'' can override behavior, so for a full understanding, you should examine this file.  (See ''target/linux/<arch>/<sub-arch>/base-files/lib/upgrade/platform.sh'' in your source tree.)
+    - The optional functions ''platform_copy_config'' and ''platform_do_upgrade'' are at the end of this list.
+  - Create list of files to preserve and store them in ''/tmp/sysupgrade.tgz'' (unless ''-f'' was supplied).
+  - If the image supplied is an http or https URL, ''wget'' is run to retrieve it.
+  - The firmware image is saved (if a URL) or copied to ''/tmp/sysupgrade.img''.
+  - Validates the firmware image, and that the root ''compatible'' node in the device tree file matches the value in ''/sys/firmware/devicetree/base/compatible'' from the existing firmware's device tree. (May be overridden with ''-F'').
+  - Copies ''[[commit>?p=project/procd.git;a=blob;f=upgraded/upgraded.c;h=db98701d8ab64da4c78f105a6182ae0db9edb062;hb=2cfc26f8456a4d5ba3836c914a742f3d00bad781|/sbin/upgraded]]'' into ''/tmp''
+  - Builds a [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=system.c;h=83aea423ec6aaceedca54e42aea18ce90d7ddfa1;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l627|json message]] and sends a message, via ''ubus'', to ''procd'', to initiate the upgrade.  Among other things, the message specifies:
+    - path: ''/tmp/sysupgrade.img'',
+    - backup: ''/tmp/sysupgrade.tgz'' if any files are being preserved, unset otherwise,
+    - force: if ''-F'' was supplied,
+    - and command: ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/upgrade/do_stage2|/lib/upgrade/do_stage2]]''.
+  - The ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=system.c;h=83aea423ec6aaceedca54e42aea18ce90d7ddfa1;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l651|sysupgrade]]'' function in ''procd'' will unpack and validate the message, then validate the firmware image and such.
+  - Notably, ''procd'' does not terminate any services here.
+  - **This is where things get funky!** If all is well, we call [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=sysupgrade.c;h=fc588b0248353137d4b81fce130d2d35d8dfa710;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l28|sysupgrade_exec_upgraded]], which further parses the original json and then passes control to ''[[commit>?p=project/procd.git;a=blob;f=upgraded/upgraded.c;h=db98701d8ab64da4c78f105a6182ae0db9edb062;hb=2cfc26f8456a4d5ba3836c914a742f3d00bad781#l66|/sbin/upgraded]]'' via ''[[https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html|execvp]]''.
+    - Note that at boot time, the kernel passes control to ''/sbin/init'' as PID=1, which in turn ''exec''s ''/sbin/procd''.  Thus, this results in ''/tmp/upgraded'' becoming the new PID=1 ("init") process.
+    - At this point, service management is no longer possible.
+  - ''/sbin/upgraded'' executes the command passed (''/lib/upgrade/stage2'') with parameters. The remaining sequence runs from this shell script.
+    - ''/bin/sh /lib/upgrade/stage2'' is run via fork/exec, so is not PID1.
+  - Terminate (''SIGKILL'') all ''telnet'', ''ash'', and ''dropbear'' processes.
+  - Loop over all remaining processes, sending them the TERM signal.
+  - Loop over all remaining processes, sending them the KILL signal.
+  - Create a new RAM filesystem, mount it, and copy over a very small set of binaries into it.
+  - Change root into the new RAM filesystem.
+  - Remount ''/overlay'' read-only, and lazy-unmount it.
+  - Write the upgraded firmware to disk.  If ''platform_do_upgrade'' is defined then it is run.  Otherwise, this is done via [[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/upgrade/common.sh#L301|default_do_upgrade]], using ''mtd'' to flash the firmware.
+  - If any files are being preserved, the tarball is passed via ''mtd'''s ''-j'' option.  This causes mtd to [[https://github.com/openwrt/openwrt/blob/6d266ef158/package/system/mtd/src/jffs2.c#L163|write a raw jffs2 inode (with id = 1) and file data]], resulting in the tarball to appearing as ''/sysupgrade.tgz'' once the jffs2 file system is mounted.
+    - A consequence is that the mtd sources are intimately tied to the jffs2 implementation and can break if changes are made to jffs2 in the kernel (it should probably be using the uapi header from the kernel ''make headers_install'' instead of copying it into its source tree).
+    - After reboot, this file will be extracted by ''/lib/preinit/80_mount_root'' (''/sbin/init'' during [[https://git.openwrt.org/?p=project/procd.git;a=blob;f=initd/preinit.c;h=46411aa413a2a65614cfc765d3d6a42dee200532;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l157|preinit]] --> ''/bin/sh [[https://github.com/openwrt/openwrt/blob/fdbdbe8eaaa6aa3acacdcb3ae1308b2a2055fc39/package/base-files/files/etc/preinit|/etc/preinit]]'' --> ''[[https://github.com/openwrt/openwrt/blob/6d266ef158/package/base-files/files/lib/preinit/80_mount_root#L20|/lib/preinit/80_mount_root]]'')
+    - and finally deleted by ''/etc/rc.d/S95done'' (''/sbin/procd'' --> at ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=state.c;h=fb81248fd7e7bc7b588a82a5e667424af188cbab;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l159|STATE_INIT]]'' --> ''[[https://git.openwrt.org/?p=project/procd.git;a=blob;f=inittab.c;h=45118f4f1eaf05867e98040562131dc7df8e233e;hb=37eed131e9967a35f47bacb3437a9d3c8a57b3f4#l273|procd_inittab_run("sysinit");]]'' --> ''/etc/inittab'')
+  - If a ''platform_copy_config'' is implemented, it is run at this point.
+  - Unmount any remaining filesystems.
+  - Reboot.
+There are plenty of potential deficiencies in this process, among them:
-SAVE_OVERLAY saves the complete ''/overlay'' directory. There's no rule what needs to be saved... sometimes I install additional packages but then include them in the later firmware builds, so I don't have to preserve those. Sometimes files are modified and the new firmware already contains the modifications so those are also not needed. But config files are needed or router will boot with default settings. But those are preserved by default (unless using -n).
+  * Hardcodes a list of "potentially-interfering" / interactive processes (''ash'', ''telnet'', ''dropbear'') to kill first; this is not exhaustive or up-to-date (e.g., ''telnet'' is no longer in the base install; ''openssh'' is not handled).
+  * Does not give processes much time in between TERM and KILL signals.
+  * Does not utilise ''procd'' to tear down services.
+  * Susceptible to fork bombs.
+  * Easily broken by open files on storage devices (e.g., for swap, or explicitly opened), as these can prevent unmounting ''/overlay''.
+  * Does not handle errors (e.g., I/O) in writing the disk image, potentially leaving a partially-upgraded system upon reboot.
+  * Various error conditions are ignored in ''mtd'', which can also result in preserved files not being written to the new jffs2 file system or and leaving the file system corrupted, but without reporting an error.
--- Does this mean, I make an archive.tar.gz of /etc and /root for example and sysupgrade -f archive.tar.gz will flash the router and afterwards restores the configs from this archive?
+Many of these deficiencies are historical artifacts, remaining simply because no one has fixed them.
-That's what is says: 'restore configuration from .tar.gz (file or url)'. Anything archived in the tgz will be written to /overlay after the flash. This way you can hand-pick the files that will be the system after new firmware boot.
+Thanks to Michael Jones for writing most of this down on the mailing list [[http://lists.openwrt.org/pipermail/openwrt-devel/2020-May/029074.html|[OpenWrt-Devel] Sysupgrade and Failed to kill all processes]].
+TODO: Explain how this process works during failsafe mode.