Sysupgrade – Technical Reference

In contrast to opkg, mtd and others, sysupgrade is merely a shell script: /sbin/sysupgrade intended to facilitate easy updates.

This page lists all sysupgrade command-line options. For the overall upgrade procedure and typical usage, please read OpenWrt OS upgrade procedure (sysupgrade or LuCI) instead.

sysupgrade supports the following options (fdbdbe8 Feb 1, 2021):

Usage: /sbin/sysupgrade [<upgrade-option>...] <image file or URL>
       /sbin/sysupgrade [-q] [-i] [-c] [-u] [-o] [-k] <backup-command> <file>

upgrade-option:
        -f <config>  restore configuration from .tar.gz (file or url)
        -i           interactive mode
        -c           attempt to preserve all changed files in /etc/
        -o           attempt to preserve all changed files in /, except those
                     from packages but including changed confs.
        -u           skip from backup files that are equal to those in /rom
        -n           do not save configuration over reflash
        -p           do not attempt to restore the partition table after flash.
        -k           include in backup a list of current installed packages at
                     /etc/backup/installed_packages.txt
        -T | --test
                     Verify image and config .tar.gz but do not actually flash.
        -F | --force
                     Flash image even if image checks fail, this is dangerous!
        -q           less verbose
        -v           more verbose
        -h | --help  display this help

backup-command:
        -b | --create-backup <file>
                     create .tar.gz of files specified in sysupgrade.conf
                     then exit. Does not flash an image. If file is '-',
                     i.e. stdout, verbosity is set to 0 (i.e. quiet).
        -r | --restore-backup <file>
                     restore a .tar.gz created with sysupgrade -b
                     then exit. Does not flash an image. If file is '-',
                     the archive is read from stdin.
        -l | --list-backup
                     list the files that would be backed up when calling
                     sysupgrade -b. Does not create a backup file.

WARNING: Preserving files across sysupgrades can be fatal (see 'NOTE: ...') on systems with weak cpu and exceptionally large rootfs_data partitions.

Files to be preserved depend on the following:

• /etc/sysupgrade.conf - customizable backup configuration.
• /lib/upgrade/keep.d/* - system configurations provided by specific packages preserved by default.
• opkg list-changed-conffiles - list of files derived by package manager.
• -o will cause the entire /overlay directory to be saved (with the -u caveat below).
• -n will cause NO files will be saved and all configuration settings will be initialized from default values.
• -u will prevent preservation of any file that has not been changed since the last sysupgrade. This prevents the need for programs to migrate an old configuration and reduces time needed for sysupgrade.
• -f will COMPLETELY OVERRIDE all behaviour described above. Instead, the exact files provided in the .tar.gz file will be extracted into /overlay/upper after the sysupgrade.

Q: Does this mean, I make an archive.tar.gz of /etc and /root for example and sysupgrade -f archive.tar.gz will flash the router and afterwards restores the configs from this archive?

A: That's what is says: 'restore configuration from .tar.gz (file or url)'. Anything archived in the tgz will be written to /overlay after the flash. This way you can hand-pick the files that will be the system after new firmware boot.

The sysupgrade process works roughly like this, starting from the execution of /sbin/sysupgrade:

1. Locate the file to upgrade from on the filesystem, or if the second option to sysupgrade starts with http://, download the firmware file using wget.
2. Do some minor validation of various things, and grab whatever config files the end user wants to be restored and packs them up into a tarball.
3. Send a message, via ubus, to procd, to initiate the upgrade.
4. procd does some stuff which I haven't finished completely understanding just yet, but it looks like firmware verification to make sure we don't upgrade to a bad firmware file.
5. Notably, procd does not terminate any services here.
6. procd replaces itself (via execvp) with the program /sbin/upgraded. Service management is no longer possible.
7. /sbin/upgraded now acts as PID1. It executes the shell script /lib/upgrade/stage2 with parameters. The remaining sequence runs from this shell script.
8. Terminate (SIGKILL) all telnet, ash, and dropbear processes.
9. Loop over all remaining processes, sending them the TERM signal.
10. Loop over all remaining processes, sending them the KILL signal.
11. Create a new RAM filesystem, mount it, and copy over a very small set of binaries into it.
12. Change root into the new RAM filesystem.
13. Remount /overlay read-only, and lazy-unmount it.
14. Write the upgraded firmware to disk. (This can include platform/device-specific upgrade steps in platform_do_upgrade, handling mtd or partitioning details.)
15. Write the saved configuration (if any) to disk. (This is a platform-specific step, via platform_copy_config.)
16. Unmount any remaining filesystems.
17. Reboot.

There are plenty of potential deficiencies in this process, among them:

• Hardcodes a list of “potentially-interfering” / interactive processes (ash, telnet, dropbear) to kill first; this is not exhaustive or up-to-date (e.g., telnet is no longer in the base install; openssh is not handled).
• Does not give processes much time in between TERM and KILL signals.
• Does not utilise procd to tear down services.
• Susceptible to fork bombs.
• Easily broken by open files on storage devices (e.g., for swap, or explicitly opened), as these can prevent unmounting /overlay.
• Does not handle errors (e.g., I/O) in writing the disk image, potentially leaving a partially-upgraded system upon reboot.

Many of these deficiencies are historical artifacts, remaining simply because no one has fixed them.

Thanks to Michael Jones for writing most of this down on the mailing list [OpenWrt-Devel] Sysupgrade and Failed to kill all processes.