| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
| infrastructure [2020/06/17 20:43] – Detail buildbot machines zorun | infrastructure [2024/06/01 08:40] – [Buildbot workers] update with latest state ynezz |
|---|
| ====== OpenWrt/LEDE Project infrastructure ====== | ====== OpenWrt project infrastructure ====== |
| The OpenWrt/LEDE Project relies on the considerable efforts of its volunteer developers, the core documentation team, the forum moderators, and the host of people who're actively using and testing the products. | The OpenWrt project relies on the considerable efforts of its volunteer developers, the core documentation team, the forum moderators, and the host of people who're actively using and testing the products. |
| The OpenWrt/LEDE team has also provisioned many physical and virtual servers that "make the OpenWrt (and former LEDE) Project go." | The OpenWrt team has also provisioned many physical and virtual servers that "make the OpenWrt project go". |
| |
| ===== Services ===== | ===== Services ===== |
| |
| ==== Internally-managed services ==== | ==== Internally-managed services ==== |
| |
| We run the following services internally to support the OpenWrt project: | We run the following services internally to support the OpenWrt project: |
| |
| ^ Service ^ Technology ^ Description ^ Server ^ Admin(s) ^ | ^ Service ^ Technology ^ Description ^ Server ^ Statistics ^ Admin(s) ^ |
| | [[:start|OpenWrt website]] and [[:docs:start|documentation]] | Dokuwiki | Wiki with open registration | DigitalOcean droplet | | | | [[:start|Website and documentation]] | [[doku>|Dokuwiki]] | Primary OpenWrt website and documentation/wiki | DigitalOcean droplet | [[https://openwrt.org/stats/|awstats]] | bobafetthotmail, [[developers:jow|jow]], [[developers:thess|thess]], [[user:tmomas|tmomas]], [[developers:zorun|zorun]] | |
| | [[https://forum.openwrt.org/|OpenWrt Forum]] | Discourse | Provides a place for people to ask and answer questions about using OpenWrt, developing it, and other topics | DigitalOcean droplet | | | | [[https://forum.openwrt.org/|Forum]] | [[https://www.discourse.org/|Discourse]] | Provides a place for people to ask and answer questions about using OpenWrt, developing it, and other topics | DigitalOcean droplet | [[https://forum.openwrt.org/stats/|awstats]] | [[developers:hauke|hauke]], [[developers:jow|jow]], [[developers:thess|thess]], [[user:tmomas|tmomas]] | |
| | [[https://git.openwrt.org/|Git server]] | [[https://git.wiki.kernel.org/index.php/Gitweb|gitweb]] | Primary OpenWrt source code repositories | DigitalOcean droplet | | | | [[https://git.openwrt.org/|Git server]] | [[https://git.wiki.kernel.org/index.php/Gitweb|gitweb]] | Primary OpenWrt source code repositories | DigitalOcean droplet | [[https://github.com/openwrt/openwrt/graphs/contributors|gitstats]] | [[developers:jow|jow]], [[developers:thess|thess]] | |
| | [[https://bugs.openwrt.org/|Bug tracker]] | Flyspray | The place where bugs and feature requests are tracked | DigitalOcean droplet | | | | [[https://bugs.openwrt.org/|Bug tracker]] | [[https://www.flyspray.org/|Flyspray]] | The place where bugs and feature requests are tracked | DigitalOcean droplet | | [[developers:jow|jow]], [[developers:thess|thess]] | |
| | Email | ? | Handles incoming emails to @openwrt.org domain | DigitalOcean droplet | | | | Email | ? | Handles incoming emails to @openwrt.org domain | DigitalOcean droplet | | [[developers:jow|jow]], [[developers:thess|thess]], [[developers:stintel|stintel]] | |
| | [[https://buildbot.openwrt.org|Buildbot master]] | [[https://buildbot.net/|Buildbot]] | Coordinates builders | Hetzner | | | | [[https://buildbot.openwrt.org|Buildbot master]] | [[https://buildbot.net/|Buildbot]] | Coordinates builders | Hetzner | | [[developers:jow|jow]], [[developers:ynezz|ynezz]] | |
| | [[:downloads|Download site]] | | Hosts all current and released images, SDK, Imagebuilder | Hetzner | | | | [[https://downloads.openwrt.org/|Downloads]] | | Hosts all current and released images, SDK, Imagebuilder | Hetzner | [[https://downloads.openwrt.org/stats/|awstats]] | [[developers:jow|jow]], [[developers:thess|thess]] | |
| | [[https://sources.openwrt.org/|Sources archive]] | | Hosts a copy of sources of all software used in OpenWrt | Hetzner | | | | [[https://archive.openwrt.org|Downloads archive]] | | Hosts old OpenWrt images and releases | private co-lo | | [[developers:wigyori|wigyori]] | |
| | [[https://archive.openwrt.org|Images archive]] | | Hosts old OpenWrt images and releases | ? | | | | [[https://oldwiki.archive.openwrt.org|Old wiki archive]] | | Old OpenWrt wiki with legacy / non-migrated content | private co-lo | | [[developers:wigyori|wigyori]] | |
| | | [[https://sources.openwrt.org/|Sources archive]] | | Hosts a copy of sources of all software used in OpenWrt | Hetzner | | [[developers:jow|jow]], [[developers:thess|thess]] | |
| | | [[https://freescout.openwrt.org|Freescout]] | [[https://freescout.net|Freescout]] | Facilitates handling of shared mailboxes (testing) | DigitalOcean droplet | | [[developers:stintel|stintel]] | |
| | | [[https://lists.openwrt.org/mailman/listinfo|Mailing lists]] | Mailman | Main medium of communication for developers and admin | infradead | | [[developers:dwmw2|dwmw2]] | |
| | | [[https://firmware-selector.openwrt.org/|Firmware selector]] | | Search and download OpenWrt firmware for your device | | | | |
| | | [[https://sysupgrade.openwrt.org/|Sysupgrade server]] | | Online ImageBuilder and upgrade server | | [[https://sysupgrade.openwrt.org/stats|stats]] | [[developers:aparcar|aparcar]] | |
| |
| ==== Externally-managed services ==== | ==== Externally-managed services ==== |
| |
| We also rely on the following services that are run by external entities: | We also rely on the following services that are run by external entities: |
| |
| ^ Service ^ Technology ^ Description ^ Host ^ Liaison ^ | ^ Service ^ Technology ^ Description ^ Host ^ Liaison ^ |
| | [[https://downloads.cdn.openwrt.org/|Download CDN]] | CDN | Alternate download location for images | [[https://www.keycdn.com/|KeyCDN]] | | | | [[https://patchwork.ozlabs.org/project/openwrt/list/|Patch tracking]] | [[http://jk.ozlabs.org/projects/patchwork/|Patchwork]] | Used to track patches sent to the openwrt-devel mailing list | Ozlab | | |
| | [[https://sources.cdn.openwrt.org|Sources CDN]] | CDN | Alternate download location for sources | [[https://www.keycdn.com/|KeyCDN]] | | | | [[https://github.com/openwrt/openwrt|Github mirror]] | Github | Mirror of OpenWrt repository. Pull-requests are accepted. | Github | | |
| | [[https://lists.openwrt.org/mailman/listinfo/openwrt-devel|openwrt-devel mailing list]] | Mailman | Main medium of communication for developers | infradead | | | | CDN | Fastly | Content delivery network | {{:fastlylogo-red-png.png?200&nolink}} | aparcar | |
| | [[https://lists.openwrt.org/mailman/listinfo/openwrt-adm|openwrt-adm mailing list]] | Mailman | Main medium of communication for administrative matters | infradead | | | |
| | [[https://patchwork.ozlabs.org/project/openwrt/list/|Patch tracking]] | [[http://jk.ozlabs.org/projects/patchwork/|Patchwork]] | Used to track patches sent to the openwrt-devel mailing list | Ozlab | | | |
| | [[https://github.com/openwrt/openwrt|Github mirror]] | Github | Mirror of OpenWrt repository. Pull-requests are accepted. | Github | | | |
| | |
| | |
| ===== Servers ===== | |
| | |
| All services that are managed internally are run on servers devoted to the OpenWrt/LEDE Project. | |
| | |
| ^ Host ^ Servers ^ Services ^ Money ^ Admin(s) ^ | |
| | Hetzner | 3x [[https://www.hetzner.com/dedicated-rootserver/matrix-ex|ex40]] | buildbot master, downloads | Paid privately | | | |
| | DigitalOcean | Many droplets | wiki, forum, git, mails | Free (open source project support) | | | |
| | OSUOSL | OpenStack VMs | | Free (open source project support) | bjonglez | | |
| |
| ===== Buildbot ===== | ===== Hosting infrastructure and costs ===== |
| | ==== Servers ==== |
| | All services that are managed internally are run on servers devoted to the OpenWrt project, except for mailing lists which run on Infradead servers. |
| |
| ==== Build matrix ==== | ^ Host ^ Servers ^ Services ^ Admin(s) ^ Expected cost ^ Actual cost ^ Comment ^ |
| | | {{:netcup_logo_rgb_color_vorschau.png?400|}} | [[https://www.netcup.de/bestellen/produkt.php?produkt=2898|RS 8000 G9.5 a1 12M]] + 6TB storage | Downloads | [[developers:aparcar|aparcar]] | 100€ / month | Free | Open source project support | |
| | | [[https://www.digitalocean.com/?utm_medium=opensource&utm_source=OpenWrt|{{:do_powered_by_badge_blue.png?400|}}]] | 7 droplets (prod) | wiki, forum, git, mails... | [[developers:jow|jow]], [[developers:ynezz|ynezz]], [[developers:stintel|stintel]] | 344.00 $ / month | Free | Open source project support | |
| | | Microsoft | 2 VMs | Buildbot workers | [[developers:ynezz|ynezz]] | 10.000 $ / year | Free | Open source project support | |
| | | Hetzner | [[https://www.hetzner.com/dedicated-rootserver/matrix-ex|ex40]] | Buildbot master | [[developers:nbd|nbd]], ? | 50 € / month | 50 € / month | Paid privately | |
| | | Hetzner | [[https://www.hetzner.com/sb/|sb59]] | Archive | [[developers:blogic|blogic]] | 49.50 € / month | 49.50 € / month | Paid privately | |
| | | Hetzner | [[https://www.hetzner.com/dedicated-rootserver/matrix-ax|ax61-nvme]] + 2x 1.92 TB SSD | Downloads | [[developers:jow|jow]] | 158.27 € / month | 158.27 € / month | Paid privately | |
| |
| There are several buildbot instances: one for master, and one for each release. In addition, there is phase1 to build images, and phase2 to build packages. | ==== Buildbot workers ==== |
| | Buildbot workers detect updates to the source and kick off the build process for each supported device. |
| |
| ^ Version ^ Phase 1 - Images ^ Phase 2 - Packages ^ Fail logs ^ Image downloads ^ | ^ Host ^ Name ^ Server spec ^ Nb servers ^ Admin(s) ^ Cost ^ Comment ^ |
| | master | [[https://buildbot.openwrt.org/master/images/|Phase 1 - Images]] | [[https://buildbot.openwrt.org/master/packages/|Phase 2 - Packages]] | [[https://downloads.openwrt.org/snapshots/faillogs/|Fail logs]] | [[https://downloads.openwrt.org/snapshots/targets/|images]] | | | Hetzner | truecz-dock-01 | Xeon E3/32GB/2TB | 1 | [[developers:ynezz|ynezz]] | 32€ / month | Paid privately | |
| | 19.07 | [[https://buildbot.openwrt.org/openwrt-19.07/images/|19.07 Images]] | [[https://buildbot.openwrt.org/openwrt-19.07/packages/|19.07 Packages]] | [[https://downloads.openwrt.org/releases/faillogs-19.07/|Fail logs]] | [[https://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/|images]] | | | OSUOSL | osuosl-vm-* | [[infrastructure:openstack|OpenStack VM]] (12 cores, 32 GB RAM, 320 GB disk) | 4 | [[developers:zorun|zorun]], [[developers:ynezz|ynezz]], [[developers:jow|jow]] | Free | Open source project support | |
| | 18.06 | [[https://buildbot.openwrt.org/openwrt-18.06/images/|18.06 Images]] | [[https://buildbot.openwrt.org/openwrt-18.06/packages/|18.06 Packages]] | [[https://downloads.openwrt.org/releases/faillogs-18.06/|Fail logs]] | [[https://downloads.openwrt.org/releases/18.06-SNAPSHOT/targets/|images]] | | | OSUOSL | osuosl-dock-* | 12C, 24T, 64 GB RAM (2x Xeon X5680 3.33 GHz) | 2 | OpenWrt sysadmins | Free | Open source project support | |
| | 17.01 | [[http://release-builds.openwrt.org/17.01/images/|17.01 Images]] | [[http://release-builds.openwrt.org/17.01/packages/|17.01 Packages]] | TODO | [[https://downloads.openwrt.org/releases/17.01-SNAPSHOT/targets/|images]] | | | VZFFNRMO | vzffnrmo-dock-* | Xeon E31230/16GB/100GB | 1 | OpenWrt sysadmins | ? | Provided by the [[https://karlsruhe.freifunk.net|Freifunk Karlsruhe Project / VZFFNRMO e.V.]] | |
| | | FFFFM | ffffm-dock-* | 28C, 56T, 448GB RAM, 1.8TB (2x Xeon E5-2680v4 2.4GHz) | 1 | OpenWrt sysadmins | Free | Provided by [[https://ffm.freifunk.net/|Freifunk Frankfurt am Main e.V.]] | |
| |
| ==== Buildbot machines ==== | === Worker hardware requirements === |
| |
| Buildbot machines detect updates to the source and kick off the build process for each supported device. | * **CPU**: a good worker needs 4-8 CPU cores for phase1/images, and 8-16 CPU cores for phase2/packages. Larger machines are split into several workers using Docker. |
| | * **Memory**: each worker needs a minimum of 1 GB of RAM for each CPU hardware thread, 2 GB / thread is preferred. For example, a 16 cores / 32 threads machine needs between 32 GB and 64 GB of RAM in total. |
| | * **Disk:** each worker needs 50 GB of disk space (phase1/images) or 100 GB of disk space (phase2/packages). |
| | ==== Other costs ==== |
| | ^ What ^ Provider ^ Admin(s) ^ Expected cost ^ Actual cost ^ Comment ^ |
| | | Domain names | Tucows | SPI Hostmaster / [[developers:jow|jow]] | 20-40 € / year | Free | Held by Tucows on behalf of SPI | |
| |
| ^ Server ^ Hardware ^ Host ^ Contact ^ | ===== Buildbot build matrix ===== |
| | fsf-dock-* | 4x 16C/32T | gcc compile farm / OSUOSL | bjonglez | | There are several buildbot instances, phase1 to build images, and phase2 to build packages. |
| |
| | ^ Version ^ Phase 1 - Images ^ Phase 2 - Packages ^ Fail logs ^ Image downloads ^ |
| | | master | [[https://buildbot.openwrt.org/images/|Phase 1 - Images]] | [[https://buildbot.openwrt.org/master/packages/|Phase 2 - Packages]] | [[https://downloads.openwrt.org/snapshots/faillogs/|Fail logs]] | [[https://downloads.openwrt.org/snapshots/targets/|images]] | |
| | | 23.05 | [[https://buildbot.openwrt.org/images/#/waterfall?tags=openwrt-23.05|23.05 Images]] | [[https://buildbot.openwrt.org/openwrt-23.05/packages/|23.05 Packages]] | [[https://downloads.openwrt.org/releases/faillogs-23.05/|Fail logs]] | [[https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/|images]] | |
| | | 22.03 | [[https://buildbot.openwrt.org/images/#/waterfall?tags=openwrt-22.03|22.03 Images]] | [[https://buildbot.openwrt.org/openwrt-22.03/packages/|22.03 Packages]] | [[https://downloads.openwrt.org/releases/faillogs-22.03/|Fail logs]] | [[https://downloads.openwrt.org/releases/22.03-SNAPSHOT/targets/|images]] | |
| |
| ===== Finances ===== | ===== Finances ===== |
| |
| ===== Donations ===== | ===== Donations ===== |
| There is a private discussion about possibilities of some sort of crowdfunding to allow the community to pay back the core team for their expenses, and fund the project going forward, but there is no public news yet. | See the [[:donate|Donations]] page. |
| |