[scummvm-devel] ATTN PORTERS: New Buildbot ahead

Philippe Valembois lephilousophe at gmail.com
Fri Mar 19 12:20:28 UTC 2021 

Hi everyone,

For the last year I spent most of my time on modernizing our current
Buildbot server.
My work is available on my Github account:
https://github.com/lephilousophe/dockerized-bb/
Obviously, I will migrate it to ScummVM organization when admins decide to.

This repository includes a massive overhaul of the Buildbot
architecture, the master configuration and the toolchains.
As of today, I think the new Buildbot is quite ready and could replace
the old one.
For those interested, it is available for testing at
https://al.scummvm.net/buildbot/

I currently need porters to test the snapshots produced by this new
Buildbot. It's difficult for me to ensure that every port still works
as I don't own any console nor Apple products. :D
I tested some platforms (Linux based ones with QEmu, emulators for
others) but more intensive tests on real environments would be better.
So, please grab your favorite environment, and report your test
results. Hopefully we'll quickly fix any issues you may encounter.
Drop me an email, this will let me keep track of which platform is
working or which one has bugs.

Hereafter is a mostly exhaustive introduction of the main changes.
Porters can skip parts by looking at their specific platforms only.

== Architecture ==

The old Buildbot consists of a master process and a worker process
running in the same environment.
Therefore, an upgrade of the whole server to use a newer Debian could
break all toolchains.

The new Buildbot has a master process and spawns a Docker container
which contains the worker and the toolchain used to build ScummVM.
This means that toolchains are decoupled from the master and can be
upgraded individually when time is convenient.

== Master configuration ==

Most of the changes are related to the migration to the latest
buildbot version (version 3.0, ready for future!).
In addition, I tried to split the configuration file into smaller
files which can be browsed more easily. For example, the platforms are
defined in a dedicated file and builds in another one.
I took advantage of this migration to improve usability: I added a new
IRC bot provided by Buildbot, and a brand new Discord webhook which
tries to present failures and successes in a clean way. It should be
easy to see if all builds are successful or not.
Besides, after a successful nightly build, the package generated is
linked directly in the build step. This let a developer grab specific
nightly snapshots for bisecting tests.
On a side note, snapshot list is now dynamically built and adding a
platform doesn't require to run a script on the server anymore (that
was mainly due to the fact the new Buildbot version doesn't allow to
serve static files anymore).
The cleaning step got the same treatment and is executed inside
Buildbot after every nightly build. For now, only the 14 last
snapshots are kept and snapshots older than 1 month are destroyed.

== Toolchains ==

The toolchains side is where I put the most effort. I tried to update
them as much as I could. This implied sometimes upgrading components
above what original (and abandoned) toolchains provided.

A vast majority of platforms are split in two parts: a toolchain image
and a worker image.
Toolchain one is where all components got generated and could be used
by anyone wanting to compile ScummVM on the platform using a shell
(docker run -it toolchains/<toolchain>).
Worker one is dedicated to Buildbot. It automatically spawns a
buildbot worker ready to make use of the toolchain.

Two objectives of the overhaul were to ease building and upgrade of a
toolchain, and make them available to everyone.
When it was possible, reproducibility was targeted.
It's also possible to pull images from Docker hub:
https://hub.docker.com/r/lephilousophe/scummvm

Please note that Apple based toolchains are not publicly available
because the SDK isn't freely redistributable. If you want them, you
will need to create an Apple developer account, download Xcode and
build images.

Toolchains, as of today, are all based on the current Debian release.
Some toolchains source code had to be patched to let them compile with
recent Debian tools.

Each image is built using a Dockerfile preprocessed by GNU M4. The
language is quite... outdated but it fits well and I think recipes are
quite clear without too much repeat.

Now I will try to list changes on every toolchain compared to current Buildbot.
First the bad news, during this process I had to drop two platforms:

- ios(< 7): I am not sure how this one was built and how easy it would
be to update it. Plus, these iPhone are now quite old. If someone can
explain to me how to create one, I could add it back.

- n64: the port is unmaintained (last release 1.6.0) and the main
library (libn64) isn't available anymore except on Web archive.

Now, the new platforms:
- macosx-arm64: this is the new architecture from Apple.
There is still a bug with ldid which is supposed to fake sign package
on Linux (equivalent of codesign on Apple which is not open source).
Builds must be un-quarantined before launching them.

- raspberrypi: this toolchain aims at supporting all RPi generations
using Raspbian.

- riscos: thanks to @ccawley2011 there is now a RiscOS platform on the buildbot.

Finally, the updated platforms:
- AmigaOS4:
  - it's based on https://github.com/sba1/adtools
  - it has a GCC 9.1 (in place of 8.3)
  - dependencies are downloaded directly from OS4Depot so they should
get updates when the image is rebuilt
  - ARexx is used to execute REXX scripts and some shims have been
developed for AmigaOS specific commands (this let packaging work)

- Android : nothing really changed as the current toolchain is quite up to date.

- Caanoo :
  - original toolchain from GamePark was available on binary form only
on Web archive
  - it has been recreated from scratch using Crosstool-NG
  - a new GCC (8.3.0) is used with the original glibc and kernel headers
  - versions of libraries present in toolchain have been matched so
they are used by binary
  - static libraries are built when they were missing

- Nintendo :
  - all Nintendo consoles use up-to-date DevkitPro binary builds
  - it was just a matter of adding missing libraries

- Dreamcast :
  - I used the same GCC version as the one chosen by KallistiOS for
testing (ie. GCC 9.3.0 instead of GCC 4.6.3)
  - libronin (which seems abandoned) has been patched to support it

- GCW0 :
  - original toolchain is based on buildroot fork available on Github
  - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.9.1)
  - all ScummVM dependencies have been migrated from buildroot to have
a matching ABIs (like in Caanoo).

- iPhone (ios7) :
  - based on cctools-port from
https://github.com/tpoechtrager/cctools-port is used
  - LLVM 11 is used for compilation (instead of 3.7.1)
  - Migration to arm64 (instead of armv7 as it isn’t supported on
recent iOS anymore)

- MacOSX (x86 and x86_64)
  - based on OSXCross from https://github.com/tpoechtrager/osxcross/
  - LLVM 11 is used for compilation (instead of LLVM 7.0.1 and GCC 4.2)
  - sadly, for x86, the libstdc++ used is the original one which
misses several C++11 features. It seems we could build a newer
libstdc++ to get C++11 support (TODO).

- Windows x86 and x86_64
  - based on MXE (https://mxe.cc/)
  - GCC 5.5 is used but there seems to be experimental support for
newer GCC (to be tested).
  - versions of libraries are the one provided by MXE

- GP2X
  - old GamePark console
  - uses unmaintained Open2x toolchain source from
https://sourceforge.net/projects/open2x/
  - locked to this toolchain because the whole console Linux is based
on now obsolete ARM OABI

- OpenDingux
  - based on original buildroot fork found at
https://github.com/mthuurne/opendingux-buildroot
  - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.4.2)
  - all ScummVM dependencies have been migrated from buildroot to have
a matching ABIs (like in Caanoo).

- OpenPandora
  - based on official toolchain bits at https://github.com/openpandora
and Angstrom, Yocto, ...
  - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.3.3)
  - all ScummVM dependencies have been migrated from toolchain to have
a matching ABIs (like in Caanoo).

- PS3
  - using up to date toolchain from https://github.com/ps3dev
  - libSDL is from bgK's Github

- PSP
  - using up to date toolchain from https://github.com/pspdev

- Vita
  - using up to date toolchain from https://github.com/vitasdk

Obviously, the Debian platform is still present in x86, x86_64 and
clang flavours. This is based on the latest Debian stable.
Special flavours which compile using plugins have been added to spot
specific failures.
C++11 flavours have been dropped as we use it by default everywhere now.
No toolchain image is specifically built, as it's easy to have a one
(hello APT),

If you think your platform could fit well on the buildbot, don't
hesitate: we can bring it to Buildbot together. The main prerequisite
is that we can cross-compile from Linux.

As a side note, there is a check-versions script which helps to track
all versions used in the toolchains. It searches for new versions of
toolchains and libraries used in the Buildbot. That should make the
task of updating more easy.

Regards,
Phil (@lephilousophe on Discord or IRC)

More information about the scummvm-devel mailing list