[scummvm-devel] ATTN PORTERS: New Buildbot ahead

Fri Mar 19 14:23:02 UTC 2021

Amazing work! Thank you for doing this!

On Fri, Mar 19, 2021 at 8:20 AM Philippe Valembois via scummvm-devel <
scummvm-devel at lists.scummvm.org> wrote:

> Hi everyone,
>
> For the last year I spent most of my time on modernizing our current
> Buildbot server.
> My work is available on my Github account:
> https://github.com/lephilousophe/dockerized-bb/
> Obviously, I will migrate it to ScummVM organization when admins decide to.
>
> This repository includes a massive overhaul of the Buildbot
> architecture, the master configuration and the toolchains.
> As of today, I think the new Buildbot is quite ready and could replace
> the old one.
> For those interested, it is available for testing at
> https://al.scummvm.net/buildbot/
>
> I currently need porters to test the snapshots produced by this new
> Buildbot. It's difficult for me to ensure that every port still works
> as I don't own any console nor Apple products. :D
> I tested some platforms (Linux based ones with QEmu, emulators for
> others) but more intensive tests on real environments would be better.
> So, please grab your favorite environment, and report your test
> results. Hopefully we'll quickly fix any issues you may encounter.
> Drop me an email, this will let me keep track of which platform is
> working or which one has bugs.
>
> Hereafter is a mostly exhaustive introduction of the main changes.
> Porters can skip parts by looking at their specific platforms only.
>
> == Architecture ==
>
> The old Buildbot consists of a master process and a worker process
> running in the same environment.
> Therefore, an upgrade of the whole server to use a newer Debian could
> break all toolchains.
>
> The new Buildbot has a master process and spawns a Docker container
> which contains the worker and the toolchain used to build ScummVM.
> This means that toolchains are decoupled from the master and can be
> upgraded individually when time is convenient.
>
> == Master configuration ==
>
> Most of the changes are related to the migration to the latest
> buildbot version (version 3.0, ready for future!).
> In addition, I tried to split the configuration file into smaller
> files which can be browsed more easily. For example, the platforms are
> defined in a dedicated file and builds in another one.
> I took advantage of this migration to improve usability: I added a new
> IRC bot provided by Buildbot, and a brand new Discord webhook which
> tries to present failures and successes in a clean way. It should be
> easy to see if all builds are successful or not.
> Besides, after a successful nightly build, the package generated is
> linked directly in the build step. This let a developer grab specific
> nightly snapshots for bisecting tests.
> On a side note, snapshot list is now dynamically built and adding a
> platform doesn't require to run a script on the server anymore (that
> was mainly due to the fact the new Buildbot version doesn't allow to
> serve static files anymore).
> The cleaning step got the same treatment and is executed inside
> Buildbot after every nightly build. For now, only the 14 last
> snapshots are kept and snapshots older than 1 month are destroyed.
>
> == Toolchains ==
>
> The toolchains side is where I put the most effort. I tried to update
> them as much as I could. This implied sometimes upgrading components
> above what original (and abandoned) toolchains provided.
>
> A vast majority of platforms are split in two parts: a toolchain image
> and a worker image.
> Toolchain one is where all components got generated and could be used
> by anyone wanting to compile ScummVM on the platform using a shell
> (docker run -it toolchains/<toolchain>).
> Worker one is dedicated to Buildbot. It automatically spawns a
> buildbot worker ready to make use of the toolchain.
>
> Two objectives of the overhaul were to ease building and upgrade of a
> toolchain, and make them available to everyone.
> When it was possible, reproducibility was targeted.
> It's also possible to pull images from Docker hub:
> https://hub.docker.com/r/lephilousophe/scummvm
>
> Please note that Apple based toolchains are not publicly available
> because the SDK isn't freely redistributable. If you want them, you
> will need to create an Apple developer account, download Xcode and
> build images.
>
> Toolchains, as of today, are all based on the current Debian release.
> Some toolchains source code had to be patched to let them compile with
> recent Debian tools.
>
> Each image is built using a Dockerfile preprocessed by GNU M4. The
> language is quite... outdated but it fits well and I think recipes are
> quite clear without too much repeat.
>
> Now I will try to list changes on every toolchain compared to current
> Buildbot.
> First the bad news, during this process I had to drop two platforms:
>
> - ios(< 7): I am not sure how this one was built and how easy it would
> be to update it. Plus, these iPhone are now quite old. If someone can
> explain to me how to create one, I could add it back.
>
> - n64: the port is unmaintained (last release 1.6.0) and the main
> library (libn64) isn't available anymore except on Web archive.
>
> Now, the new platforms:
> - macosx-arm64: this is the new architecture from Apple.
> There is still a bug with ldid which is supposed to fake sign package
> on Linux (equivalent of codesign on Apple which is not open source).
> Builds must be un-quarantined before launching them.
>
> - raspberrypi: this toolchain aims at supporting all RPi generations
> using Raspbian.
>
> - riscos: thanks to @ccawley2011 there is now a RiscOS platform on the
> buildbot.
>
> Finally, the updated platforms:
> - AmigaOS4:
>   - it's based on https://github.com/sba1/adtools
>   - it has a GCC 9.1 (in place of 8.3)
>   - dependencies are downloaded directly from OS4Depot so they should
> get updates when the image is rebuilt
>   - ARexx is used to execute REXX scripts and some shims have been
> developed for AmigaOS specific commands (this let packaging work)
>
> - Android : nothing really changed as the current toolchain is quite up to
> date.
>
> - Caanoo :
>   - original toolchain from GamePark was available on binary form only
> on Web archive
>   - it has been recreated from scratch using Crosstool-NG
>   - a new GCC (8.3.0) is used with the original glibc and kernel headers
>   - versions of libraries present in toolchain have been matched so
> they are used by binary
>   - static libraries are built when they were missing
>
> - Nintendo :
>   - all Nintendo consoles use up-to-date DevkitPro binary builds
>   - it was just a matter of adding missing libraries
>
> - Dreamcast :
>   - I used the same GCC version as the one chosen by KallistiOS for
> testing (ie. GCC 9.3.0 instead of GCC 4.6.3)
>   - libronin (which seems abandoned) has been patched to support it
>
> - GCW0 :
>   - original toolchain is based on buildroot fork available on Github
>   - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.9.1)
>   - all ScummVM dependencies have been migrated from buildroot to have
> a matching ABIs (like in Caanoo).
>
> - iPhone (ios7) :
>   - based on cctools-port from
> https://github.com/tpoechtrager/cctools-port is used
>   - LLVM 11 is used for compilation (instead of 3.7.1)
>   - Migration to arm64 (instead of armv7 as it isn’t supported on
> recent iOS anymore)
>
> - MacOSX (x86 and x86_64)
>   - based on OSXCross from https://github.com/tpoechtrager/osxcross/
>   - LLVM 11 is used for compilation (instead of LLVM 7.0.1 and GCC 4.2)
>   - sadly, for x86, the libstdc++ used is the original one which
> misses several C++11 features. It seems we could build a newer
> libstdc++ to get C++11 support (TODO).
>
> - Windows x86 and x86_64
>   - based on MXE (https://mxe.cc/)
>   - GCC 5.5 is used but there seems to be experimental support for
> newer GCC (to be tested).
>   - versions of libraries are the one provided by MXE
>
> - GP2X
>   - old GamePark console
>   - uses unmaintained Open2x toolchain source from
> https://sourceforge.net/projects/open2x/
>   - locked to this toolchain because the whole console Linux is based
> on now obsolete ARM OABI
>
> - OpenDingux
>   - based on original buildroot fork found at
> https://github.com/mthuurne/opendingux-buildroot
>   - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.4.2)
>   - all ScummVM dependencies have been migrated from buildroot to have
> a matching ABIs (like in Caanoo).
>
> - OpenPandora
>   - based on official toolchain bits at https://github.com/openpandora
> and Angstrom, Yocto, ...
>   - Crosstool-NG is used to get a newer GCC (8.3.0 instead of 4.3.3)
>   - all ScummVM dependencies have been migrated from toolchain to have
> a matching ABIs (like in Caanoo).
>
> - PS3
>   - using up to date toolchain from https://github.com/ps3dev
>   - libSDL is from bgK's Github
>
> - PSP
>   - using up to date toolchain from https://github.com/pspdev
>
> - Vita
>   - using up to date toolchain from https://github.com/vitasdk
>
> Obviously, the Debian platform is still present in x86, x86_64 and
> clang flavours. This is based on the latest Debian stable.
> Special flavours which compile using plugins have been added to spot
> specific failures.
> C++11 flavours have been dropped as we use it by default everywhere now.
> No toolchain image is specifically built, as it's easy to have a one
> (hello APT),
>
> If you think your platform could fit well on the buildbot, don't
> hesitate: we can bring it to Buildbot together. The main prerequisite
> is that we can cross-compile from Linux.
>
> As a side note, there is a check-versions script which helps to track
> all versions used in the toolchains. It searches for new versions of
> toolchains and libraries used in the Buildbot. That should make the
> task of updating more easy.
>
> Regards,
> Phil (@lephilousophe on Discord or IRC)
> _______________________________________________
> scummvm-devel mailing list
> scummvm-devel at lists.scummvm.org
> https://lists.scummvm.org/listinfo/scummvm-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scummvm.org/pipermail/scummvm-devel/attachments/20210319/5e066609/attachment-0001.htm>