[Scummvm-devel] The new Buildbot: what’s happening, what needs to happen

timofonic timofonic timofonic at gmail.com
Wed Jan 3 00:12:45 CET 2018


Thanks a lot, Snover. I want to test newer ScummVM releases again,
this will be useful for users in the bleeding edge too ;)

What's the status of the Android port? And the buildbot+Docker to
compile it? If there's lack of manpower for this platform, I recommend
to make an announcement in communities such as XDA.

It's sad to see certain platforms to disappear, but the lack of a
maintainer is enough reason to happen. I'm happy to see the Dreamcast
port is alive and kicking, it seems it's using an updated GCC version
too!

About OS/2:
Have you contacted with the OS/2 community?
- There's an ongoing OS/2 resurrection thanks to ArcaOS: The
eComStation sucessor, from Russia with love.
- There's onging crowfunding campaigns to port software, like one for
a more modern web browser (they talk about Chromium, but I would
prefer Firefox Nightly): http://articles.os2voice.org
- There's new osFree activity these days by Yuri Prokushev, a Russian
developer You can look it at the svn2github mirror, better than the
nasty SF interface: https://github.com/svn2github/osFree
- What about contacting ArcaOS company? Maybe they would donate a
license to the project that can run in a Docker container. Maybe they
could help with the cross-compiler setup. Maybe they can donate ArcaOS
licenses and help with the cross-compiling. After all, who doesn't
want to have ScummVM to run in their Operating System? ScummVM has the
fame of being portable even to dishwashers, so it shameful that an
alive Operating System doesn't have an updated ScummVM port.
. If you go the totally absurd, insane, crazy and ridiculously funny
way about OS/2 porting and cross-compiling: Jason Stevens (Neozeed)
from VirtuallyFun.com (I added him as CC, maybe he can provide some
info or ask some OS/2 dev interested to help) ported OS/2 2.0 build
environment to Windows x64, maybe it could be ported to Linux too. And
maybe someone can convince this guy to port ScummVM to OS/2 2.0, it
can be very funny to see it happening.
https://virtuallyfun.com/category/os2/ (NOTE: This part of the message
is a joke. I don't think there's someone crazy enough to port newest
ScummVM versions to an ancient OS/2 version and automatize the process
over Docker and buildbot!).

Kind regards.


2017-12-17 4:08 GMT+01:00 Colin Snover <scummvm-devel at zetafleet.com>:
> Team,
>
> One of my goals for the next release of ScummVM is to automate the
> process of building and generating release packages. My hope is that
> this will allow us to have a faster release cycle, more consistent
> release quality, and free up manpower to focus on the many parts of
> ScummVM which cannot be so easily automated (like adding more engines! :)).
>
> The first two major parts of this project are nearly complete: the
> upgrade of Buildbot, and the introduction of infrastructure management
> using Ansible.
>
> Here’s an overview of these changes.
>
> Buildbot:
>
> * Each compiler is now in its own Docker container and its mechanism of
> generation is well-known and recorded in Git
> * Buildbot workers can now be securely distributed across servers
> * Obsolete workers in the old Buildbot have been replaced by new workers
> in the new Buildbot
> * All compilers have been updated; the oldest is now GCC 4.9.1
> * Buildbot has been updated, it is now using Buildbot 0.9
> * New packages are generated immediately upon successful build, instead
> of only once nightly
> * Login for manually controlling builds is integrated with GitHub
> * Manual builds using different configure flags can be triggered from
> the admin panel
> * Failed workers run first on the next run, instead of whenever Buildbot
> feels like it
> * Debug symbols are split into separate packages when using the default
> packager
> * New build downloads are linked directly from the Buildbot interface
> * [Policy change] Pending approval, porters will be responsible for
> maintaining their own containers moving forward
> * [Policy change] Pending approval, ports in the master branch will need
> to have a Buildbot worker
> * [Request for access] I need consent and admin access to the GitHub org
> to transfer the new Buildbot repository to the org
> * [Request for access] I need consent and admin access to the GitHub org
> to set up OAuth for the new Buildbot, or someone to create and send me
> tokens
> * [Request for access] When it is time to deploy, I would be like to
> have access to, or have someone available with access to, vm2’s
> management panel so that if there is a big explosion (unlikely, but
> non-zero since the kernel needs updating) I can boot into recovery mode
> and fix it
>
> Infrastructure:
>
> * Installation, configuration, and management of new server software
> (e.g. new Buildbot) and users is now recorded in Ansible playbooks in Git
> * Server timezones change to UTC to follow international standards
> * Firewalls are a thing now
> * User management is a thing now, using keys pulled automatically from
> GitHub
> * [Policy change] Pending approval, server changes will be managed
> through Ansible instead of directly installing and modifying software on
> servers going forward
> * [Request for access] I need consent and admin access to the GitHub org
> to transfer the Ansible repository to the org
> * [Request for information] vm.scummvm.org needs to have its final
> remaining services evaluated and moved away (subversion, planet, and
> doxygen), and then needs to be wiped and reloaded with an up-to-date
> 64-bit OS. Who can do this?
>
> Please send any feedback about the questions and policy changes above,
> after reading this entire email since some questions may be addressed by
> the extra information below.
>
> The remaining timeline for this work is:
>
> 1. Run full builds of ScummVM on all of the workers and address any
> remaining compilation failures (I’ve fixed most of these already);
> 2. Once everything is building, a PR for all the necessary build patches
> will be created;
> 3. Make any other necessary tweaks to the deployment configuration in
> Ansible so everything critical will continue to be backed up appropriately;
> 4. Back up the old Buildbot stuff somewhere else in case everything
> explodes horribly and needs to be rolled back;
> 5. Once the PR has landed, and any outstanding access issues or other
> concerns have been resolved, deploy the new Buildbot.
>
> Beyond this point, I hope that other people will take over and do what’s
> needed so that we can do things like create proper packages/installers
> for the various platforms (currently only the Windows worker generates
> proper installers), code sign the releases properly, publish to app
> stores using e.g. fastlane, etc.. This has been a big distraction for me
> for the last month or more and I just want to get back to working on
> ScummVM itself.
>
> Here’s more information on these changes:
>
> The Buildbot upgrade
> --------------------
>
> Currently, Buildbot consists of a big lump of binaries built in an
> unknown manner and which must all run against the same host system. This
> is not maintainable nor upgradable, and it prevents the host system from
> being updated without running the risk of breaking Buildbot completely.
>
> The new Buildbot is containerised using Docker, so each platform has its
> own separate environment which can be managed and upgraded independently
> from the host system and from the other workers. The steps used to
> create each environment are written in Dockerfiles, which are saved in a
> Git repository, so it is never a mystery how the compilers and libraries
> were generated, and newer images can be regenerated from this source
> information to incorporate new library or compiler updates. Workers can
> be easily moved around to different servers as necessary in order to
> balance load and disk space, and developers can trivially create copies
> of part or all of the entire Buildbot system to run locally.
>
> (Shameless plug: “<rsn8887> the new buildbot is awesome. I can built
> *ANY* binary for *ANY* system using the same commands without any hassle”)
>
> The new Buildbot was used to create ScummVM 2.0 for Windows and is
> successfully generating binaries[1] for every active port listed on our
> Platforms page in the wiki.
>
> Shortly, I will be opening a pull request containing various patches
> related to this work. Some of these patches will break the current
> Buildbot, and users still building certain ports using older toolchains
> and libraries. As such, Buildbot will be out of commission for a short
> time so that the old Buildbot data may be archived away in case we need
> to roll back for some reason, and the new Buildbot installed in its place.
>
> An ongoing problem with the current Buildbot has been disk space
> exhaustion, and while this is in the process of being resolved by _sev
> and our generous hosting provider, it is possible that some workers will
> not be enabled initially in order to ensure there is enough available
> space on the main server over the near term. The hope is that
> vm.scummvm.org can be paved over with a fresh OS and then will be able
> to receive any remaining workers, which will also reduce the amount of
> time it takes to run builds.
>
> The loadout of ports built on the new Buildbot has changed to reflect
> the up-to-date list of actively maintained ports on the wiki. These are
> the changes:
>
> Added          | Removed        | Updated
> -------------- | -------------- | --------------
> FreeMiNT (GCC  | Android-MIPS   | AmigaOS (5.4)
>  7.2)          | Android-x86    | Android-ARM (NDK r15c / Clang 5)
> Haiku (5.4)    | Dingux         | Debian (6.3)
> Maemo (5.1)    | DS             | Dreamcast (7.2)
> Raspberry Pi   | GameCube       | GCW0 (4.9.1)
>  (6.3)         | GP2X           | Haiku (5.4)
>                | GP2XWiz        | iOS 7+ (Clang 3.8.1)
>                | iOS 3-6        | macOS (Clang 3.8.1)
>                | N64            | PS3 (7.2)
>                | OpenPandora    | PSP (4.9.3)
>                | Ouya           | Vita (7.2)
>                | PS2            | Windows (6.3)
>                | WebOS
>
> I did try to add SamsungTV and OS/2 workers, since these are listed as
> active ports, and was unsuccessful. For SamsungTV, I was not able to
> create a working compilation of glibc for its ancient kernel. For OS/2,
> I could not find any cross-compiler, although there are apparently still
> up-to-date GCCs being produced for that platform.
>
> Once the new Buildbot up and running, I would like to make it a
> requirement for all ports in the master branch to have a Buildbot worker
> image maintained by the porter for that platform. This will save
> everyone time over the long run and improve the experience for our users
> since the release process will be automated and it will free core
> ScummVM team members from being wholly responsible for maintaining ports
> infrastructure in addition to everything else they already need to do.
>
> The repository for this work is currently at
> <https://github.com/csnover/scummvm-buildbot>.
>
> Infrastructure management
> -------------------------
>
> As with the old Buildbot, our server management process is currently a
> mystery process where changes are made directly on the servers without
> any way to create new deployments or record change histories.
>
> In figuring out how to deploy the new Buildbot to ScummVM servers, bgK
> mentioned having used Ansible to perform deployments at work, and others
> I know who work in sysops shared their positive experiences managing
> systems using this tool.
>
> Over the last week, I created an initial deployment plan for the new
> Buildbot using Ansible which I hope can be used (along with Docker) for
> all service deployments moving forward in order to make maintaining our
> server infrastructure easier, safer, and more transparent. As Docker
> allows anyone to create an identical copy of any of our application
> environments locally, Ansible allows anyone to create an identical[2]
> copy of our host server environments by running Ansible against any
> other machine running the same base OS.
>
> This is a less user-facing change than the Buildbot change and mostly
> only affects anyone responsible for maintaining our services, though
> will probably also result in some short downtime for the main site as
> some badly deferred server updates need to be applied, such as a kernel
> update to prevent service failures when restarting containers. As such,
> since we are near release time, I do not plan on deploying these changes
> until at least a few weeks after the release to avoid unnecessary
> downtime at this critical moment.
>
> Since I needed to set up VM environments for testing the deployment
> anyway, I also introduced some playbooks for securing the servers, so
> ingress firewalls will now exist, and user accounts for team members who
> are no longer team members or do not need access to maintain server
> infrastructure will not exist.
>
> At this point I’ve deployed the site playbook to some fresh VMs, and
> everything seems to be working correctly, though there is also the
> chance that some additional cleaning up will need to be done on the
> actual production server since none of the current services have been
> copied over into playbooks except for bits and pieces of what was needed
> for the new Buildbot. Any issues with this can hopefully be addressed on
> an ad-hoc basis and then services can be put into playbooks to solve
> this problem for deployments to whatever different servers may exist in
> the future.
>
> The repository for this work is currently at
> <https://github.com/csnover/scummvm-infrastructure>.
>
> ---
>
> [1] Since I don’t have actual hardware to test all the binaries, I can
> only say that the Debian, macOS, Maemo, PSP, Raspberry Pi, Vita, and
> Windows binaries have been verified as working. Testing (and probably
> tweaking) will be needed for the remaining ports.
>
> [2] Obviously, our security keys are encrypted so are not available for
> any person to install. I don’t necessarily expect this repository to
> remain public since there isn’t much reason for the general public to
> duplicate our infrastructure, it’s mostly for allowing current and
> future server maintainers the ability to easily set up a local testing
> environment and for making it unburdensome to deploy our services to new
> servers in the future.
>
> That’s all from me for now. Looking forward to your feedback. Thanks for
> reading this long, long email!
>
> --
> Colin Snover
> https://zetafleet.com
>
>
> _______________________________________________
> Scummvm-devel mailing list
> Scummvm-devel at lists.scummvm.org
> http://lists.scummvm.org/listinfo/scummvm-devel



More information about the Scummvm-devel mailing list