[Scummvm-devel] The new Buildbot: what’s happening, what needs to happen

Colin Snover scummvm-devel at zetafleet.com
Sat Jan 6 03:59:33 CET 2018


Hi timofonic,

Thanks for your feedback! I’m glad to hear that you feel like this new
Buildbot will be useful. Here are my answers for your questions:

The Android port has no clear maintainer at the moment. I’d ended up
cobbling together a build for 2.0 since this is our third most popular
platform and there wasn’t one made at release time. Over the holidays
tsoliman was looking into how to fix this situation so that the Android
port is more well-maintained and gives a better user experience, and
then a few days ago lubomyr (whom I do not know, and have not spoken
with, but who has sent some things in the past) opened a pull request
for updating the Android-SDL port. I have failed to follow up on that PR
from lubomyr since I’m still working out what to say and have been
focusing on some other things which are taking up a lot of my time. I
hope to be able to follow up soon. tsoliman may be able to provide more
information.

With regards to OS/2, I don’t know anything about that community, and
because of the limited amount of time I have to spend on these things, I
don’t choose to extend myself in that direction. If OS/2 users want this
port, I am open to someone doing that work to provide an up-to-date
worker, and I’m unfortunately not able to help facilitate that directly.
I can tell you feel passionately about this, so I hope you do not find
this too disappointing.

Thank you for the suggestion regarding XDA, that sounds like it could be
a good approach if nobody is able to get things up and running, and it
wasn’t an idea that I’d had, so I will be keeping it in my mind as time
goes on.

Thank you again for your thoughts.

Best,

On 2018-01-02 17:12, timofonic timofonic wrote:
> Thanks a lot, Snover. I want to test newer ScummVM releases again,
> this will be useful for users in the bleeding edge too ;)
>
> What's the status of the Android port? And the buildbot+Docker to
> compile it? If there's lack of manpower for this platform, I recommend
> to make an announcement in communities such as XDA.
>
> It's sad to see certain platforms to disappear, but the lack of a
> maintainer is enough reason to happen. I'm happy to see the Dreamcast
> port is alive and kicking, it seems it's using an updated GCC version
> too!
>
> About OS/2:
> Have you contacted with the OS/2 community?
> - There's an ongoing OS/2 resurrection thanks to ArcaOS: The
> eComStation sucessor, from Russia with love.
> - There's onging crowfunding campaigns to port software, like one for
> a more modern web browser (they talk about Chromium, but I would
> prefer Firefox Nightly): http://articles.os2voice.org
> - There's new osFree activity these days by Yuri Prokushev, a Russian
> developer You can look it at the svn2github mirror, better than the
> nasty SF interface: https://github.com/svn2github/osFree
> - What about contacting ArcaOS company? Maybe they would donate a
> license to the project that can run in a Docker container. Maybe they
> could help with the cross-compiler setup. Maybe they can donate ArcaOS
> licenses and help with the cross-compiling. After all, who doesn't
> want to have ScummVM to run in their Operating System? ScummVM has the
> fame of being portable even to dishwashers, so it shameful that an
> alive Operating System doesn't have an updated ScummVM port.
> . If you go the totally absurd, insane, crazy and ridiculously funny
> way about OS/2 porting and cross-compiling: Jason Stevens (Neozeed)
> from VirtuallyFun.com (I added him as CC, maybe he can provide some
> info or ask some OS/2 dev interested to help) ported OS/2 2.0 build
> environment to Windows x64, maybe it could be ported to Linux too. And
> maybe someone can convince this guy to port ScummVM to OS/2 2.0, it
> can be very funny to see it happening.
> https://virtuallyfun.com/category/os2/ (NOTE: This part of the message
> is a joke. I don't think there's someone crazy enough to port newest
> ScummVM versions to an ancient OS/2 version and automatize the process
> over Docker and buildbot!).
>
> Kind regards.
>
>
> 2017-12-17 4:08 GMT+01:00 Colin Snover <scummvm-devel at zetafleet.com>:
>> Team,
>>
>> One of my goals for the next release of ScummVM is to automate the
>> process of building and generating release packages. My hope is that
>> this will allow us to have a faster release cycle, more consistent
>> release quality, and free up manpower to focus on the many parts of
>> ScummVM which cannot be so easily automated (like adding more engines! :)).
>>
>> The first two major parts of this project are nearly complete: the
>> upgrade of Buildbot, and the introduction of infrastructure management
>> using Ansible.
>>
>> Here’s an overview of these changes.
>>
>> Buildbot:
>>
>> * Each compiler is now in its own Docker container and its mechanism of
>> generation is well-known and recorded in Git
>> * Buildbot workers can now be securely distributed across servers
>> * Obsolete workers in the old Buildbot have been replaced by new workers
>> in the new Buildbot
>> * All compilers have been updated; the oldest is now GCC 4.9.1
>> * Buildbot has been updated, it is now using Buildbot 0.9
>> * New packages are generated immediately upon successful build, instead
>> of only once nightly
>> * Login for manually controlling builds is integrated with GitHub
>> * Manual builds using different configure flags can be triggered from
>> the admin panel
>> * Failed workers run first on the next run, instead of whenever Buildbot
>> feels like it
>> * Debug symbols are split into separate packages when using the default
>> packager
>> * New build downloads are linked directly from the Buildbot interface
>> * [Policy change] Pending approval, porters will be responsible for
>> maintaining their own containers moving forward
>> * [Policy change] Pending approval, ports in the master branch will need
>> to have a Buildbot worker
>> * [Request for access] I need consent and admin access to the GitHub org
>> to transfer the new Buildbot repository to the org
>> * [Request for access] I need consent and admin access to the GitHub org
>> to set up OAuth for the new Buildbot, or someone to create and send me
>> tokens
>> * [Request for access] When it is time to deploy, I would be like to
>> have access to, or have someone available with access to, vm2’s
>> management panel so that if there is a big explosion (unlikely, but
>> non-zero since the kernel needs updating) I can boot into recovery mode
>> and fix it
>>
>> Infrastructure:
>>
>> * Installation, configuration, and management of new server software
>> (e.g. new Buildbot) and users is now recorded in Ansible playbooks in Git
>> * Server timezones change to UTC to follow international standards
>> * Firewalls are a thing now
>> * User management is a thing now, using keys pulled automatically from
>> GitHub
>> * [Policy change] Pending approval, server changes will be managed
>> through Ansible instead of directly installing and modifying software on
>> servers going forward
>> * [Request for access] I need consent and admin access to the GitHub org
>> to transfer the Ansible repository to the org
>> * [Request for information] vm.scummvm.org needs to have its final
>> remaining services evaluated and moved away (subversion, planet, and
>> doxygen), and then needs to be wiped and reloaded with an up-to-date
>> 64-bit OS. Who can do this?
>>
>> Please send any feedback about the questions and policy changes above,
>> after reading this entire email since some questions may be addressed by
>> the extra information below.
>>
>> The remaining timeline for this work is:
>>
>> 1. Run full builds of ScummVM on all of the workers and address any
>> remaining compilation failures (I’ve fixed most of these already);
>> 2. Once everything is building, a PR for all the necessary build patches
>> will be created;
>> 3. Make any other necessary tweaks to the deployment configuration in
>> Ansible so everything critical will continue to be backed up appropriately;
>> 4. Back up the old Buildbot stuff somewhere else in case everything
>> explodes horribly and needs to be rolled back;
>> 5. Once the PR has landed, and any outstanding access issues or other
>> concerns have been resolved, deploy the new Buildbot.
>>
>> Beyond this point, I hope that other people will take over and do what’s
>> needed so that we can do things like create proper packages/installers
>> for the various platforms (currently only the Windows worker generates
>> proper installers), code sign the releases properly, publish to app
>> stores using e.g. fastlane, etc.. This has been a big distraction for me
>> for the last month or more and I just want to get back to working on
>> ScummVM itself.
>>
>> Here’s more information on these changes:
>>
>> The Buildbot upgrade
>> --------------------
>>
>> Currently, Buildbot consists of a big lump of binaries built in an
>> unknown manner and which must all run against the same host system. This
>> is not maintainable nor upgradable, and it prevents the host system from
>> being updated without running the risk of breaking Buildbot completely.
>>
>> The new Buildbot is containerised using Docker, so each platform has its
>> own separate environment which can be managed and upgraded independently
>> from the host system and from the other workers. The steps used to
>> create each environment are written in Dockerfiles, which are saved in a
>> Git repository, so it is never a mystery how the compilers and libraries
>> were generated, and newer images can be regenerated from this source
>> information to incorporate new library or compiler updates. Workers can
>> be easily moved around to different servers as necessary in order to
>> balance load and disk space, and developers can trivially create copies
>> of part or all of the entire Buildbot system to run locally.
>>
>> (Shameless plug: “<rsn8887> the new buildbot is awesome. I can built
>> *ANY* binary for *ANY* system using the same commands without any hassle”)
>>
>> The new Buildbot was used to create ScummVM 2.0 for Windows and is
>> successfully generating binaries[1] for every active port listed on our
>> Platforms page in the wiki.
>>
>> Shortly, I will be opening a pull request containing various patches
>> related to this work. Some of these patches will break the current
>> Buildbot, and users still building certain ports using older toolchains
>> and libraries. As such, Buildbot will be out of commission for a short
>> time so that the old Buildbot data may be archived away in case we need
>> to roll back for some reason, and the new Buildbot installed in its place.
>>
>> An ongoing problem with the current Buildbot has been disk space
>> exhaustion, and while this is in the process of being resolved by _sev
>> and our generous hosting provider, it is possible that some workers will
>> not be enabled initially in order to ensure there is enough available
>> space on the main server over the near term. The hope is that
>> vm.scummvm.org can be paved over with a fresh OS and then will be able
>> to receive any remaining workers, which will also reduce the amount of
>> time it takes to run builds.
>>
>> The loadout of ports built on the new Buildbot has changed to reflect
>> the up-to-date list of actively maintained ports on the wiki. These are
>> the changes:
>>
>> Added          | Removed        | Updated
>> -------------- | -------------- | --------------
>> FreeMiNT (GCC  | Android-MIPS   | AmigaOS (5.4)
>>  7.2)          | Android-x86    | Android-ARM (NDK r15c / Clang 5)
>> Haiku (5.4)    | Dingux         | Debian (6.3)
>> Maemo (5.1)    | DS             | Dreamcast (7.2)
>> Raspberry Pi   | GameCube       | GCW0 (4.9.1)
>>  (6.3)         | GP2X           | Haiku (5.4)
>>                | GP2XWiz        | iOS 7+ (Clang 3.8.1)
>>                | iOS 3-6        | macOS (Clang 3.8.1)
>>                | N64            | PS3 (7.2)
>>                | OpenPandora    | PSP (4.9.3)
>>                | Ouya           | Vita (7.2)
>>                | PS2            | Windows (6.3)
>>                | WebOS
>>
>> I did try to add SamsungTV and OS/2 workers, since these are listed as
>> active ports, and was unsuccessful. For SamsungTV, I was not able to
>> create a working compilation of glibc for its ancient kernel. For OS/2,
>> I could not find any cross-compiler, although there are apparently still
>> up-to-date GCCs being produced for that platform.
>>
>> Once the new Buildbot up and running, I would like to make it a
>> requirement for all ports in the master branch to have a Buildbot worker
>> image maintained by the porter for that platform. This will save
>> everyone time over the long run and improve the experience for our users
>> since the release process will be automated and it will free core
>> ScummVM team members from being wholly responsible for maintaining ports
>> infrastructure in addition to everything else they already need to do.
>>
>> The repository for this work is currently at
>> <https://github.com/csnover/scummvm-buildbot>.
>>
>> Infrastructure management
>> -------------------------
>>
>> As with the old Buildbot, our server management process is currently a
>> mystery process where changes are made directly on the servers without
>> any way to create new deployments or record change histories.
>>
>> In figuring out how to deploy the new Buildbot to ScummVM servers, bgK
>> mentioned having used Ansible to perform deployments at work, and others
>> I know who work in sysops shared their positive experiences managing
>> systems using this tool.
>>
>> Over the last week, I created an initial deployment plan for the new
>> Buildbot using Ansible which I hope can be used (along with Docker) for
>> all service deployments moving forward in order to make maintaining our
>> server infrastructure easier, safer, and more transparent. As Docker
>> allows anyone to create an identical copy of any of our application
>> environments locally, Ansible allows anyone to create an identical[2]
>> copy of our host server environments by running Ansible against any
>> other machine running the same base OS.
>>
>> This is a less user-facing change than the Buildbot change and mostly
>> only affects anyone responsible for maintaining our services, though
>> will probably also result in some short downtime for the main site as
>> some badly deferred server updates need to be applied, such as a kernel
>> update to prevent service failures when restarting containers. As such,
>> since we are near release time, I do not plan on deploying these changes
>> until at least a few weeks after the release to avoid unnecessary
>> downtime at this critical moment.
>>
>> Since I needed to set up VM environments for testing the deployment
>> anyway, I also introduced some playbooks for securing the servers, so
>> ingress firewalls will now exist, and user accounts for team members who
>> are no longer team members or do not need access to maintain server
>> infrastructure will not exist.
>>
>> At this point I’ve deployed the site playbook to some fresh VMs, and
>> everything seems to be working correctly, though there is also the
>> chance that some additional cleaning up will need to be done on the
>> actual production server since none of the current services have been
>> copied over into playbooks except for bits and pieces of what was needed
>> for the new Buildbot. Any issues with this can hopefully be addressed on
>> an ad-hoc basis and then services can be put into playbooks to solve
>> this problem for deployments to whatever different servers may exist in
>> the future.
>>
>> The repository for this work is currently at
>> <https://github.com/csnover/scummvm-infrastructure>.
>>
>> ---
>>
>> [1] Since I don’t have actual hardware to test all the binaries, I can
>> only say that the Debian, macOS, Maemo, PSP, Raspberry Pi, Vita, and
>> Windows binaries have been verified as working. Testing (and probably
>> tweaking) will be needed for the remaining ports.
>>
>> [2] Obviously, our security keys are encrypted so are not available for
>> any person to install. I don’t necessarily expect this repository to
>> remain public since there isn’t much reason for the general public to
>> duplicate our infrastructure, it’s mostly for allowing current and
>> future server maintainers the ability to easily set up a local testing
>> environment and for making it unburdensome to deploy our services to new
>> servers in the future.
>>
>> That’s all from me for now. Looking forward to your feedback. Thanks for
>> reading this long, long email!
>>
>> --
>> Colin Snover
>> https://zetafleet.com
>>
>>
>> _______________________________________________
>> Scummvm-devel mailing list
>> Scummvm-devel at lists.scummvm.org
>> http://lists.scummvm.org/listinfo/scummvm-devel
> _______________________________________________
> Scummvm-devel mailing list
> Scummvm-devel at lists.scummvm.org
> http://lists.scummvm.org/listinfo/scummvm-devel


-- 
Colin Snover
https://zetafleet.com





More information about the Scummvm-devel mailing list