[Scummvm-devel] Bring testers to the team?

[Johannes Schickel]

Max Horn wrote:
> In fact, I think that with proper coordination we can keep the  
> "burnout" risk at a minimum. E.g. no tester should have to test a  
> given game more than once for each release. Instead, make sure we test  
> "key games" on as many different platforms as possible, ideally each  
> by a different tester. (Key game here means that for each engine, we  
> should have at least one game tested on each platform). For the next  
> release, cycle to different key games. Of course, ideally we'd like to  
> test each game on every platform (and ideally, on major variants of  
> each platform, such as different WinCE/Symbian cell phones). But it's  
> unlikely this will ever happen, so it'd be better to try to test at  
> least some games from each engine on each platform. Currently, this is  
> not at all always the case.

At least for 'official' test entries the last time we had a game report 
listed on our game testing wiki page it was *one* WinCE test for 0.12.0.

I see two main reasons for that:

- First I guess most test results were only listed in the port subforum 
and thus not catched by the people updating the wiki

- Next looking at the 0.13.0 release time line:

 January 12th: Feature freeze
 January 17th: Start of playtesting
 January 19th: Branch, trunk is unfrozen
 February 11th: All ports should be frozen, prerelease binaries built
for testing
 February 19th:Tagging
 February 28th: Release

It is clearly visible that we just required the porters to be ready a one week before our final release tarball was finished.

This release it looks a bit better planned:

  July 10th -- feature freeze, testing starts (users test 0.14.0svn)
  July 24th -- branch, trunk is unfrozen (users test 1.0.0pre)
  August 10th -- all ports are frozen, porters start test builds.
  August 21st -- tagging, builds start getting uploaded
  August 31st -- release


It at least requires the porters to be ready nearly two weeks before 
tag, but still one month after the play testing starts.

I would *guess* that depending on the status of the ports, people will 
rather try their games on Win/Linux/OS X then. Thus they will have less 
motivation to test them on some of the other ports, they might have 
access to, again. Of course if the ports are fine that won't argument is 
non existent, but for some ports, which are only updated shortly before 
release this might be a killer for release testing.

- Also in the past we didn't have buildbot, thus it was harder to get 
test builds, if the porter was too busy to create them.

// Johannes