[Scummvm-devel] Bring testers to the team?

Travis Howell kirben at optusnet.com.au
Wed Jul 15 11:26:44 CEST 2009


Eugene Sandulenko wrote:
> On Wed, 15 Jul 2009 16:50:06 +1000
> Travis Howell <kirben at optusnet.com.au> wrote:
>> Recruiting dedicated testers will very likely drawn from our existing
>> pool of testers, especially if you take into account how rare many of
>> the supported games (and their ports) still are.
> I thought we do not have this existing pool of testers. Each release
> different people test the games.

I meant people which frequently help out with public testing, during 
release cycles in the past.

>> I fear we would end up relying more and more on these game testers,
>> which could result in faster testing, but in less quality testing
>> overall. If testers have played through game many times before, they
>> are less likely to explore as many options (ie talking to less
>> people), even if they aren't burned out on a particular game.
> Well, once the game was thoroughtly tested on a stabilized engine,
> regressions if any are quite obvious. I.e. knowing the game is in fact
> a plus, as the eye can be more catchy for the differences.

Really depends on the bug, I recall minor regression in one game (Simon 
the Sorcerer 2) I tested in the past, which only had slight effect 
(minor color differences) during conversions in one location (late in 
the game).

>> With only a minimum of 5 games been required for testers, it could
>> result in some play testers playing through the same games, due to a
>> limited range of games too.
> Minimum could be raised, it is not a problem. We just need to come with
> max number where potential people will still exist, i.e. how many
> people out there own 100 ScummVM-supported games?
> 
>> If major issue(s) are found in a game or ScummVM port, it can mean
>> play testing through a single game several times, for a single
>> release cycle. Which could mean testers are still required to play
>> through the same game again, depending on how many testers own that
>> game and platform.
> When this major issue is fixed, devs usually know what could be
> impacted. And although they playtesting up to the point could be
> fast-forwarded, a dedicated tester could devote more time for the
> affected areas. Also not that often fixes break the saves, thus, a
> prepared tester can have number of saves for faster and more quality
> testing.

I meant a more risky late code change, to fix major issue, which could 
have other side effects. Meaning a complete play through of a game is 
required again, to confirm no further regressions occur in that game.

It usually isn't that easy to tell exactly where specific code is used 
in a game though, even when you have the script output.

>> With many supported games (and their ports) still been quite rare, I'm
>> not convinced that changing the set of games for each tester, in each
>> release cycle would even be viable.
> It may even happen that not all games could be covered by such testers,
> and we will still rely purely on the userbase.

Yes, unless several game collectors offered to help with the play 
testing. Difficult to find most of the rarer games for sale, especially 
at anywhere near a decent price.




More information about the Scummvm-devel mailing list