[Scummvm-devel] Bring testers to the team?

[Eugene Sandulenko]

On Wed, 15 Jul 2009 16:50:06 +1000
Travis Howell <kirben at optusnet.com.au> wrote:
> Recruiting dedicated testers will very likely drawn from our existing
> pool of testers, especially if you take into account how rare many of
> the supported games (and their ports) still are.
I thought we do not have this existing pool of testers. Each release
different people test the games.

> I fear we would end up relying more and more on these game testers,
> which could result in faster testing, but in less quality testing
> overall. If testers have played through game many times before, they
> are less likely to explore as many options (ie talking to less
> people), even if they aren't burned out on a particular game.
Well, once the game was thoroughtly tested on a stabilized engine,
regressions if any are quite obvious. I.e. knowing the game is in fact
a plus, as the eye can be more catchy for the differences.

> With only a minimum of 5 games been required for testers, it could
> result in some play testers playing through the same games, due to a
> limited range of games too.
Minimum could be raised, it is not a problem. We just need to come with
max number where potential people will still exist, i.e. how many
people out there own 100 ScummVM-supported games?

> If major issue(s) are found in a game or ScummVM port, it can mean
> play testing through a single game several times, for a single
> release cycle. Which could mean testers are still required to play
> through the same game again, depending on how many testers own that
> game and platform.
When this major issue is fixed, devs usually know what could be
impacted. And although they playtesting up to the point could be
fast-forwarded, a dedicated tester could devote more time for the
affected areas. Also not that often fixes break the saves, thus, a
prepared tester can have number of saves for faster and more quality
testing.

> With many supported games (and their ports) still been quite rare, I'm
> not convinced that changing the set of games for each tester, in each
> release cycle would even be viable.
It may even happen that not all games could be covered by such testers,
and we will still rely purely on the userbase.


Eugene