[Scummvm-devel] Bring testers to the team?

Travis Howell kirben at optusnet.com.au
Wed Jul 15 08:50:06 CEST 2009


Max Horn wrote:
> Am 14.07.2009 um 11:33 schrieb Travis Howell:
>> Max Horn wrote:
>>> Am 14.07.2009 um 04:28 schrieb Travis Howell:
>>>> Walter van Niftrik wrote:
>>>>> If we add testers to the team that doesn't mean we need to stop
>>>>> public
>>>>> testing. ;)
>>>> I know, I just meant it offers more disadvantages than advantages,
>>>> compared to our public testing.
>>> Hu?? How can having dedicated testers *in addition* to the current
>>> testing have *any* drawbacks?
>> I thought I made that clear already, if the same person is testing the
>> same game again and again over time, it will usually result in less
>> thorough testing (essential parts only) each time.
> 
> Yes, you stated that already, but I don't agree with this conclusion  
> at all. It is based on various implicit assumptions that I simply  
> don't share, such as the believe that recruiting dedicated testers  
> would exclusively draw from the existing pool of testers -- I believe  
> we can draw additional ones. Nor do I share the assumption that this  
> approach necessarily leads to burn out quicker. This would only be the  
> case if we did something stupid, like requiring each tester to play  
> the same game on several platforms during a single release.

Recruiting dedicated testers will very likely drawn from our existing
pool of testers, especially if you take into account how rare many of
the supported games (and their ports) still are.

I fear we would end up relying more and more on these game testers,
which could result in faster testing, but in less quality testing
overall. If testers have played through game many times before, they are
less likely to explore as many options (ie talking to less people), even
if they aren't burned out on a particular game.

With only a minimum of 5 games been required for testers, it could
result in some play testers playing through the same games, due to a
limited range of games too.

> In fact, I think that with proper coordination we can keep the  
> "burnout" risk at a minimum. E.g. no tester should have to test a  
> given game more than once for each release. Instead, make sure we test  
> "key games" on as many different platforms as possible, ideally each  
> by a different tester. (Key game here means that for each engine, we  
> should have at least one game tested on each platform). For the next  
> release, cycle to different key games. Of course, ideally we'd like to  
> test each game on every platform (and ideally, on major variants of  
> each platform, such as different WinCE/Symbian cell phones). But it's  
> unlikely this will ever happen, so it'd be better to try to test at  
> least some games from each engine on each platform. Currently, this is  
> not at all always the case.

If major issue(s) are found in a game or ScummVM port, it can mean play
testing through a single game several times, for a single release cycle.
Which could mean testers are still required to play through the same
game again, depending on how many testers own that game and platform.

With many supported games (and their ports) still been quite rare, I'm
not convinced that changing the set of games for each tester, in each
release cycle would even be viable.

>> It should still be up to each porter to a least confirm each game  
>> engine
>> is actually working though.
> 
> That's actually not possible, because not every porters owns all games.

Well we have a large archive of game demos available now (at
http://scummvm.sourceforge.net/demos/ and http://demos.robertmegone.com/
) , which should allow porters to a least try a game from almost all
game engines.






More information about the Scummvm-devel mailing list