[Scummvm-devel] Bring testers to the team?

[Max Horn]

Am 14.07.2009 um 11:33 schrieb Travis Howell:

> Max Horn wrote:
>> Am 14.07.2009 um 04:28 schrieb Travis Howell:
>>> Walter van Niftrik wrote:
>>>> If we add testers to the team that doesn't mean we need to stop
>>>> public
>>>> testing. ;)
>>> I know, I just meant it offers more disadvantages than advantages,
>>> compared to our public testing.
>>
>> Hu?? How can having dedicated testers *in addition* to the current
>> testing have *any* drawbacks?
>
> I thought I made that clear already, if the same person is testing the
> same game again and again over time, it will usually result in less
> thorough testing (essential parts only) each time.

Yes, you stated that already, but I don't agree with this conclusion  
at all. It is based on various implicit assumptions that I simply  
don't share, such as the believe that recruiting dedicated testers  
would exclusively draw from the existing pool of testers -- I believe  
we can draw additional ones. Nor do I share the assumption that this  
approach necessarily leads to burn out quicker. This would only be the  
case if we did something stupid, like requiring each tester to play  
the same game on several platforms during a single release.

In fact, I think that with proper coordination we can keep the  
"burnout" risk at a minimum. E.g. no tester should have to test a  
given game more than once for each release. Instead, make sure we test  
"key games" on as many different platforms as possible, ideally each  
by a different tester. (Key game here means that for each engine, we  
should have at least one game tested on each platform). For the next  
release, cycle to different key games. Of course, ideally we'd like to  
test each game on every platform (and ideally, on major variants of  
each platform, such as different WinCE/Symbian cell phones). But it's  
unlikely this will ever happen, so it'd be better to try to test at  
least some games from each engine on each platform. Currently, this is  
not at all always the case.




>
>> That makes no sense. Especially since some porters (e.g. Neil for the
>> NDS) have specifically stated that testing all games on the port,
>> which they have to do themselves, is one of the major time wastes in
>> the release cycle. Public testing is already employed there, but
>> simply not enough. If we have a handful people who test beta builds  
>> of
>> the NDS port, for example, that would greatly help Neil.
>
> Since when was the porter responsible for all testing? in the past  
> beta
> builds of the NDS port have been posted via the forums, before each
> release, and received feedback on issues.

I didn't mean that porters are responsible for all testing -- I just  
meant that in reality, they often end up having to do the bulk of  
testing. In the NDS case, Neil posted those test builds, and got  
feedback, but from what I understood from the last time, this only  
covered a fraction of the testing work *he* did; and many serious bugs  
would have slipped through if he hadn't done that extensive testing.  
So, he was not *obliged* to do lots of testing, but it was the only  
way to really get good test coverage, apparently. And even then he  
doesn't have all games.



> It should still be up to each porter to a least confirm each game  
> engine
> is actually working though.

That's actually not possible, because not every porters owns all games.


Bye,
Max