[Scummvm-devel] Bring testers to the team?

[Neil Millstone]

Max Horn wrote:
>>> That makes no sense. Especially since some porters (e.g. Neil for the
>>> NDS) have specifically stated that testing all games on the port,
>>> which they have to do themselves, is one of the major time wastes in
>>> the release cycle. Public testing is already employed there, but
>>> simply not enough. If we have a handful people who test beta builds  
>>> of
>>> the NDS port, for example, that would greatly help Neil.
>>>       
>> Since when was the porter responsible for all testing? in the past  
>> beta
>> builds of the NDS port have been posted via the forums, before each
>> release, and received feedback on issues.
>>     
>
> I didn't mean that porters are responsible for all testing -- I just  
> meant that in reality, they often end up having to do the bulk of  
> testing. In the NDS case, Neil posted those test builds, and got  
> feedback, but from what I understood from the last time, this only  
> covered a fraction of the testing work *he* did; and many serious bugs  
> would have slipped through if he hadn't done that extensive testing.  
> So, he was not *obliged* to do lots of testing, but it was the only  
> way to really get good test coverage, apparently. And even then he  
> doesn't have all games.
>   
Yes, I always test my builds thoroughly.  I usually run at least one 
game in each engine, more for the Scumm engine, because that's the one 
most people use.  In each game I check that several rooms work, perform 
some actions to check mouse input and overall performance, and do a load 
and save a couple of times.  In all, I probably test 15 games this way, 
and fix the issues found before releasing my first beta.

Usually I discover a whole lot of bugs this way.  Public testing has 
been patchy in the past.  People usually only post to a forum when 
things don't work, which leaves me with an empty forum thread.  Then I'm 
left wondering whether it all works fine, or maybe nobody even 
downloaded it!

So I think recruiting dedicated people to test ScummVM, and some for the 
DS port would still be incredibly useful.  Maybe if they were a full 
team member and got on the credits they would be more inclined to spent 
lots of time testing!

To do it properly, I would have to get a team of people and tell them to 
play specific games, reporting problems with each.  There are many 
issues though, people could only test games they own, and filtering out 
those bug reports which are down to corrupted SD cards or broken drivers 
etc. is sometimes an issue.

But I say, let's give it a go.

- Neil