[Scummvm-devel] Bring testers to the team?

[Max Horn]

Am 15.07.2009 um 11:51 schrieb Walter van Niftrik:
[...]
>> Someone with some
>> knowledge about programming, would usually be required to track  
>> down the
>> exact code segment causing the regression.
>
> If we're tracking down a regression that occurred somewhere in the  
> past
> 5 months, finding the exact revision is most of the work. Doing this
> requires no programming skill.

[This is getting somewhat offtopic now, but anyway... ;) ]
Well, they still would need to be able to get their hands on test  
builds for that time period. Our daily builds go back to May right  
now. I am not sure whether we can keep an archive like that forever,  
though; disk space is cheap, but not unlimited, and we are using quite  
a bunch of it now.

Maybe we can come up with a way to store these nightlies efficiently  
(maybe using a binary diff approach, i.e., only store a binary diff  
between builds; of course then one would need an efficient retrieval  
system, too). But I think it's too far outside our scope to work on  
something like this, so unless there is an off-the-shelf solution, I'd  
not bother with it.

On-demand-building of certain revisions would open up a denial-of- 
service attack possibility, so I don't see that as an alternative.

Hence, to find out when a regression was introduced precisely, we  
still would have to either provide custom builds to testers; or ask  
them to compile SVN checkouts themselves, and then do bisecting on it.  
Still doable even for a non-developer, but non-trivial if you don't  
have a compilation environment already. If we can find a couple  
testers able and willing to do this, it would be awesome; we should  
then try to make the barrier as low as possible.

(And of course finding the right revision is a *lot* easier and faster  
with "git bisect" or the equivalent mercurial command... ;).


Cheers,
Max