[Scummvm-devel] ScummVM 0.12.0: Missing builds

[Max Horn]

Am 02.09.2008 um 22:32 schrieb Neil Millstone:
>
> Hi Max,

Hi Neil!

>
>
> I would say that six weeks isn't really enough.  It took me quite a  
> while to get a build I was happy releasing as a beta, as there were  
> many changes and many thing broken.  Except during release time, I  
> usually play only the Lucasarts games, so there were problems with  
> the rest of the code that hadn't been noticed.
>
> After I released my first beta on Aug 12th, there were a few rounds  
> of changes, but lots was discovered late on.  There's just too much  
> to test, and I can't play hundreds of hours of games on my own.

Certainly not! Looking at the forums, part of the problem seems to be  
that there are many eager DS users who jump immediately on your betas;  
but when it comes to reporting issues, there seems to be little  
feedback (at least on the forums; maybe you get more feedback via  
email) ?

Automated testing, as Betrand suggest, would help a lot, but I just  
don't see how. Maybe if there was a good DS emu, one could at least do  
some testing (better than nothing), but I think there is none (at  
least no free one).

So, best I can think of: we need a better QA team who helps with this.  
A group of dedicated NDS users; a group of dedicated PSP users; a  
group of dedicated PS2 users; etc.; and those groups jump on new betas  
of the resp. port, and start play testing; ideally each individual on  
a different game. And then collect the feedback: Worked/crashed doing  
XYZ/... And of course, these test groups would have to be able to cope  
with lots of fluctuation: New people come in all the time, others get  
bored and stop testing, etc...

Organizing one, let alone a dozen, QA teams, is hard work on its own.  
On the plus side, it doesn't have to be organized by the porter, at  
least not by him alone. And by using good tools, a lot of this can be  
streamlined. The easier it is to give feedback, and to give feedback,  
the more we get (I'd hope, at least ;). The easier it is to find out  
which game/platform/beta combinations have not yet been tested, the  
easier to avoid duplicating work...


Does anybody know a good & free web app that is targeting Q&A work?  
Ideally including features like:
* keeping track of the status of
   k platforms with m different intermediary revisions running of p  
game variants
* making it very easy for users to submit feedback on each of these  
k*m*p entries: Worked / Crashed ... / ..
-> that is, more than a bug tracker (also track success, to avoid  
duplicate testing), and definitely more
    than what SF.net offers, but not as complicated as BugZilla (which  
scares even me, not to mention regular users)
* helping us to manage pools of QA testing volunteers, and allowing  
them to pick certain QA tools, tracking who is looking into what, and  
thus avoiding duplicate work; and also allowing us to see whether a  
certain QA test is overdue and hence should be assigned to somebody  
else.


And we *do* have the money to pay for this a bit, e.g. a server to  
host it.

Also note, there are lots of interesting documents and links on this  
site: <http://www.aptest.com/resources.html>.

Finally, this is definitely something we should ask folks about during  
the Google SoC mentor summit, where me and somebody else (Eugene?)  
will be. I mean, huge projects like FireFox, GCC etc. all have to cope  
with this and more, but of course they also have bigger resources.  
Certainly worthwhile to look at their system (though for GCC,  
automated testing is a bit easier than for us, hence FireFox /  
OpenOffice / etc. are the projects we really need to look at).


> My current issue is that Inherit the Earth is now about 100Kb over  
> the available memory.  This seems to be mostly code size gained  
> since 0.11.0.  While we can't really prevent the binary from  
> growing, I can't imagine that I'll be able to get this game working  
> if the same happens next time.  If anyone knows how I can save 100Kb  
> in this game, please let me know!

To repeat what Filipppos wrote: This is the first time I hear about  
this :(. Well, I assume you found out about this only recently, else  
I'd now start with my "we need to communicate more and better" jingle  
*g*. Anyway, now that we know, the SAGA folks can also look into the  
issue; given that they know the engine best, they might be able to help.


Bye,
Max