[Scummvm-devel] What is happening to the ScummVM team?

[Max Horn]

Hi Eugene,

thanks for your reasonable mail on the subject, and thanks to all the  
others who so far wrote very constructive mails. I am very happy to  
know more about the status of cruise, igor, made, m4 now, and hope  
they will prosper.


Am 13.02.2009 um 06:40 schrieb Eugene Sandulenko:

> OK, now it's my turn. I've addressed some issues in other e-mail, see
> the rest below.

[...]

>
> Still, I am really don't like the way you point this out. As I
> announce with each release schedule, I do not put it in stone. A
> simple request with sound reasons will easily make me push release
> date or even cancel it altogether. It is not that I am dictating it,
> although certain degree of dictatorship is required from the one
> wearing a release manager hat ;).

I wanted to quickly chime in and say this: Eugene, I really appreciate  
the immense work you have put into acting as release manager for  
ScummVM! I only did this a few times, and even back then it was a lot  
of work -- now it's a lot like herding bees. Given the limited  
resources, I think you are doing an awesome job.

Now I am not saying everything is perfect, but indeed, we are  
learning, and trying to improve things. And help is always welcome! It  
is very tiresome to get a release together. And I agree with Kostas  
that frequent releases are important; we can't wait until all bugs are  
fixed, as that way we'll never release. Even if in one engine we know  
about a series regression, we may have to release anyway (with that  
engine disabled and/or a big warning sign to users), as to not impede  
the others. But of course first we should make every effort to fix  
those issues.

>
>
>> Of course we now have a special date for ports to create prerelease
>> binaries. That is a step in the right direction. I am currently not  
>> sure
>> in how far our ports adapted that though.
> I saw a delighting reaction from our porters on that. Some of them
> took a step ahead and started to publish pre-release builds even
> earlier. Thank you very much, guys.
>
>> I also know some might say, "but we should really have a release  
>> soon"
>> or "we always release every 6 months", but in my opinion we should
>> rather have a quality release than a release in a fixed period, which
>> has too many annoying bugs. Thus my vote for this release is: wait  
>> for
>> more testing results! (means delay it)
> Although I wholeheartedly agree with you on this, unfortunately
> experience shows that in this case it will NEVER happen. The hardcore
> users just refuse to replay even beloved games 130-th time. And the
> newcomers just don't care.

[...]

> For the sake of this, and hoping that we will participate in GSoC'09,
> I am looking into automated testing task which I recently added to
> http://wiki.scummvm.org/index.php/OpenTasks#Implement_recorded_play
> Otherwise I see no bright light in the way of quality release testing.


Exactly. Automated game play testing would be nice. SSyke suggested  
that years ago, but it's one of those many features "someone" has to  
implement, and it is difficult :-/. Maybe the event recording code we  
have these days could be used to help with this...

Hey, we really should make adding testing (unit tests, automatic game  
reply tests, whatever) to the GSoC tasks.

Game tests could consists of a savestate for a game, plus an even  
sequence, and then there must be some way to verify things are in  
order. That could include:
1) comparing screenshots of the game state at certain intervals (right  
after loading; after executing a certain event sequence; etc.)
2) comparing sound / midi output
3) comparing save states: Load a fixed savestate, run an even  
sequence, save. Compare (uncompressed) savestae to a reference sample.  
Are there differences? Allow exluding sections of the savegame that  
are allowed to change (the latter could break with savegame revisions,  
of course)
4) add engine specific code to verify state, e.g. certain variables.

Of course none of these are perfect, none can catch all issues, and  
this might not even work in all games/engines/scenes, but as syke  
pointed out back then: Even partial testing can help *a lot*. And the  
above two points would be relatively generic, working for all engines,  
hence more likely to be usable soon

So, we could assemble a large test suite of these, and whenever we  
encounter regressions, we add a new test just for that, to make sure  
it doesn't re-appear.

besides automated play testing, of course we would have to add tons of  
unit tests, including test for the things LordHoto already pointed  
out; but in addition, a framework allowing *engines* to add unit tests  
would be nice. Of course this would require work by engine authors to  
actually be usable, but it would be optional anyway.

A lot of wild ideas, I know. GSoC ideas? Or maybe "someone" is willing  
to work on it???




All the best,
Max