[Scummvm-devel] Game testings

Thierry Crozat criezy at scummvm.org
Sat May 21 13:37:53 CEST 2016


Hi team,

There have been several discussions in the past about game testing. We used to have testing for all games before releases, but with the increasing number of games and failure to attract more testers this became unsustainable and we now only ask for testing for a few games before releases. There are some worries however that it may result in unseen regressions and impact the quality of the ScummVM releases.

Finishing the work on the Event Recorder and thus allowing automatic testing would help. If somebody is interested in working on this I am sure this will make some people very happy ;) If you have other ideas, they are also off course welcome, especially if you are willing to do something to implement those.

But in this email  I would like to focus on an idea raised by Kirben recently:
> A brief news item, and sticked forum post could help get more reports of games completed by users in general, which could be tracked via a page on the ScummVM wiki.
> 
> Or an even better idea:
> The DOSBox web site offers a database where users can register, and report games known to work with each version, along with comments on any specific issues. Maybe we could offer a similar database for ScummVM? where users can report which games they complete, the ScummVM version and platform was used, along with any helpful comments.


 This resulted in a small paragraph in our news post calling for release testing for ScummVM 1.8.1:
> Also from now on we encourage you to test any game which we support and report any bugs or your success on the forum so we can keep track of current state of affairs in the ScummVM Land.


I am not sure how many people noticed this paragraph and understood what that meant. Also I would like to go a bit further. I have no experience with databases and web site design. If somebody wants to implement a proper database with an online form for user to report their testing and a web page to see those and search the database that is fine with me. But since I suspect this might not happen here is why I propose to do:
1/ Create a sticky forum post indicating users are encourage to report in this forum thread bugs or completion success for any game supported by ScummVM with the following information:
   Game:
   Game version and platform:
   ScummVM version:
   Played on platform:
   Satus: No issues/Minor bugs/Major bugs/Blocking bugs
   Description of issues found  (with bug id when applicable)

2/ A forum thread is not a convenient place for searching and sorting entries. So propose for reports to be copied in a different one, but where?
The information is somewhat similar to the compatibility pages on the web site and could possibly be added there (e.g. in the Game compatibility details when clicking on a game). But I think it will be most useful if sorted by games rather than ScummVM versions.
I see several options, and any feedback or additional ideas you may have is welcome.
   - Add the information on the game pages on the wiki
   - Or create a new wiki page to centralise the information on a single page
   - Or add the information on the game pages on the wiki and use the DPL extension to also automatically gather all the information on one page (as we do with the platform page (http://wiki.scummvm.org/index.php/Platforms). One issue with this approach is that the automatically generated page may be slow to access.
   - Or create a new page on the web site.

3/ Possibly start populating the page(s) with information from past release testings.

4/ Post a news items to announce this (on its own, not mixed with other news such as 1.8.1 release, to increase its visibility).
I think the news items get automatically reported on our twitter/facebook/Google+ pages? If not it would be good to have it there as well.

This will happen after the 1.8.1 release as I would like feedback on the idea and on the options listed in the second point above (and I will be unavailable in the coming week anyway). If the idea proves popular I will start implementing it around mid-June. Of course volunteers are welcome as well. And maybe a better alternative idea will emerge.

Thierry / criezy



More information about the Scummvm-devel mailing list