[Scummvm-devel] Evaluating GitHub issues tracker

[Max Horn]

Hi folks,

here are some remarks of mine on the subject of issue trackers: It should be as easy as *sensibly* possible to submit *good* bug reports as possible. This has various ramifications.

Anyhow:

1) User management:
I do not like anonymous bug reports; it makes it impossible to get back to people to ask for more information. We may loose some reports, but given our limited resources to handle bug reports, I think this is a very important and very acceptable trade-off.
Conclusion: While people should be required to be "logged in" / leave a verified email contact address, this should be as easy as possible. Ideally, people would not have to create yet another user account! With SF.net, this is solved in an IMHO acceptable way: SF.net accounts are useful far beyond ScummVM, and these days you can even use OpenID to login on SF.net!
Any alternative shouldn't make this worse than SF.net. IMHO github is worse than SF.net in this regard
Regarding "sharing" our forums accounts: This has some appeal. But it would only be possible if we hosted our own, *and* there are thousands of unused / spammer accounts in there. I think I'd prefer a solution with OpenID over this anytime.
A middle ground would be using a trac or mantisbt instance hosted by SF.net, see e.g. <http://sourceforge.net/apps/mantisbt/scummvm/> or <http://sourceforge.net/apps/trac/scummvm/>. But both of these are by now heavily *dated* and I have serious doubts that SF.net will update them -- they seem to change their strategy every year, right now they are developing a complete new SF.net system, with yet another hand rolled issue tracker and all that...

2) Self-hosted vs. external hosted.
In general, I think I'd prefer an external provider over a self-maintained solution, *assuming* that provider was constantly applying security fixes, installing updates etc.. Working on ScummVM is taxing enough, and I'd prefer to avoid getting tangled up in maintenance of yet another piece of software. I don't care about branding for our tracker either. 
SF.net offers trac and mantis BT -- but these installations are very dated, and we can't update them, so I consider this a con. There are many alternatives for hosted bug trackers, though. Here are some:
* SF.net issue tracker
* SF.net trac, mantisbt, ...
* SF.net *NEW* issue tracker, sadly only available to new projects. See e.g. here <https://sourceforge.net/u/fingolfin/tickets/>
* github issues tracker
* http://lighthouseapp.com/
* Google Code
* Jira <http://www.atlassian.com/software/jira/>
* ...

There are also alternatives to self-hosted bug trackers. Some that were mentioned already:
* trac
* redmine
* bugzilla
* ... tons more ...

3) Ease-of-use for simple users:
 The standard interface for submitting a bug should be easy to use and understand, so that even newbies can use it to make a good report. Yet it should not prevent more experienced users from providing extra data.
SF.net tracker is OK in this regard. The (old?) bugzilla interface was horrible, in that even I as a developer felt daunted by its sheer mass of options, widgets, popups, fields etc. (They improved this a lot in modern versions, though). github, trac, redmine all are doing OK here.
Mantis BT is IMO about as bad as old Bugzilla versions were. 


4) Power and ease-of-use for developers:
It should be possible to add useful meta-info to the reports (by us developers, I mean) without having to resort to nasty tricks (like prefixing an "ENGINE NAME:" in bug titles, like we do on SF.net). I like that SF.net provides a category & group field, but it is far too limited in that it does not allow arbitrary "categories", nor tags. The github issue tracker is IMHO worse, as it only allows "labels" (and in a test I just did on Firefox 4, these did not work *at all* :/).

Another important feature: Good "reporting", i.e. being able to list "all open bug reports about the TOON engine on platform WinCE" and stuff. SF.net tracker is doing reasonably well here. But do not *force* this on me like trac does (I hate that). 

SF.net tracker does not allow formatting in comments, does not allow editing comments; both would be OK, but since it also has no *preview* button, this becomes quite annoying. I like how github uses markdown for this stuff.

5) Importing existing data: This seems to be quite problematic, as not much code for that seems to exist, and that which does exist is not actively maintained and underfeatured. For example, while SF.net allows exporting the tracker data to XML, this data is not complete; e.g. attachments are missing. So, a proper importer IMHO needs to be able to grab the XML, then extract all attachment URLs from that, and then either download all attachment data; or at least transfer the links to the new system. Haven't yet seen any tool doing that... :/

6) Spam protection: I am not sure why, but somehow with SF.net trackers, we get no tracker spam. Curiously, on the official SF.net *trac* and also on some other trac installations, I have observed a considerably share of spam comments. Spam fighting takes away time, anything that helps avoid this is good.


All in all, I feel that the github tracker is too limited for us.  And the really hard part seems to be 5, but maybe we don't need / want it? Hrm. Lastly, any transition would only make sense if it had a sufficiently high positive effect on our productivity, after considering the amount of work we'd have to invest.


Bye,
Ma