[Scummvm-devel] Avoiding loading of all plugins during startup

[Max Horn]

Hi there,

Tony "toneman" Puccinelli is one of our GSoC students and is working on improving plugin support.

One of the issues with plugins is that we currently have to load all engine plugins before starting a game. This is what happens: When the user asks to play game (or rather, "target") "xyz", we look through the config file for a suitable entry. Once found, we then iterate over all engine plugins, and ask each in turn whether it is able to run that game. Once we found one, we use it, possibly unloading all other plugins.

Clearly this is wasteful, as we have to first load all plugins into memory (so we have to have enough memory to keep all in), and then unload all but one (which might leave the memory fragment).

So how to improve this situation? I'll try to outline some of my thoughts below. Remarks and further ideas are highly welcome!


1) Quick and simple refinement
------------------------------

A very simple way to improve this situation slightly is to not load all engine plugins at once, but rather load one, ask it whether it supports the given game; if not, unload it and proceed to the next. This way only one plugin is held in memory and fragmentation should be kept to a minimum. But we still have an overhead (as in the worst case we still have to load all plugins). Might be worse investigating this as a quick temporary measure, though.


2) Suggested long term approach
-------------------------------

On the long run, I propose this: We add to every game target in the config file an "engine=BLABLA" key-value pair. This way, we can pick the right engine plugin right away. To ease transition to this new format, there are several ways: One is that if the "engine=" field is missing, we just revert to the old behavior. Another is that during startup, we scan through all config targets, and if we find any missing the "engine" key, we try to add it. By re-running the detector, matching the output with the gameid that is already there, and setting up the engine field. For this, the upgradeTargets()  function in base/commandLine.cpp could be adapted.

In order to avoid having to do this re-scanning every time, I would suggest also finally putting the "version" field in our config file to some use: Right now it is worthless, because we simply set it to the ScummVM version whenever we load a config file. We could change that: If loading a config file made with a newer ScummVM version, issue a warning? And if the version is older than XYZ, run the target upgrader. 

(As an aside, an "engine" key would also make it possible to let multiple engines use the same gameid (think about LSL1 in both AGI and SCI). I am not sure whether it would be a good idea to open that can of worms, though, esp. for backward compatibility reasons.)


3) Alternate approach
---------------------
Another approach has been suggested in the past: Split the plugins into a pure detector part, and a part which runs the games. THe idea is that the detector plugins would be relatively small and could just all be loaded, while the full plugin would be loaded only when a game is actually started.
This has its merits, but I am a bit reluctant to pursue this venue, as I feel it might complicate engine plugin code considerably. E.g. in the SCI engine, the detection code heavily release on the ResourceManaager class, which makes up a sizeable chunk of the actual engine. It seems difficult (and artificial) to separate those.

A limited version of this, however, might be doable and already solve most of our problems: We only really need the mapping from gameid to engine to solve most of our problems! So if we come up with a way to provide that to ScummVM... Some approaches: for every engine plugin, add ...
a) ... a second mini-plugin which provides just this mapping
b) ... a simple text file which provides this mapping
c) ... a mapping table "gameid -> engine" into the ScummVM binary itself

a) means doubling the plugin count, but keeps the code specific to one engine bundled in one place, while c) would spread some of it into the ScummVM core. Also c) means that only plugins known at compile time of the scummvm binary can be loaded. Approach b) is easy to maintain and require little code, but the downside is that it is more work to make sure it stays up-to-date.
All three approaches bear the risk that one may end up with a mismatch between the "gameid->engine" table used and the engine versions used. 



4) Additional thoughts
----------------------

That's the rough short version, anyway. Note that combinations of 1), 2) and 3) are of course possible!

Some additional issues to keep in mind:

* We currently support starting a game without a config entry, too: If you type
  scummvm monkey2
and have no monkey2 target configured, it will try to locate monkey2 in the current directory. We can't know the engine this way. Some "solutions":
** If approach 3 is realized in some way, this problem does not exist ;)
** Drop this feature (I don't think the feature is very important; then again, dropping a long standing feature just because your new code has difficulties supporting is nasty -- I don't like this :/)
** Require the user to specify a --engine=FOOBAR option -- bad from a user's perspective, I don't like this much either
** Fall back to the current mode of loading all plugins (possibly with the refinement from 1) above). Easy to do, but not perfect. 


* We still need to support adding games in the launcher, including the mass add feature. We can just keep the current way: Load all plugins, then detect with each in turn. Or we do it iteratively: Load plugin 1, scan for games; load plugin 2, scan for games; etc.
For a plain "Add game", both should work well. If "Mass Add" is used, though, then we may need to traverse many many directories; if we use an iterative scanning approach, we now have to iterate all those directories N times, where N is the number of engine plugins. Depending on the system, this may be a lot slower than loading all engines, then traversing the directory tree only once.
I don't see an ideal solution for this right now, only various tradeoff. Of course one can argue that a huge "Mass Add" is something relatively rare, and probably mostly happens on "higher end" machines, so it's not so bad if it is slow... But maybe this thought is wrong? Porters?


* We need to still support "--list-games" which may require loading all plugins (but could be mitigated by approach 3)


* There might be further pitfalls, would be good if everybody gave this some  thought... :)


Cheers,
Max