[Scummvm-devel] Avoiding loading of all plugins during startup

Johannes Schickel lordhoto at scummvm.org
Wed Jun 23 19:50:32 CEST 2010


On 06/23/2010 11:54 AM, Max Horn wrote:
> 1) Quick and simple refinement
> ------------------------------
>
> A very simple way to improve this situation slightly is to not load all engine plugins at once, but rather load one, ask it whether it supports the given game; if not, unload it and proceed to the next. This way only one plugin is held in memory and fragmentation should be kept to a minimum. But we still have an overhead (as in the worst case we still have to load all plugins). Might be worse investigating this as a quick temporary measure, though.
>    

Sounds fine for a temporary solution to me. I would imagine this could 
be used to first focus on implementing the plugin loaders and test them 
(for example on NDS, where we know of serious memory limitations ;-). 
Though I would be rather against merging this into the mainline later on.


> 2) Suggested long term approach
> -------------------------------
>
> On the long run, I propose this: We add to every game target in the config file an "engine=BLABLA" key-value pair. This way, we can pick the right engine plugin right away. To ease transition to this new format, there are several ways: One is that if the "engine=" field is missing, we just revert to the old behavior. Another is that during startup, we scan through all config targets, and if we find any missing the "engine" key, we try to add it. By re-running the detector, matching the output with the gameid that is already there, and setting up the engine field. For this, the upgradeTargets()  function in base/commandLine.cpp could be adapted.
>    

Sounds fine to me. We might also use that engine key to check whether 
the plugin for a given target really exists.

I have one question though: what exactly do we store there? The base 
name of the engine? If so what will we do about engine renaming (like it 
happened with simon -> agos in the past)?


> In order to avoid having to do this re-scanning every time, I would suggest also finally putting the "version" field in our config file to some use: Right now it is worthless, because we simply set it to the ScummVM version whenever we load a config file. We could change that: If loading a config file made with a newer ScummVM version, issue a warning? And if the version is older than XYZ, run the target upgrader.
>    

Sounds fine, but what exactly would we want to store there? Currently we 
store the ScummVM version there plus optionally the SVN revision. This 
behavior seems too inconsistent to use it directly IMHO. So we would 
need some proper versioning there too.

Apart it seems fine to issue a warning on a newer config file being 
found. In case the target upgrader really only upgrades the targets (and 
maybe stuff like the guioptions etc.), but not the game name etc. I'm 
having no strong feelings against running that one silently in case an 
too old config file version was found.

> (As an aside, an "engine" key would also make it possible to let multiple engines use the same gameid (think about LSL1 in both AGI and SCI). I am not sure whether it would be a good idea to open that can of worms, though, esp. for backward compatibility reasons.)
>    

I am not sure, but actually in my config file all SCI games I have added 
(KQ4-6) use only the "sci" gameid, on the other hand KQ1-3 are using 
"agi" as gameid, so I am not sure why you bring up LSL1 as an example? 
Maybe those games use a different gameid.... or maybe even nowadays 
there's a different gameid for them...

Personally I rather envision the gameid as some hint to the engine, so 
I'm not really against this, but I can understand that it might be bad 
for backwards compatibility.


> 3) Alternate approach
> ---------------------
> Another approach has been suggested in the past: Split the plugins into a pure detector part, and a part which runs the games. THe idea is that the detector plugins would be relatively small and could just all be loaded, while the full plugin would be loaded only when a game is actually started.
> This has its merits, but I am a bit reluctant to pursue this venue, as I feel it might complicate engine plugin code considerably. E.g. in the SCI engine, the detection code heavily release on the ResourceManaager class, which makes up a sizeable chunk of the actual engine. It seems difficult (and artificial) to separate those.
>    

This one definitely has its merits, i.e. less memory wasted when running 
the games because of all the detection related code being unload on runtime.

We will of course need proper versioning of the engine plugins then, so 
we can check whether a detection plugin matches the actual engine plugin.


> A limited version of this, however, might be doable and already solve most of our problems: We only really need the mapping from gameid to engine to solve most of our problems! So if we come up with a way to provide that to ScummVM... Some approaches: for every engine plugin, add ...
>    

Sounds nothing like for the long term to me though. I.e. we would 
require tons of extra tables (1 per engine at least) or the like to map 
that and thus for every gameid change (may it be updates etc.) we would 
need to also maintain those. That's rather annoying IMHO and I suspect 
it would be easier to just split the whole detection into a separate 
plugin then.

> a) ... a second mini-plugin which provides just this mapping
>    

Sounds best to me for this mini approach.

> b) ... a simple text file which provides this mapping
>    

Much too easy to manipulate IMHO ;-). I.e. a user could accidentally 
change that file for whatever reasons and it would thus result in some 
maybe not too nice behavior...

> c) ... a mapping table "gameid ->  engine" into the ScummVM binary itself
>    

This one kind of destroys the possibility of third party plugins, thus I 
am really against it. It will also add "useless" memory overhead for all 
those gameid -> engine entries IMHO.

> a) means doubling the plugin count, but keeps the code specific to one engine bundled in one place, while c) would spread some of it into the ScummVM core. Also c) means that only plugins known at compile time of the scummvm binary can be loaded. Approach b) is easy to maintain and require little code, but the downside is that it is more work to make sure it stays up-to-date.
>    

I guess you guys got my opinion on this one, just to make it clear: I 
rather tend to (a) if at all and I would even more so prefer the general 
detection / engine split (when it comes to approach 3).

> All three approaches bear the risk that one may end up with a mismatch between the "gameid->engine" table used and the engine versions used.
>    

Actually for (a) we could store a plugin version in both the mapping 
plugin and the engine plugin, in case that mismatches we could then 
throw a warning/error to the user. So (a) sounds best to me in that 
regard. (Like we would need to do for separating the detection from the 
engine code too).


> 4) Additional thoughts
> ----------------------
>
> That's the rough short version, anyway. Note that combinations of 1), 2) and 3) are of course possible!
>
> Some additional issues to keep in mind:
>
> * We currently support starting a game without a config entry, too: If you type
>    scummvm monkey2
> and have no monkey2 target configured, it will try to locate monkey2 in the current directory. We can't know the engine this way. Some "solutions":
> ** If approach 3 is realized in some way, this problem does not exist ;)
> ** Drop this feature (I don't think the feature is very important; then again, dropping a long standing feature just because your new code has difficulties supporting is nasty -- I don't like this :/)
> ** Require the user to specify a --engine=FOOBAR option -- bad from a user's perspective, I don't like this much either
> ** Fall back to the current mode of loading all plugins (possibly with the refinement from 1) above). Easy to do, but not perfect.
>    

I actually like this feature. Doesn't even the iPhSoft version use this 
for its Queen bundle for the iPhone btw.? (Not that I would consider 
this a major reason for keeping it though).

When it comes to our "old" command line features, is the --save-path 
stuff working correctly these days btw.? I remember it being broken a 
while ago.


> * We still need to support adding games in the launcher, including the mass add feature. We can just keep the current way: Load all plugins, then detect with each in turn. Or we do it iteratively: Load plugin 1, scan for games; load plugin 2, scan for games; etc.
> For a plain "Add game", both should work well. If "Mass Add" is used, though, then we may need to traverse many many directories; if we use an iterative scanning approach, we now have to iterate all those directories N times, where N is the number of engine plugins. Depending on the system, this may be a lot slower than loading all engines, then traversing the directory tree only once.
> I don't see an ideal solution for this right now, only various tradeoff. Of course one can argue that a huge "Mass Add" is something relatively rare, and probably mostly happens on "higher end" machines, so it's not so bad if it is slow... But maybe this thought is wrong? Porters?
>    

I am not sure, but what I remember from the WinCE port, it actually 
rather asked you for a mass add when you tried to use Add game... My 
memory might be wrong though, it's been some time since I used the WinCE 
port.

Actually we could probably do some optimization here: For example we 
might build up some tree of possible game directories first. For that we 
might drop all empty directories from the search. We might also want to 
drop all directories from the search which only have subdirectories and 
no actual files, thus only doing a detection try on directories with at 
least on file. Of course the last one is a bit problematic when we 
actually have an engine for a game, which only uses directories in its 
top level directory (Is that possible? Sensible? Do we have such a case? 
Do we know of such a case?). Next we might also remove all directories 
(and maybe its children, of course that would force the user to have a 
certain directory structure), which already contain a setup game (Do we 
allow multiple games in one directory?). Since we can do that all in 
advance before running the actual detection code, that could lower the 
access to the file system from the detection code from zero to a lot, 
depending on the directory structure of the user.

Anyway before we really know how slow it is (and even more so why it is 
slow), we should not worry too much about premature optimizations.

> * We need to still support "--list-games" which may require loading all plugins (but could be mitigated by approach 3)
>    

Even with approach 2 we could only load one at a time (i.e. using 
approach 1 there) and will thus not be that heavy on memory usage, which 
I consider the biggest factor for the whole plugin code.

Also since I guess actually "--list-games" might only be used really on 
desktop systems, I do consider this a less important factor for deciding 
on which approach to use.

// Johannes




More information about the Scummvm-devel mailing list