[Scummvm-devel] Avoiding loading of all plugins during startup
Johannes Schickel
lordhoto at scummvm.org
Wed Jun 23 19:50:32 CEST 2010
On 06/23/2010 11:54 AM, Max Horn wrote:
> 1) Quick and simple refinement
> ------------------------------
>
> A very simple way to improve this situation slightly is to not load all engine plugins at once, but rather load one, ask it whether it supports the given game; if not, unload it and proceed to the next. This way only one plugin is held in memory and fragmentation should be kept to a minimum. But we still have an overhead (as in the worst case we still have to load all plugins). Might be worse investigating this as a quick temporary measure, though.
>
Sounds fine for a temporary solution to me. I would imagine this could
be used to first focus on implementing the plugin loaders and test them
(for example on NDS, where we know of serious memory limitations ;-).
Though I would be rather against merging this into the mainline later on.
> 2) Suggested long term approach
> -------------------------------
>
> On the long run, I propose this: We add to every game target in the config file an "engine=BLABLA" key-value pair. This way, we can pick the right engine plugin right away. To ease transition to this new format, there are several ways: One is that if the "engine=" field is missing, we just revert to the old behavior. Another is that during startup, we scan through all config targets, and if we find any missing the "engine" key, we try to add it. By re-running the detector, matching the output with the gameid that is already there, and setting up the engine field. For this, the upgradeTargets() function in base/commandLine.cpp could be adapted.
>
Sounds fine to me. We might also use that engine key to check whether
the plugin for a given target really exists.
I have one question though: what exactly do we store there? The base
name of the engine? If so what will we do about engine renaming (like it
happened with simon -> agos in the past)?
> In order to avoid having to do this re-scanning every time, I would suggest also finally putting the "version" field in our config file to some use: Right now it is worthless, because we simply set it to the ScummVM version whenever we load a config file. We could change that: If loading a config file made with a newer ScummVM version, issue a warning? And if the version is older than XYZ, run the target upgrader.
>
Sounds fine, but what exactly would we want to store there? Currently we
store the ScummVM version there plus optionally the SVN revision. This
behavior seems too inconsistent to use it directly IMHO. So we would
need some proper versioning there too.
Apart it seems fine to issue a warning on a newer config file being
found. In case the target upgrader really only upgrades the targets (and
maybe stuff like the guioptions etc.), but not the game name etc. I'm
having no strong feelings against running that one silently in case an
too old config file version was found.
> (As an aside, an "engine" key would also make it possible to let multiple engines use the same gameid (think about LSL1 in both AGI and SCI). I am not sure whether it would be a good idea to open that can of worms, though, esp. for backward compatibility reasons.)
>
I am not sure, but actually in my config file all SCI games I have added
(KQ4-6) use only the "sci" gameid, on the other hand KQ1-3 are using
"agi" as gameid, so I am not sure why you bring up LSL1 as an example?
Maybe those games use a different gameid.... or maybe even nowadays
there's a different gameid for them...
Personally I rather envision the gameid as some hint to the engine, so
I'm not really against this, but I can understand that it might be bad
for backwards compatibility.
> 3) Alternate approach
> ---------------------
> Another approach has been suggested in the past: Split the plugins into a pure detector part, and a part which runs the games. THe idea is that the detector plugins would be relatively small and could just all be loaded, while the full plugin would be loaded only when a game is actually started.
> This has its merits, but I am a bit reluctant to pursue this venue, as I feel it might complicate engine plugin code considerably. E.g. in the SCI engine, the detection code heavily release on the ResourceManaager class, which makes up a sizeable chunk of the actual engine. It seems difficult (and artificial) to separate those.
>
This one definitely has its merits, i.e. less memory wasted when running
the games because of all the detection related code being unload on runtime.
We will of course need proper versioning of the engine plugins then, so
we can check whether a detection plugin matches the actual engine plugin.
> A limited version of this, however, might be doable and already solve most of our problems: We only really need the mapping from gameid to engine to solve most of our problems! So if we come up with a way to provide that to ScummVM... Some approaches: for every engine plugin, add ...
>
Sounds nothing like for the long term to me though. I.e. we would
require tons of extra tables (1 per engine at least) or the like to map
that and thus for every gameid change (may it be updates etc.) we would
need to also maintain those. That's rather annoying IMHO and I suspect
it would be easier to just split the whole detection into a separate
plugin then.
> a) ... a second mini-plugin which provides just this mapping
>
Sounds best to me for this mini approach.
> b) ... a simple text file which provides this mapping
>
Much too easy to manipulate IMHO ;-). I.e. a user could accidentally
change that file for whatever reasons and it would thus result in some
maybe not too nice behavior...
> c) ... a mapping table "gameid -> engine" into the ScummVM binary itself
>
This one kind of destroys the possibility of third party plugins, thus I
am really against it. It will also add "useless" memory overhead for all
those gameid -> engine entries IMHO.
> a) means doubling the plugin count, but keeps the code specific to one engine bundled in one place, while c) would spread some of it into the ScummVM core. Also c) means that only plugins known at compile time of the scummvm binary can be loaded. Approach b) is easy to maintain and require little code, but the downside is that it is more work to make sure it stays up-to-date.
>
I guess you guys got my opinion on this one, just to make it clear: I
rather tend to (a) if at all and I would even more so prefer the general
detection / engine split (when it comes to approach 3).
> All three approaches bear the risk that one may end up with a mismatch between the "gameid->engine" table used and the engine versions used.
>
Actually for (a) we could store a plugin version in both the mapping
plugin and the engine plugin, in case that mismatches we could then
throw a warning/error to the user. So (a) sounds best to me in that
regard. (Like we would need to do for separating the detection from the
engine code too).
> 4) Additional thoughts
> ----------------------
>
> That's the rough short version, anyway. Note that combinations of 1), 2) and 3) are of course possible!
>
> Some additional issues to keep in mind:
>
> * We currently support starting a game without a config entry, too: If you type
> scummvm monkey2
> and have no monkey2 target configured, it will try to locate monkey2 in the current directory. We can't know the engine this way. Some "solutions":
> ** If approach 3 is realized in some way, this problem does not exist ;)
> ** Drop this feature (I don't think the feature is very important; then again, dropping a long standing feature just because your new code has difficulties supporting is nasty -- I don't like this :/)
> ** Require the user to specify a --engine=FOOBAR option -- bad from a user's perspective, I don't like this much either
> ** Fall back to the current mode of loading all plugins (possibly with the refinement from 1) above). Easy to do, but not perfect.
>
I actually like this feature. Doesn't even the iPhSoft version use this
for its Queen bundle for the iPhone btw.? (Not that I would consider
this a major reason for keeping it though).
When it comes to our "old" command line features, is the --save-path
stuff working correctly these days btw.? I remember it being broken a
while ago.
> * We still need to support adding games in the launcher, including the mass add feature. We can just keep the current way: Load all plugins, then detect with each in turn. Or we do it iteratively: Load plugin 1, scan for games; load plugin 2, scan for games; etc.
> For a plain "Add game", both should work well. If "Mass Add" is used, though, then we may need to traverse many many directories; if we use an iterative scanning approach, we now have to iterate all those directories N times, where N is the number of engine plugins. Depending on the system, this may be a lot slower than loading all engines, then traversing the directory tree only once.
> I don't see an ideal solution for this right now, only various tradeoff. Of course one can argue that a huge "Mass Add" is something relatively rare, and probably mostly happens on "higher end" machines, so it's not so bad if it is slow... But maybe this thought is wrong? Porters?
>
I am not sure, but what I remember from the WinCE port, it actually
rather asked you for a mass add when you tried to use Add game... My
memory might be wrong though, it's been some time since I used the WinCE
port.
Actually we could probably do some optimization here: For example we
might build up some tree of possible game directories first. For that we
might drop all empty directories from the search. We might also want to
drop all directories from the search which only have subdirectories and
no actual files, thus only doing a detection try on directories with at
least on file. Of course the last one is a bit problematic when we
actually have an engine for a game, which only uses directories in its
top level directory (Is that possible? Sensible? Do we have such a case?
Do we know of such a case?). Next we might also remove all directories
(and maybe its children, of course that would force the user to have a
certain directory structure), which already contain a setup game (Do we
allow multiple games in one directory?). Since we can do that all in
advance before running the actual detection code, that could lower the
access to the file system from the detection code from zero to a lot,
depending on the directory structure of the user.
Anyway before we really know how slow it is (and even more so why it is
slow), we should not worry too much about premature optimizations.
> * We need to still support "--list-games" which may require loading all plugins (but could be mitigated by approach 3)
>
Even with approach 2 we could only load one at a time (i.e. using
approach 1 there) and will thus not be that heavy on memory usage, which
I consider the biggest factor for the whole plugin code.
Also since I guess actually "--list-games" might only be used really on
desktop systems, I do consider this a less important factor for deciding
on which approach to use.
// Johannes
More information about the Scummvm-devel
mailing list