[Scummvm-devel] Avoiding loading of all plugins during startup

Max Horn max at quendi.de
Thu Jun 24 12:12:21 CEST 2010


[brief note to start with: I am traveling through europe till sunday and can only infrequently access the internet, hence I'll only infrequently answer emails. So I am sorry if my replies seem a bit disconnected from the active discussion, as they may arrive with a day or more delay... :) ]

Am 23.06.2010 um 19:50 schrieb Johannes Schickel:

> On 06/23/2010 11:54 AM, Max Horn wrote:
>> 1) Quick and simple refinement
>> ------------------------------
>> 
>> A very simple way to improve this situation slightly is to not load all engine plugins at once, but rather load one, ask it whether it supports the given game; if not, unload it and proceed to the next. This way only one plugin is held in memory and fragmentation should be kept to a minimum. But we still have an overhead (as in the worst case we still have to load all plugins). Might be worse investigating this as a quick temporary measure, though.
>> 
> 
> Sounds fine for a temporary solution to me. I would imagine this could 
> be used to first focus on implementing the plugin loaders and test them 
> (for example on NDS, where we know of serious memory limitations ;-). 
> Though I would be rather against merging this into the mainline later on.

Hm, and why?

> 
> 
>> 2) Suggested long term approach
>> -------------------------------
>> 
>> On the long run, I propose this: We add to every game target in the config file an "engine=BLABLA" key-value pair. This way, we can pick the right engine plugin right away. To ease transition to this new format, there are several ways: One is that if the "engine=" field is missing, we just revert to the old behavior. Another is that during startup, we scan through all config targets, and if we find any missing the "engine" key, we try to add it. By re-running the detector, matching the output with the gameid that is already there, and setting up the engine field. For this, the upgradeTargets()  function in base/commandLine.cpp could be adapted.
>> 
> 
> Sounds fine to me. We might also use that engine key to check whether 
> the plugin for a given target really exists.

Of course. And if it doesn't, show an according error message.

> 
> I have one question though: what exactly do we store there? The base 
> name of the engine? If so what will we do about engine renaming (like it 
> happened with simon -> agos in the past)?

Foremost, we'd try to avoid it ;) but if it happens, we do the same what we've done with engine renames: Keep a mapping table from old to new engine somewhere. In ScummVM itself, for example. Not beautiful, but simple and effective.


> 
> 
>> In order to avoid having to do this re-scanning every time, I would suggest also finally putting the "version" field in our config file to some use: Right now it is worthless, because we simply set it to the ScummVM version whenever we load a config file. We could change that: If loading a config file made with a newer ScummVM version, issue a warning? And if the version is older than XYZ, run the target upgrader.
>> 
> 
> Sounds fine, but what exactly would we want to store there? Currently we 
> store the ScummVM version there plus optionally the SVN revision. This 
> behavior seems too inconsistent to use it directly IMHO. So we would 
> need some proper versioning there too.

Exactly. My idea would be that we use a simple integer value, as for savegames in SCUMM, Tinsel, AGI,... which we increment when we feel the need for it. And if we notice that "version" is *not* a simple integer, we can assume that an old ScummVM version last wrote that config file.


> Apart it seems fine to issue a warning on a newer config file being 
> found. In case the target upgrader really only upgrades the targets (and 
> maybe stuff like the guioptions etc.), but not the game name etc. I'm 
> having no strong feelings against running that one silently in case an 
> too old config file version was found.

The upgrader would of course not touch the description, only the gameid/engine/guioptions. Well, we *could* add a way to allow the user to explicitly request an update for the description, too, but again, may intention was to have this at most optionally.

> 
>> (As an aside, an "engine" key would also make it possible to let multiple engines use the same gameid (think about LSL1 in both AGI and SCI). I am not sure whether it would be a good idea to open that can of worms, though, esp. for backward compatibility reasons.)
>> 
> 
> I am not sure, but actually in my config file all SCI games I have added 
> (KQ4-6) use only the "sci" gameid, on the other hand KQ1-3 are using 
> "agi" as gameid, so I am not sure why you bring up LSL1 as an example? 
> Maybe those games use a different gameid.... or maybe even nowadays 
> there's a different gameid for them...

Right, my fault, I forgot that those engines were switched to use a single gameid!  With all its pros and cons... E.g. Pro: Reduces pressure on the gameid namespace, reduces inconsistencies between what the config file claims and what is really there, no more troubles when a gameid changes... Con: Forces the engine to redetect the game each time it is started.

> 
> Personally I rather envision the gameid as some hint to the engine, so 
> I'm not really against this, but I can understand that it might be bad 
> for backwards compatibility.


Yes, we could make the gameid completely optional and only rely on the engine "id". But as you correctly point out,  that would hamper backward compatibility, so I'd not consider this until we've had "engine=" support in at least one major release.

(BTW, maybe the key should be "engineid" for consistency with "gameid" ?


> 
> 
>> 3) Alternate approach
>> ---------------------
>> Another approach has been suggested in the past: Split the plugins into a pure detector part, and a part which runs the games. THe idea is that the detector plugins would be relatively small and could just all be loaded, while the full plugin would be loaded only when a game is actually started.
>> This has its merits, but I am a bit reluctant to pursue this venue, as I feel it might complicate engine plugin code considerably. E.g. in the SCI engine, the detection code heavily release on the ResourceManaager class, which makes up a sizeable chunk of the actual engine. It seems difficult (and artificial) to separate those.
>> 
> 
> This one definitely has its merits, i.e. less memory wasted when running 
> the games because of all the detection related code being unload on runtime.
> 
> We will of course need proper versioning of the engine plugins then, so 
> we can check whether a detection plugin matches the actual engine plugin.
> 
> 
>> A limited version of this, however, might be doable and already solve most of our problems: We only really need the mapping from gameid to engine to solve most of our problems! So if we come up with a way to provide that to ScummVM... Some approaches: for every engine plugin, add ...
>> 
> 
> Sounds nothing like for the long term to me though. I.e. we would 
> require tons of extra tables (1 per engine at least) or the like to map 
> that and thus for every gameid change (may it be updates etc.) we would 
> need to also maintain those. That's rather annoying IMHO and I suspect 
> it would be easier to just split the whole detection into a separate 
> plugin then.

Hm, while I am not *that* fond of this approach myself, I think this is not at all that bad. :)

Most engines already have a table of gameids they support, and it would be a simple matter of duplicating that table. Not much work at all. A simple approach to do that: Require all engines to provide a gameid-table.h file of a consistent format. That table can then be shared by the engine plugin, and X. Where X is: a separate plugin with the list of gameids; or the main executable could include them; or a separate tool could turn them into text files (resp. conversely: A separate tool could create the gameid-table.h files from a text file).
All of these could be implemented in very short time.


This approach is not beautiful, but it's simple, easy to implement, and has very low overhead. And it helps a lot to ease transition to the "engine=" config key system. 


> 
>> a) ... a second mini-plugin which provides just this mapping
>> 
> 
> Sounds best to me for this mini approach.
> 
>> b) ... a simple text file which provides this mapping
>> 
> 
> Much too easy to manipulate IMHO ;-). I.e. a user could accidentally 
> change that file for whatever reasons and it would thus result in some 
> maybe not too nice behavior...

Well, yes, but then again, if a user choose to shot themselves into the foot... E.g. by removing the second mini-plugin, or by mismatching various versions of the plugins, or, or, or -- if we follow that argument, we should not provide engine plugins at all :).


> 
>> c) ... a mapping table "gameid ->  engine" into the ScummVM binary itself
>> 
> 
> This one kind of destroys the possibility of third party plugins, thus I 
> am really against it. It will also add "useless" memory overhead for all 
> those gameid -> engine entries IMHO.

I agree on your first argument (though I don't consider the memory overhead as such a big problem -- the mini-plugins will take up a lot more memory, too ;). Anyway, I mostly listed it for completeness :).



I am going to stop my mail here, it's already too long. I may reply to some of the other points you mention later on individually, but i should first go through the rest of this thread ;).


Cheers,
Max



More information about the Scummvm-devel mailing list