[Scummvm-devel] File handling revamp

[Max Horn]

Am Fr, 8.08.2008, 07:33, schrieb sunmax at libero.it:
> Hi there ScummVM Team!
>
> Ahhh files! (it sounds like a porn ;-)
>
> I hope I am not too late to drop my 2 cents...
>
> ... I am just catching up with 2 weeks of FSNode redesign interesting
> e-mails. I copied and pasted the parts I thought were most important
> and added some notes and drop some ideas.
>
> Here they come!
>
>
>>> This approach makes it easy to dynamically add/remove dirs from the
>>> search list (just dump the SearchDirectory object); or to search for
>>> files only in a subset of all search dirs.
>
> In how many dirs do we need to search for a file? ;-)
>
> I thought it was just one (or at least one per kind: gamedata,
> savedata, config), and its subdirs of course.

global extra path (for add-on data files); game specific extra path;
themepath. Also some games have their data in a single dir in one version
and distributed over various subdirs in other versions; so for these
games, it is convenient for the engine to be able to just say: "Look for
this file in PATH , in PATH/SUBDIR1 (if it exists), ... etc.


>
> Why do we provide dedicated code for "savedata" and "config files" ?


If you mean: Why do we treat savedata & config files different than normal
files:

Because on some ports, they are implemented in a completly different
fashion (like on some consoles, savefiles are stored on memory cards with
little memory, while game files are loaded from big read-only storage
devices). Also, we sometimes want to dump debug data into files (like
resource dumps, screenshots, log files, ...); on desktops, those go into
regular files; on console ports, these might simply be ignored (these
devices don't have space to waste for debug stuff).

Hence, it is very helpful for porters to know in advance what kind of file
they are dealing with. Hence the desire to be able to treat them
differently. In the past (and to some degree even now), porters sometimes
had to do very very complicated things to correctly "guess" in their code
how a certain file was going to be used by calling code. Not nice.


OTOH, maybe you meant to ask why config files are not just save files?
Well, on consoles, they may end up in the same spot, but on desktops,
savefiles are stored in a fixed savepath, while config files can be loaded
from arbitrary locations; in particular, the default location is almost
always *different* from the savepath. In particular, the savepath is not
known before some config file has been loaded.

Hence, we have a single *default* config file (now, after my OSystem
changes), which, if the port desires so, may be modelled via a savefile;
but on desktop machines, it just is a regular file, not a savestate.


>
>
>>> easier to implement read buffering. If desired, we could even try
>>> to provide some generic read buffering code.
>
> I will check the current implementation on the PS2 backend which I
> inherited and get back on this point later.

I think the PS2 does do some read buffering. But it looks to be very
optimized code, which takes things into account like whether anything else
is peforming a read right now; things which the common BufferedReadStream
class can't do. So, maybe it's best to keep that code... Well, or not,
you'll have to determine that. :)



>
>
>>> opening files in directories: case (in-)sensivity.
>
> I know it may sound radical and fascist, but since we have do "Add"
> the game anyhow, why don't we just refactor the filename cases at
> that stage, I mean rename all of them to the only one case we allow,
> let's say all lowercase? (or whatever the platform dictates).

One reason: We'd then have to implement file renaming in a platform
independant fashion. Much more work than just ignoring case in some spots.
Also: If I was the user, I'd *not* be happy if ScummVM messed with my data
files for no good reason... :)

>
> The only issue would be for games played from CD where we cannot
> rename the media, but how many people out there are doing this?
> (well, I actually am on PS2 ;-).
>

Actually, that is the main reason. On some platforms (e.g. the Dreamcast),
the game data is *always* read-only. And yeah, people may play the games
directly from CD, even if we don't recommend this. Hence, we would still
have to support systems which don't support renaming the files. Which
means no work is saved at all with your proposal :)

Hence, to sum up, it makes no sense to attempt to rename files, unless we
can guarantee that it is *always* possible to do so, which we can't.


> We could even provide on the website games data already with the
> correct case.

For most games, we can't do that since the games are not freely available.


>
>
>>> FSList
>
> I think we should go for a FSMap instead: have a key used by the
> engines to locate the data (e.g. "res0.dat") and match it to a
> FSNode.

That is more or less what peres' GameDirectory class amounts to, only that
it also allows things like using a simple wildcard pattern.

[...]

>>> A SearchDirectory object sounds like a good idea
>
> What would be the advantage of a SearchDirectory object over
> a simple Map?

Depends on the details you envision; either they are just different names
for the same thing. Or the SearchDir/GameDir adds some nice extra
features, like allowing you to transparently access files spread over
multiple dirs w/o having to worry about the details.


[...]

>>> Btw, if we are going to redo the fs, I'd again like to propose
>>> having different calls for creating a file node and a directory
>>> node
>
> Why do we need a "directory node" at all, when we have a full list of
> cached nodes contained in "root" (ie. gamedata folder) ?

Well, we still need to be able to navigate the filesystem to be able to
show a "File/folder chooser" dialog, e.g. when doing "Add Game...".

Engines are only one of the clients for the FSNode stuff :)



[...]

>
> A FSNode is a pretty small entity, and there is a limited amount of
> files in the few games I played, I would consider to TOC-cache them
> all at start (I mean all the ones in gamedata folder), so that we
> don't have to bother anymore later. Everything in -the list- exists
> and it's reachable with a lowercase key.


The GameDir would do *exactly* this caching for the stuff in the gamepath
(and extrapaths, etc.) -- but we can't just cache everything below the
root, at least I wouldn't want ScummVM to wait some minutes on my system
to cache entries for half a million files ... :).


>
> I know, I am repeating myself ;-)
>
>
>>> Search in the global list of default seach paths
>
> We should have only a search path active at a time. When we play the
> game is "gamedata dir", when we save a game is "savedata dir", and so
> on. I don't think we need to search all of them all the times, right?

See above, there are more paths to be searched. Specific example> You have
a CD inserted with the game files from Kyrandia. You have a game specific
extrapath which points to a dir on your HD with kyra.dat, a file provided
by ScummVM, which is required to play kyra. Finally, you have the master
extrapath pointing to another dir which contains the ROM files required to
use the MT-32 emulator. Also, you have a themedir set to use your
uber-funky extra cool GUI themefile. Finally, also the current dir is
searched (because those MT-32 ROM files, or your custom theme file, or
your virtual keyboard pack file, etc., could all be located there, too).

(I admit that the "game specific extrapath" really doesn't make to much
sense to me, and we could drop it, but right now it's there and so I
mentioned it for completeness).

You may rightfully argue now that the engine shouldn't have to search all
of those, e.g. the themepath. I agree! But the GUI code does have to
search that path. Problem is, our current code has one single search path
system: If any code wants to be able to transparently load files from both
a special path, and the current dir, for example, then it *has* to use
that central system. And so, since the GUI wants to find all files in its
themepath, the engine ends up searching that as well...

OK, I didn't explain this too well, and my description is a bit convoluted
but so is the current system: Which is why we want to overhaul it :).


Hence the wish to have searchpaths / gamedirs *objects*, which can be used
locally only: So the GUI code creates a searchpath list containing say the
themedir and current dir (and maybe also the global extra dir); the engine
adds game dir, extra dir & current dir to its search list (and whatever
subdirs it likes); the MT-32 emu searches extra dir & current dir; etc.

Also, this system makes it possible to define a clear list of priorities:
E.g. first look in the game dir, then in extra dir, only last in current
dir (or not at all in current dir, etc.).

All of this can't be done with the current File::addDefaultDirectory system.

[...]


>>> 1) f.open("foo/bar.dat");
>
> Is there any reason why do we allow that? Are there multiple files with
> the same name at different level in the same game?

Yes!

[...]
>>> When reading files, we want to search them everywhere.
>
> How could they be outside gamedata when you are playing the game?

Mainly to support files like kyra.dat, sky.cpt, etc. (which can be stored
in the game path, or in another central location -- both setups make
sense!).

[...]

>
>>> We could use the full path provided by FSNode as the key into the
>>> HashMap.
>
> I would just use the keyword (ie. the resource name) that the engine
> is going to access, and map it to a full FSNode, which then contain
> the full path of the file.
>
>  HashMap<String, FSNode>
>            |
>            |
>            ---- lowercase resource key used by engines

I think that's how the "cache" of the GameDir object peres' proposed would
do, so besides differing names, and some extra features, we are not really
disagreeing here, it seems (at least AFAICT).

[...]

Cheers,
Max