[Scummvm-devel] PlayStation2 : files caching, TOC & data

[Max Horn]

Am 10.07.2008 um 07:42 schrieb sunmax at libero.it:

> A-ok scummvm-team,
>
> one more technical e-mail then time for bed!
>
> The focus here is on games data and caching strategy for scummvm
> console backends, with special attention to PlayStation2.
>
> We have 3 kinds of data:
>
> * games data (original or re-mastered, e.g. a mp3 instead of audio
>  track)
[...]
> * user data (config files, GUI themes, saved games)
[...]
> * special data (I am only aware of kyra.dat)

There are more, see "dists/engine-data" in trunk:
  drascula.dat  igor.tbl  kyra.dat  lure.dat  m4.dat  queen.tbl  sky.cpt

>   -> these were not in the original game set of data, so the game
>      CD won't be enough, I will need this file too. For games
>      installed from a CD we could just copy it over (or if available
>      for a certain version of scummvm on the net we could just
>      dowlnload it on the fly while installing). But if we play the
>      game from CD (and we don't want to burn a new one with the
>      originala data + this file) we whould make this 'special data'
>      file accessible in some other way.

Well, those data files correspond to specific engines/games, but at  
the same time, they are also tied to a certain ScummVM version. So the  
executable should be bundled together with them. In the Mac OS X  
version, we copy those files into the ".app" bundle, together with the  
executable and some other files. I don't know how the PS2 port ships/ 
installs the binary, but ideally, if one installs a version of  
ScummVM, one should just get the matching versions of these files  
automatically. That way, nothing can go wrong.

In some sense, it might be good to actually "include" these files into  
the binary, but we don't do that because it would increase its size by  
several MB, but not all users even have those games...

> But let's keep it simple. When a game starts everything we need is
> going to be in:
>
> - gamedata
>
> - userdata
>
> Now only the first one need to be cached. This is very important for
> slower media like CD/DVD on PS2 or playing from host: (i.e. a CD
> folder accessed from the network). There are two things that may
> be cached:
>
> - TOC
>
> - data
>
> The first one is a must cause we cannot check every time after we
> started the game if the file exists by trying to access to it, etc.
>
> And starting from 0.11.0 with all thos .exists() & .isDirectory() it
> happens a lot. On more powerful architecture the OS abstract part of
> the caching logic, but on consoles (at least on PS2) you have to do
> it yourself.

Well, the idea/hope there was/is that with exists/isDirectory, and a  
port specific FSNode class it should be a lot easier for a port to  
cache TOC data then previously -- you can simply implement TOC caching  
within the FSNode framework. At least I'd hope so... assuming memory  
is not too tight (but TOC data does not take that much memory, does  
it?).


> On the PS2 fopen (which is redefined in common/file.cpp as ps2_fopen)
> implements the second logic (data caching), where File &  
> FilesystemNode
> implement a TOC. A simpler kind of TOC too, is implemented internally
> to ps2_fopen (<-fopen), so we have kind-of a messy duplication.

That *is* messy. And one of the reasons why I'd like to refactor  
Common::File: Instead of forcing porters to emulate fopen, let porters  
override the internals of Common::File. That is, instead of doing a  
"media break" (by converting FSNodes to a path which then is passed on  
fopen), let ports implement a function which "opens" a file at a  
specified FSNode. The default would do what we do now (convert to a  
path, call fopen), but porters could implement it differently.

Some extra things:
* We do not use fopen directly, we wrap it in fopenNoCase(), which is  
there to ignore case -- this does not use FSNode, and can not be  
easily override by porters right now. It is important that client code  
trying to open "FOO.DAT" also finds "foo.dat", but the way we  
implement this needs to be changed

* Client code (read: our engines) when using files most often wants to  
do it like this: File.open("FOO.DAT"): i.e. they just specify a file  
name. We don't want to change that to FSNodes, but that's fine.
However, we do support searching for files in multiple dirs via  
"File::addDefaultDirectory" et al. These functions also create a file  
cache of sorts, by recursively scanning all dirs for files and caching  
that.

So the current File::open code is extremely convoluted and various  
things (like TOC caching, or code for ignoring case) are implemented  
several times in different ways. Bad. A rewrite of all this would be a  
great thing, I believe, and help future & current ports.


> On the other hand the actual common File / FileSystem which are used
> by the engines don't let me cache data as far as I know, you may see
> Ps2ReadFile::Ps2ReadFile for the way this was implemented on PS2 by
> previous coders, for the ps2_fopen case (But not the File::* case).
>
> So my question is: how do I get the best of the 2 worlds?
>
> How do I cache a TOC + some data? Should I extend File? Should I
> bypass it, in the way ps2_fopen does with fopen?


I would suggest that we simply work on reimplementing File. We need to  
discuss how, with all porters and other folks, though.



Cheers,
Max