[Scummvm-devel] Progress with Archive & SearchManager

[Max Horn]

Hi folks,

it would be really nice to finally get the whole Archive &  
SearchManager business completed, so that we can start definite bug &  
regression hunting and convert all ports to the new system. There are  
two open tracker items, with pending patches, but overall, I would  
like to discuss where we are, where we want to go and what we need for  
that.

We successfully reached most of the goals outlined in the "File  
handling revamp" thread. The following list is taken from my local  
TODO file:

* Split off DumpFile from File
  -> DONE
* add Buffered(Seekable)ReadStream
  -> DONE
* add config file load/save API to OSystem
  -> DONE
* make FSNode case insensitive ??
  -> REJECTED
* add openForReading()/openForWriting() methods to FSNode
  -> DONE
* change File::open to use those method
  -> DONE
* add POSIX path parser for use in File::open
  -> pending, probably REJECTED as we will phase out File completely
* change Stream ioFailed() semantics to match POSIX ferror
  -> DONE
* SearchDirectory ("SearchPath"?) class which caches all files in a  
dir (and its subdirs?)
  -> in progress
* add docs to class File
  -> in progress

Some extra stuff cropped up in the meantime:
* convert .ZIP access code to the new Archive API
  -> DONE
* convert .ARJ access code to the new Archive API
  -> pending, help welcome!
* convert ScummFile class(es) to new Archive API
  -> pending, help welcome!


One of the pending patches is about adding a new ArchiveMember class,  
which are primarily meant to simplify things in Kyra, if I recall it  
right. Look at patch #2034983 for details. I think this patch by peres  
is mostly ready to be checked in; although I must admit I do have some  
doubts whether this is the best way to go (by introducing the  
ArchiveMember class, we add another indirection layer, which on the  
one hand may simplify some client code, but on the other hand makes it  
quite a bit harder to implement an Archive subclass; and I am not yet  
sure whether this is a good tradeoff. But I think it's fine to keep  
working a bit longer on this particular patch.

The main missing piece, in my eyes, is finishing the SearchManager.  
This is a singleton which is meant to manager the "search paths" of  
ScummVM; think of it as the successor to the  
File::addDefaultDirectory() API. The quicker we get this finished, the  
sooner engines can be converted to the new APIs.

The first step to this would be to initialize SearchMan properly, in  
base/main.cpp and elsewhere; look at Patch #2093502 for some work on  
that, and various comments for what one has to watch out. A good start  
for this might be to convert class File to use SearchMan instead of  
its own s_searchSet. But there is a catch: namely when/where to call  
OSystem::addSysArchivesToSearchSet. Maybe we need to add a reset()  
method to SearchMan, and then let that method invoke  
OSystem::addSysArchivesToSearchSet.

By following this approach, it gets easier to migrate Engines  
*gradually* away from the File::addDefaultDirectory() API. This would  
be the next step. First, convert File::addDefaultDirectory calls. Then  
replace anything which does "new File" to use SearchMan::openFile  
instead. Finally, consider replacing all uses of File (but let's not  
rush that). Oh yeah, one thing we must watch out here is that class  
File has a hack which makes it identify the filenames "FOO" and  
"FOO."; this is, AFAICT, for Simon's GAMEPC file only, so my hope is  
that we can push this hack to the AGOS engine.

In parallel, other code should be converted away. For example, the GUI  
code, the MT-32 emulator, anything which uses class File in some way.  
By properly using the Archive class, there is even some potential to  
considerably simplify some code which right now tries to open stuff  
both via File::open and ZIPArchive -- this can be merged into a single  
call by using our fancy new "SearchSet" class (see common/archive.h).



Cheers,
Max

P.S.: originally I meant to send this mail to LordHoto and peres  
alone, but I figured it wouldn't hurt to send it to the public, in  
cases others want to chime in -- as highly unlikely as it is :)