[Scummvm-tracker] [ScummVM :: Bugs] #13276: AGS: Cannot open files with non-ASCII characters in their name

[ScummVM :: Bugs]

#13276: AGS: Cannot open files with non-ASCII characters in their name
-------------------+--------------------------
Reporter:  criezy  |       Owner:  (none)
    Type:  defect  |      Status:  new
Priority:  normal  |   Component:  Engine: AGS
 Version:          |  Resolution:
Keywords:          |        Game:
-------------------+--------------------------
Comment (by criezy):

 I now suspect the encoding used is CP 1252, which apparently is what is
 used by Windows for the file name encoding (at least in most Western
 countries), rather than ISO-8859-1.

 I have also verified that the same issue exists with the standalone AGS
 interpreter. But I suspect it might work on Windows, although maybe not on
 Russian or Japanese Windows for example.

 Here is the call stack where the error occurs:
 {{{
   frame #0: AGS3::init_translation(lang, fallback_lang,
 quit_on_error=false) at translation.cpp:61:6
   frame #1: AGS3::Game_ChangeTranslation(newFilename="Fran\xe7ais") at
 game.cpp:725:6
   frame #2: AGS3::Sc_Game_ChangeTranslation(params, param_count=1) at
 game.cpp:1491:2
   frame #3: AGS3::ccInstance::Run(this, curpc=0) at
 cc_instance.cpp:1023:20
   frame #4: AGS3::ccInstance::CallScriptFunction(this,
 funcname="hFrancais_AnyClick", numargs=0, params) at
 cc_instance.cpp:341:15
   frame #5: AGS3::RunScriptFunctionIfExists(sci,
 tsname="hFrancais_AnyClick", numParam=0, params) at script.cpp:363:32
   frame #6: AGS3::RunTextScript(sci, tsname="hFrancais_AnyClick") at
 script.cpp:412:14
   frame #7: AGS3::RunScriptFunction(sc_inst=kScInstRoom,
 fn_name="hFrancais_AnyClick", param_count=0, p1, p2) at script.cpp:269:4
   frame #8: AGS3::post_script_cleanup() at script.cpp:554:3
   frame #9: AGS3::RunScriptFunctionIfExists(sci,
 tsname="hFrancais_AnyClick", numParam=1, params) at script.cpp:380:2
 }}}

 So basically `Game_ChangeTranslation` gets called from a script with the
 filename of the translation to use. I cannot see any encoding conversion,
 so it looks like the script is expected to provide the filename in
 whatever encoding is used by the OS. Since the game was only released for
 Windows (and since that is the main system targeted by AGS), it makes
 sense to assume CP 1252 encoding.

 I also checked that the AGS documentation for `Game_ChangeTranslation` and
 for the `File` class does not mention anything about the encoding.
 https://adventuregamestudio.github.io/ags-
 manual/Game.html#gamechangetranslation
 https://adventuregamestudio.github.io/ags-manual/File.html

 I also suspect that it doesn't work on Windows in ScummVM. If I understand
 properly the following two PRs, the ScummVM FS code now expects filenames
 in UTF-8 on Windows:
 https://github.com/scummvm/scummvm/pull/3003
 https://github.com/scummvm/scummvm/pull/3122

 UTF-8 is also what is used on macOS, and most of the time on Linux I
 think.

 So maybe we should assume CP 1252 in `Game_ChangeTranslation`, assume we
 need UTF-8 in ScummVM, and do the encoding conversion for the file name?
 That would not impact translation set in our GUI as it calls directly
 `init_translation `.
 However are those assumptions correct? Could a Russian game for example
 use a different encoding if it has file names with non ASCII characters?
 We could make an assumption on the encoding based on the game language if
 that is needed. But for translations that would be useless as then it is
 multilingual and we don't specify a game language in the detection.

 And is this something that should be done at a lower level (e.g. in the
 AGS `File` class) so that it is also done when opening other types of
 files the game may be using? However there we would have another issue as
 we have filenames that come from ScummVM (e.g. savegame files, translation
 files when set in the ScummVM GUI) and those would already be in UTF-8.

 I am hoping this is an issue that are restricted to only a few games and
 that most games don't have the strange idea to use non-ASCII characters
 for file names. So maybe we don't need to overthink it.

 A simple fix here would be to do CP-1252 to UTF-8 conversion in
 `Game_ChangeTranslation` (or maybe in `SC_ Game_ChangeTranslation `)? It
 will not break games that only use ASCII characters, and for games that
 have non-ASCII characters in another encoding (if there is any) they would
 not work anyway currently. If in the future we find such a game, the code
 could be updated. And if we see the same issue for other types of files,
 we could add a similar conversion in the corresponding script function.

 Finally, as an aside, the reason it works when selecting the translation
 in the ScummVM GUI is because there the encoding for the file name is
 properly handled (it is consistent). And that sets the translation when
 starting the game. Then when it gets to the language selection screen it
 fails to change the translation when selection French and thus remains on
 the one previously set.
-- 
Ticket URL: <https://bugs.scummvm.org/ticket/13276#comment:2>
ScummVM :: Bugs <https://bugs.scummvm.org>
ScummVM