[Scummvm-devel] Fwd: PS2: stack overlfow

[Andre Heider]

On Sun, Mar 8, 2009 at 2:42 AM, Max Horn <max at quendi.de> wrote:

>> I spent then some more time to modify the linker scripts
>> (keep in mind that there is no OS on PS2 so you have to
>> do this by hand) to increase the stack size, up to 8X
>> and it still does not help...
>>
>> So I would say it's an underrun.
>>
>> Unluckily I ran out of ideas and of time, so I'll have
>> to leave it like this. Even if I got gdb to work, I am
>> not sure it would help with this kind of issue, and we
>> don't have valgrind for PS2.

I'm sort of in the same boat on the Wii side. We run on bare metal,
there's a limited amount of memory, and the debugging facilities are
limited. I don't know how the PS2 works in these aspects, but maybe I
can give some hints:

One example: COMI on the Gamecube (same port as Wii), which has 24mb
of ram, ran into the memory limit issue on 0.13.0. All engined enabled
per default from the configure script were built in. It crashes at
different points, when I'm lucky I get a exception and a misleading
backtrace, but thats not always the case. Sometimes it hangs,
sometimes the game engine does "weird stuff".

How it's on the GC/Wii:
There's the usual memory area, and the stack is located at the "end"
of the memory, that's defined in the linkerscript.
Loaders place the ScummVM binary at the start of that area, and the
end of the bss section defines the start area for the malloc()
implementation. Hence, the memory available to the application is
defined by its code and data sections.
If an application requests memory (malloc, new, ...) a syscall called
sbrk() (which is just a function pointer/callback here, see "man 2
sbrk") is issued to enlarge the area used by it. This implementation
keeps a pointer to the upper limit at all times.
As ScummVM requests more and more memory, this pointer and the end of
the stack (the stack grows downwards) get closer to each other. Not
every malloc/new by ScummVM checks the return value, and there're
situations where code writes into the stack's memory due to an
uninitialized pointer. Since we do not have virtual memory with any
protections, this is valid and doesn't yield an error (I can just do
"*(stackptr + x) = y"). That of course can't be good, but shit happens
:P

To analyse this problem there are a few tricks.
- You can just try to run the problematic code with less engines
compiled in the binary, giving you more ram to allocate

- Check the used pointer from sbrk(). I did this, see
http://scummvm.svn.sourceforge.net/viewvc/scummvm/scummvm/trunk/backends/platform/wii/main.cpp?revision=35985&view=markup
wii_memstats() checks "SYS_GetArenaHi() - SYS_GetArenaLo()" where the
1st is the end of the stack and the latter is the sbrk() pointer. It's
a poor man's way to check for free memory ;)

- a replacement new/delete, which do additional checks

- fix gdb :P

I'm sure I missed a few things I tried, but I gotta run now...

Andre