553357 - gdb "aswhere" command crashes if the ActionScript callstack is empty

Status: RESOLVED FIXED

(Tamarin Graveyard :: Tools, defect, P3)

Description

[Mike Morearty]

A minor bug with the "aswhere" command in utils/as3.gdbinit: If you call that function when the ActionScript callstack is completely empty, it throws a null pointer exception:

(gdb) aswhere

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000014
0x00052cc7 in avmplus::CallStackNode::next (this=0x0) at StackTrace.h:159
159			inline CallStackNode* next() const { return m_next; }
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on"
Evaluation of the expression containing the function (avmshell::DebugCLI::bt()) will be abandoned.

The reason for this is that the avmplus::Debugger::frameCount() function in avmplusDebugger.cpp was originally intended for use only by source-level ActionScript debuggers such as fdb and Flash Builder, and the source-level debugger functionality in the Flash Player does not allow you to halt at a point where the ActionScript callstack is completely empty.  Since it's easy to do that from gdb, you end up in a situation where core->callStack is NULL, so frameCount(), which assumes that core->callStack always has at least one entry, fails.

I think it's fine to modify avmplus::Debugger::frameCount() to first do a null check, and return zero in that case; returning zero can't be any worse than throwing an exception.  Another way to fix it would be to change DebugCLI::bt() to first check core->callStack before it does anything else; that has the slight advantage of avoiding messing with the VM's behavior, but really I don't see any reason to avoid messing with the VM.

Comment 1

[Mike Morearty]

This can go to either Krzysztof or me; it's an easy fix.  Not needed for Flash 10.1.

Comment 2

[Krzysztof Palacz]

I could fix it in a new patch for 545538

Comment 3

[Mike Morearty]

Fixed by Krzysztof in http://hg.mozilla.org/tamarin-redux/rev/c80b3e1daf28