417872 - Sometimes crashes when running trace-malloc leak test

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect)

Description

[Boris Zbarsky [:bzbarsky]]

This is a followup to bug 379484.  Until we twiddled the timeouts in that case, we were apparently somehow getting into a state where we would crash in the trace-malloc code.  All the line numbers etc that follow are from 2007-05-02.

The stack for that crash is <https://bugzilla.mozilla.org/attachment.cgi?id=263530>.

rhelmer had the thing in gdb, and got some locals:

frame 7
#7  0x00ae2d3b in backtrace (skip=-1)
    at /builds/tinderbox/Fx-Trunk-Debug/Linux_2.6.9-42.ELsmp_Depend/mozilla/tools/trace-malloc/lib/nsTraceMalloc.c:1399
<rhelmer> p bp is $1 = (void **) 0xbff04668
<rhelmer> p *bp is $2 = (void *) 0x0

frame 6
#6  0x00ae22a6 in calltree (bp=0x9078ea0)
    at /builds/tinderbox/Fx-Trunk-Debug/Linux_2.6.9-42.ELsmp_Depend/mozilla/tools/trace-malloc/lib/nsTraceMalloc.c:1049
>p depth
<rhelmer> $3 = 1
> p bp
<rhelmer> $4 = (void **) 0x8b8eec0
> p bpdown
<rhelmer> $5 = (void **) 0x7374656e
> p bpdown[0]
<rhelmer> Cannot access memory at address 0x7374656e

So as far as I can tell, we call |calltree| with a pointer to null.  Then the
execution works as follows, I think:

First time through the loop:

1047 bpdown = (void**)(0)
1048 (*0xbff04668) = NULL
1049 Test false because RHS is 0.
1051 bpup = 0xbff04668
1052 bp = 0

Second time through the loop:

1047 bpdown = (void**)(*(void**)0)  (whatever that is!)
1048 (*0) = 0xbff04668              (not sure why this works)
1049 Test crashes because bpdown is some random pointer

So it seems like the "simple" thing to do would be to either have calltree bail
out if its arg points to null or have backtrace() not call calltree with such
an arg.

Now the question is why we have such an arg in the first place, if this analysis is correct....

Comment 1

[Serge Gautherie (:sgautherie)]

Still happening?