586944 - xulrunner-stub / fennec built with --enable-trace-malloc fails assert and does not run

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[Pirkka Kärenlampi]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100723 Ubuntu/9.04 (jaunty) Firefox/3.6.8
Build Identifier: XULRunner 2.0b4pre, Fennec 2.0a1pre, Ubuntu 9.04

Fennec cannot be used when built with --enable-trace-malloc on my system. I have observed this behavior also with xulrunner-stub running a simple "hello world"-type XUL application. xulrunner works and can be used to run e.g. the fennec application.ini.

While debugging this problem, I noticed that instead of the backtrace function from tools/trace-malloc/lib/nsTraceMalloc.c glibc:s backtrace is called by code from that file, leading to corruption in the thread-specific data structure.

Reproducible: Always

Steps to Reproduce:
1. build fennec and xulrunner with --enable-trace-malloc
2. start fennec with --trace-malloc=tm.log command line option and observe
Actual Results:  
The message:
"Assertion failure: t->suppress_tracing == 0, at /home/pkarenla/Code/Mozilla/mozilla-central/tools/trace-malloc/lib/nsTraceMalloc.c:2008"
followed by a stack trace is printed.

Expected Results:  
fennec will run and store trace-malloc logs into tm.log

I will attach a text file of the stack trace to this bug.

Comment 1

[Pirkka Kärenlampi]

Attachment: Error message and stack trace

Comment 2

[Pirkka Kärenlampi]

I have been able to fix this problem by declaring backtrace static in nsTraceMalloc.c. Maybe someone can comment or help me check if this will break something else? It seems safe enough, as that backtrace function is not exported via any header files and as far as I can see is not used directly from any other source file.

Comment 3

[Pirkka Kärenlampi]

Attachment: Patch declaring backtrace static

With this simple change xulrunner-stub and fennec work with trace-malloc.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 465620 [details] [diff] [review]
Patch declaring backtrace static

r=dbaron

Comment 5

[timeless]

http://hg.mozilla.org/mozilla-central/rev/91427ac7438e