518634 - Problems with linux stacks since this mornings downtime

Status: RESOLVED INVALID

(Release Engineering :: General, defect, P3)

Description

[Jeff Muizelaar [:jrmuizel]]

The following crash has a stack but doesn't have symbols:

http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox-Unittest&errorparser=unittest&logfile=1253811759.1253813517.1323.gz&buildtime=1253811759&buildname=Linux%20mozilla-central%20test%20opt%20mochitests&fulltext=1

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We also don't get a stack for crashes when running the the alive tests on the debug build (ie on the same box we built on), eg 
http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox-Unittest&errorparser=unittest&logfile=1253811759.1253813517.1323.gz&buildtime=1253811759&buildname=Linux%20mozilla-central%20test%20opt%20mochitests&fulltext=1

I'm guessing it's that we don't have gstsabs set, or fallout from the addition of --enable-libxul.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

May also need to set 
 export CFLAGS="-gdwarf-2"
 export CXXFLAGS="-gdwarf-2"
in the debug mac mozconfig for parity with non-debug.

Meant to say last time that bug 372581 added test package uploads, which in turn enables 'make buildsymbols' and upload of same. On linux the zip is empty, on Mac we're getting something.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

So I morphed this bug badly, apologies for that. To restate:

1, In comment #0 Jeff reported a mochitest crash with unresolved symbols in the stack, on the new "Linux mozilla-central test opt mochitests" builder. The call to mochitest looks fine, symbols are downloaded & unpacked OK, so the stack should have resolved symbols. I'm trying to reproduce this on the same machine with more verbose output from the stack dumper.

2, Linux debug builds are not providing stacks now that we started doing packaging up tests to run elsewhere (comment #1). I'm farmed this out to bug X. Adding -gstabs+ to the linux mozconfig means buildsymbols actually produces something useful (except for NSS/NSPR!), but still get no stack on the crash. We also need to set MINIDUMP_STACKWALK to get a stack.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

1, can't reproduce that crash in half a dozen attempts on moz2-linux-slave09, where this was seen first, and there are no other examples in subsequent builds. Lets defers until it's can be triggered demand. Slave returned to operation.

2, I think we used to get stacks from debug/leak crashes without setting MINIDUMP_STACKWALK, but my brain is kinda mushy so take that with a kilo of NaCl. The stack in attachment 402718 [details] was a rebuild on moz2-linux-slave06 with -gstabs+ and exporting MINIDUMP_STATCKWALK. That machine has now been returned to operation.

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to comment #3)
> 2, Linux debug builds are not providing stacks now that we started doing
> packaging up tests to run elsewhere (comment #1). I'm farmed this out to bug X.
> Adding -gstabs+ to the linux mozconfig means buildsymbols actually produces
> something useful (except for NSS/NSPR!), but still get no stack on the crash.
> We also need to set MINIDUMP_STACKWALK to get a stack.

I think this is the fault of "make buildsymbols". We are in fact stripping the binaries in the objdir when we run that on Linux:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#648

to implement the "symbol server" sort of thing for Linux. We could just disable that for now, I don't know that anyone is actually relying on it. It's just a matter of removing the -c here:
http://mxr.mozilla.org/mozilla-central/source/Makefile.in#166

Comment 6

[(not currently active) Ted Mielczarek]

Also, not helping that we have a couple of issues mashed into this one bug :)

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #3)
> 2, Linux debug builds are not providing stacks now that we started doing
> packaging up tests to run elsewhere (comment #1). I'm farmed this out to bug X.

For what value of X?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> Also, not helping that we have a couple of issues mashed into this one bug :)

yes, my FAIL

(In reply to comment #7)
> For what value of X?

Didn't actually farm it out, FAIL again.

Comment 9

[Chris AtLee [:catlee]]

Closing the bug out since it's got too much going.

The problem with no symbols in stacks on optimized builds is bug 519194.

The problem with no symbols in stacks on debug builds is bug 519196.

The problem with stacks of alive tests on debug builds is bug 519195.

Please let me know if I've missed anything that's not captured by any of the 3 bugs.