1272738 - 64-bit windows builds missing symbols for xul.dll in crash-stats for Nightly builds since 5-11

Status: RESOLVED WORKSFORME

(Toolkit :: Crash Reporting, defect)

Description

[Andrew McCreight [:mccr8]]

Something seems to have recently broken in the symbolification of Nightly builds. In the list of Nightly 5-11 top crashes I see [@ mozalloc_abort | xul.dll@0x7cde08 | strspn] and [@ mozalloc_abort | xul.dll@0x7cde08 | ucrtbase.dll@0xe41bf], where the xul.dll should be something else, I assume. There are 23 signatures with xul.dll in the top list for 5-12, and zero for 5-10, so maybe it is something that is getting worse?

dbaron said that things work if he loads the minidump in MSVC.

I'm getting the top crash list by platform from the links on this page:
http://dbaron.org/mozilla/crashes-by-build

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

11:41:15 <ted> oh
11:41:20 <ted> dbaron: apparently my original theory must be correct
11:41:23 <ted>  "corrupt_symbols": true, "frame": 0, "module": "xul.dll",
11:41:26 <ted> if you look at the raw dump
11:41:37 <ted> breakpad is saying it failed to parse the symbol file

Comment 2

[Andrew McCreight [:mccr8]]

His "original theory" is bug 1268029.

Comment 3

[Andrew McCreight [:mccr8]]

Sorry, bug 1272278.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This is clearly 64-bit only.  Compare the 64-bit crashes:
https://crash-stats.mozilla.com/search/?product=Firefox&build_id=20160512030253&platform=Windows&date=%3E%3D2016-05-12&release_channel=nightly&cpu_arch=amd64
with the 32-bit crashes:
https://crash-stats.mozilla.com/search/?product=Firefox&build_id=20160512030253&platform=Windows&date=%3E%3D2016-05-12&release_channel=nightly&cpu_arch=x86

Comment 5

[Nicholas Nethercote [inactive]]

This is a big problem. In Nightly 20160513030539 nearly half of the top 20 Windows crash signatures have xul.dll in them and appear to be affected:

> 1 	xul.dll@0xa616f2 | PR_NativeRunThread | pr_root   Add term 	68 	16.35 % 	
> 2 	xul.dll@0x171af24 | ucrtbase.dll@0xe41bf   Add term 	31 	7.45 % 	
> 4 	xul.dll@0x171af24 | strspn   Add term 	18 	4.33 % 	
> 5 	xul.dll@0xc8adc7 | strspn   Add term 	18 	4.33 % 	
> 7 	mozalloc_abort | xul.dll@0x77dc10 | xul.dll@0x2671abf   Add term 	11 	2.64 % 	
> 11 	xul.dll@0x1869ef7   Add term 	7 	1.68 % 	
> 13 	xul.dll@0x100e008 | msvcp140.dll@0x3b3f2   Add term 	4 	0.96 % 	
> 18 	xul.dll@0x1fbeafe | arena_malloc_small   Add term 	3 	0.72 % 	
> 19 	xul.dll@0x98aa8 | PR_Lock   Add term 	3 	0.72 % 	

The results are similar in the subsequent Nightly builds. Furthermore, the numbers of crashes in these builds is significantly higher than normal, so it's a doubly bad that we are getting incomplete info for them.

ted, can you please investigate ASAP?

Comment 6

[Nicholas Nethercote [inactive]]

Tracking for 49 because this greatly reduces the information we can get from our crash reports :(

Comment 7

[(not currently active) Ted Mielczarek]

I started debugging this (in bug 1272278), I have a pair of band-aid patches that should help. I will try to get them landed today.

Comment 8

[Marcia Knous [:marcia]]

Tracking for 49 as crash symbols are definitely important.

Comment 9

[(not currently active) Ted Mielczarek]

Should be fixed when bug 1272278 hits Nightly. I may also see if I can fix the few days' worth of broken symbol files in the symbol store.

Comment 10

[Marcia Knous [:marcia]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Should be fixed when bug 1272278 hits Nightly. I may also see if I can fix
> the few days' worth of broken symbol files in the symbol store.

Ted: Should I file a bug now to get the crashes reprocessed? Bug 1273658 has most of the signatures -or does that have to wait until the production push happens as noted in Bug 1272278?

Comment 11

[(not currently active) Ted Mielczarek]

Reprocessing won't fix anything--this was due to us uploading broken symbol files. I'd have to manually fix them and I just haven't found time.

Comment 12

[(not currently active) Ted Mielczarek]

We fixed the problem in symbol dumping. I never fixed the broken symbol files, but the problem has mostly gone away as new non-broken builds have gone out, so we'll just call this WORKSFORME.