Closed Bug 1272738 Opened 7 years ago Closed 6 years ago

64-bit windows builds missing symbols for xul.dll in crash-stats for Nightly builds since 5-11

Categories

(Toolkit :: Crash Reporting, defect)

defect
Not set
normal

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox49 + fixed

People

(Reporter: mccr8, Assigned: ted)

References

Details

Something seems to have recently broken in the symbolification of Nightly builds. In the list of Nightly 5-11 top crashes I see [@ mozalloc_abort | xul.dll@0x7cde08 | strspn] and [@ mozalloc_abort | xul.dll@0x7cde08 | ucrtbase.dll@0xe41bf], where the xul.dll should be something else, I assume. There are 23 signatures with xul.dll in the top list for 5-12, and zero for 5-10, so maybe it is something that is getting worse?

dbaron said that things work if he loads the minidump in MSVC.

I'm getting the top crash list by platform from the links on this page:
http://dbaron.org/mozilla/crashes-by-build
Summary: Missing symbols in crash-stats for 5-11 and 5-12 Nightly builds → Missing symbols for xul.dll in crash-stats for 5-11 and 5-12 Nightly builds
11:41:15 <ted> oh
11:41:20 <ted> dbaron: apparently my original theory must be correct
11:41:23 <ted>  "corrupt_symbols": true, "frame": 0, "module": "xul.dll",
11:41:26 <ted> if you look at the raw dump
11:41:37 <ted> breakpad is saying it failed to parse the symbol file
His "original theory" is bug 1268029.
Sorry, bug 1272278.
Component: General → Breakpad Integration
Product: Socorro → Toolkit
This is clearly 64-bit only.  Compare the 64-bit crashes:
https://crash-stats.mozilla.com/search/?product=Firefox&build_id=20160512030253&platform=Windows&date=%3E%3D2016-05-12&release_channel=nightly&cpu_arch=amd64
with the 32-bit crashes:
https://crash-stats.mozilla.com/search/?product=Firefox&build_id=20160512030253&platform=Windows&date=%3E%3D2016-05-12&release_channel=nightly&cpu_arch=x86
Blocks: 1272278
Summary: Missing symbols for xul.dll in crash-stats for 5-11 and 5-12 Nightly builds → 64-bit windows builds missing symbols for xul.dll in crash-stats for 5-11 and 5-12 Nightly builds
No longer blocks: 1272278
Blocks: 1268617
Depends on: 1272278
This is a big problem. In Nightly 20160513030539 nearly half of the top 20 Windows crash signatures have xul.dll in them and appear to be affected:

> 1 	xul.dll@0xa616f2 | PR_NativeRunThread | pr_root   Add term 	68 	16.35 % 	
> 2 	xul.dll@0x171af24 | ucrtbase.dll@0xe41bf   Add term 	31 	7.45 % 	
> 4 	xul.dll@0x171af24 | strspn   Add term 	18 	4.33 % 	
> 5 	xul.dll@0xc8adc7 | strspn   Add term 	18 	4.33 % 	
> 7 	mozalloc_abort | xul.dll@0x77dc10 | xul.dll@0x2671abf   Add term 	11 	2.64 % 	
> 11 	xul.dll@0x1869ef7   Add term 	7 	1.68 % 	
> 13 	xul.dll@0x100e008 | msvcp140.dll@0x3b3f2   Add term 	4 	0.96 % 	
> 18 	xul.dll@0x1fbeafe | arena_malloc_small   Add term 	3 	0.72 % 	
> 19 	xul.dll@0x98aa8 | PR_Lock   Add term 	3 	0.72 % 	

The results are similar in the subsequent Nightly builds. Furthermore, the numbers of crashes in these builds is significantly higher than normal, so it's a doubly bad that we are getting incomplete info for them.

ted, can you please investigate ASAP?
Flags: needinfo?(ted)
Tracking for 49 because this greatly reduces the information we can get from our crash reports :(
Summary: 64-bit windows builds missing symbols for xul.dll in crash-stats for 5-11 and 5-12 Nightly builds → 64-bit windows builds missing symbols for xul.dll in crash-stats for Nightly builds since 5-11
I started debugging this (in bug 1272278), I have a pair of band-aid patches that should help. I will try to get them landed today.
Flags: needinfo?(ted)
Tracking for 49 as crash symbols are definitely important.
Should be fixed when bug 1272278 hits Nightly. I may also see if I can fix the few days' worth of broken symbol files in the symbol store.
Assignee: nobody → ted
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Should be fixed when bug 1272278 hits Nightly. I may also see if I can fix
> the few days' worth of broken symbol files in the symbol store.

Ted: Should I file a bug now to get the crashes reprocessed? Bug 1273658 has most of the signatures -or does that have to wait until the production push happens as noted in Bug 1272278?
Flags: needinfo?(ted)
Reprocessing won't fix anything--this was due to us uploading broken symbol files. I'd have to manually fix them and I just haven't found time.
Flags: needinfo?(ted)
We fixed the problem in symbol dumping. I never fixed the broken symbol files, but the problem has mostly gone away as new non-broken builds have gone out, so we'll just call this WORKSFORME.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.