Closed
Bug 1506781
Opened 6 years ago
Closed 6 years ago
Large spike in crash reports with missing data - "EMPTY: no crashing thread identified; unknown error"
Categories
(Socorro :: Infra, task, P1)
Socorro
Infra
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: philipp, Assigned: willkg)
References
Details
Attachments
(1 file)
in the past couple of hours, the [@ EMPTY: no crashing thread identified; unknown error] signature is sky-rocketing: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20unknown%20error#graphs
this started happening yesterday at exactly 19:00 UTC. the crash comments and sparse information in the reports indicate the signature is happening cross-platform and in many different circumstances, so i'd guess there is a problem processing the crashes by socorro (related to bug 1506669 or bug 1506635?).
Flags: needinfo?(willkg)
Comment 1•6 years ago
|
||
I can consistently reproduce this problem on Linux (stable 63 and Nightly 65):
1. Open browser content toolbox.
2. Run: Object.keys(tabs[0].content.Proxy.prototype); // this crash is related to bug 1506736
3. Submit crash report.
Example of result: https://crash-stats.mozilla.com/report/index/1320578c-776c-4384-8cd9-3564a0181113
Comment 2•6 years ago
|
||
All of my Mac crashes are coming up with the Empty signature. I noticed it yesterday when trying to reproduce a web replay crash.
Assignee | ||
Comment 3•6 years ago
|
||
Grabbing it to look into. Those crashes have this in the processor notes:
/stackwalk/stackwalker output failed in json: Expected object or value
I'll look at why mdsw is failing.
After figuring it out and fixing it, I'll reprocess affected crashes.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Flags: needinfo?(willkg)
Priority: -- → P1
Assignee | ||
Comment 5•6 years ago
|
||
I figured out the problem and I'm testing a fix now. I sent an email to the stability list to let everyone know where things are at.
If all goes well, I can have the fix in production in a couple of hours. I'll reprocess affected crashes after that.
Comment 6•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/c8e9c8d4a3e52ab07c8a04c9ed6259a0abb01fe4
fix bug 1506781: fix silent ujson errors
minidump-stackwalker was failing with exit code 134 because there was no
raw crash data to work on. There was no raw crash data to work on because
ujson and json handle parsing errors differently. Particularly, ujson
would shrug when trying to JSON serialize a configman DotDict. That's
really unhelpful.
This fixes the underlying issue (need to convert configman DotDict
to a dict before serializing to JSON), purges ujson from the codebase,
and fixes some tests that were testing what happens if the constants
of the universe were different which will never be the case.
https://github.com/mozilla-services/socorro/commit/398da01caffd3b5d63caffc471029715ba6eafce
Merge pull request #4709 from willkg/1506781-empty
fix bug 1506781: fix silent ujson errors
Updated•6 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 7•6 years ago
|
||
Assignee | ||
Comment 8•6 years ago
|
||
I'm reopening this and will close it after I've pushed the fix to prod and reprocessed affected crashes.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 9•6 years ago
|
||
I pushed the fix to prod and reprocessed 266k crashes. We should be all set now. Marking as FIXED.
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•