Closed Bug 1506781 Opened 3 years ago Closed 3 years ago

Large spike in crash reports with missing data - "EMPTY: no crashing thread identified; unknown error"

Categories

(Socorro :: Infra, task, P1)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: philipp, Assigned: willkg)

References

Details

Attachments

(1 file)

in the past couple of hours, the [@ EMPTY: no crashing thread identified; unknown error] signature is sky-rocketing: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20unknown%20error#graphs

this started happening yesterday at exactly 19:00 UTC. the crash comments and sparse information in the reports indicate the signature is happening cross-platform and in many different circumstances, so i'd guess there is a problem processing the crashes by socorro (related to bug 1506669 or bug 1506635?).
Flags: needinfo?(willkg)
I can consistently reproduce this problem on Linux (stable 63 and Nightly 65):

1. Open browser content toolbox.
2. Run: Object.keys(tabs[0].content.Proxy.prototype);  // this crash is related to bug 1506736
3. Submit crash report.

Example of result: https://crash-stats.mozilla.com/report/index/1320578c-776c-4384-8cd9-3564a0181113
All of my Mac crashes are coming up with the Empty signature. I noticed it yesterday when trying to reproduce a web replay crash.
Grabbing it to look into. Those crashes have this in the processor notes:

/stackwalk/stackwalker output failed in json: Expected object or value

I'll look at why mdsw is failing.

After figuring it out and fixing it, I'll reprocess affected crashes.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Flags: needinfo?(willkg)
Priority: -- → P1
Duplicate of this bug: 1506830
I figured out the problem and I'm testing a fix now. I sent an email to the stability list to let everyone know where things are at.

If all goes well, I can have the fix in production in a couple of hours. I'll reprocess affected crashes after that.
Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/c8e9c8d4a3e52ab07c8a04c9ed6259a0abb01fe4
fix bug 1506781: fix silent ujson errors

minidump-stackwalker was failing with exit code 134 because there was no
raw crash data to work on. There was no raw crash data to work on because
ujson and json handle parsing errors differently. Particularly, ujson
would shrug when trying to JSON serialize a configman DotDict. That's
really unhelpful.

This fixes the underlying issue (need to convert configman DotDict
to a dict before serializing to JSON), purges ujson from the codebase,
and fixes some tests that were testing what happens if the constants
of the universe were different which will never be the case.

https://github.com/mozilla-services/socorro/commit/398da01caffd3b5d63caffc471029715ba6eafce
Merge pull request #4709 from willkg/1506781-empty

fix bug 1506781: fix silent ujson errors
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
I'm reopening this and will close it after I've pushed the fix to prod and reprocessed affected crashes.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I pushed the fix to prod and reprocessed 266k crashes. We should be all set now. Marking as FIXED.
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.