Closed Bug 1506781 Opened 3 years ago Closed 3 years ago

Large spike in crash reports with missing data - "EMPTY: no crashing thread identified; unknown error"


(Socorro :: Infra, task, P1)



(Not tracked)



(Reporter: philipp, Assigned: willkg)




(1 file)

in the past couple of hours, the [@ EMPTY: no crashing thread identified; unknown error] signature is sky-rocketing:

this started happening yesterday at exactly 19:00 UTC. the crash comments and sparse information in the reports indicate the signature is happening cross-platform and in many different circumstances, so i'd guess there is a problem processing the crashes by socorro (related to bug 1506669 or bug 1506635?).
Flags: needinfo?(willkg)
I can consistently reproduce this problem on Linux (stable 63 and Nightly 65):

1. Open browser content toolbox.
2. Run: Object.keys(tabs[0].content.Proxy.prototype);  // this crash is related to bug 1506736
3. Submit crash report.

Example of result:
All of my Mac crashes are coming up with the Empty signature. I noticed it yesterday when trying to reproduce a web replay crash.
Grabbing it to look into. Those crashes have this in the processor notes:

/stackwalk/stackwalker output failed in json: Expected object or value

I'll look at why mdsw is failing.

After figuring it out and fixing it, I'll reprocess affected crashes.
Assignee: nobody → willkg
Flags: needinfo?(willkg)
Priority: -- → P1
Duplicate of this bug: 1506830
I figured out the problem and I'm testing a fix now. I sent an email to the stability list to let everyone know where things are at.

If all goes well, I can have the fix in production in a couple of hours. I'll reprocess affected crashes after that.
Commits pushed to master at
fix bug 1506781: fix silent ujson errors

minidump-stackwalker was failing with exit code 134 because there was no
raw crash data to work on. There was no raw crash data to work on because
ujson and json handle parsing errors differently. Particularly, ujson
would shrug when trying to JSON serialize a configman DotDict. That's
really unhelpful.

This fixes the underlying issue (need to convert configman DotDict
to a dict before serializing to JSON), purges ujson from the codebase,
and fixes some tests that were testing what happens if the constants
of the universe were different which will never be the case.
Merge pull request #4709 from willkg/1506781-empty

fix bug 1506781: fix silent ujson errors
Closed: 3 years ago
Resolution: --- → FIXED
I'm reopening this and will close it after I've pushed the fix to prod and reprocessed affected crashes.
Resolution: FIXED → ---
I pushed the fix to prod and reprocessed 266k crashes. We should be all set now. Marking as FIXED.
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.