1405151 - Crash in EMPTY: no crashing thread identified; OK

Status: VERIFIED FIXED

(Core :: General, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-ac31cf11-b066-4b86-9a03-4411a0171001.
=============================================================

[Tracking Requested - why for this release]:
this crash signature is spiking for mac osx users in 57.0b. it's accounting for 44% of all content crashes for os x users:

https://crash-stats.mozilla.com/signature/?process_type=%3Dcontent&platform=Mac%20OS%20X&release_channel=beta&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20OK&date=%3E%3D2017-08-01#graphs

without a crashing stack there is little to go on - correlations only bring up a long list of apple system libraries there is little to go on. a couple of user comments say that the crash happened while they where on google maps though:
https://crash-stats.mozilla.com/signature/?process_type=%3Dcontent&platform=Mac%20OS%20X&version=57.0b&release_channel=beta&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20OK&date=%3E%3D2017-09-01#comments

Comment 1

[Jim Mathies [:jimm]]

Hey Stephen, would you please investigate here, maybe by looking at landings around the time this spiked on the 27th.

https://crash-stats.mozilla.com/signature/?process_type=%3Dcontent&platform=Mac%20OS%20X&release_channel=beta&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20OK&date=%3E%3D2017-08-01#graphs

Comment 2

[Stephen A Pohl [:spohl]]

Ted, this has come up before in bug 1385531 comment 9 and bug 1385531 comment 15. Is there any help you can offer here?

Keeping n-i set to investigate on my side.

Comment 3

[(not currently active) Ted Mielczarek]

There's a Socorro bug here that needs to be filed, in that it's not displaying any thread stacks at all in this case. If you look at the "raw dump" tab you can see that there are stacks from various threads, we just don't have a crashing thread identified.

Aside from that, the crash reason here is "EXC_SOFTWARE / SIGABRT", which is what Breakpad uses for writing dumps from its SIGABRT handler:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc#617

I need to look at the dump to see if there's some specific reason that we don't have a crashing thread identified.

Comment 4

[(not currently active) Ted Mielczarek]

OK, I think this is a bug in Breakpad's SIGABRT handler in the out-of-process case on Mac. We made that code work about 3 years ago (bug 1012912), but I think it's never been quite right.

The signal handler (in the content process) calls `ExceptionHandler::WriteMinidumpWithException` and passes `mach_thread_self()` as `thread_name`:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc#621

In the OOP case, that method calls `CrashGenerationClient::RequestDumpForException`, passing that `thread_name` parameter down as `crashing_thread`:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc#379

That method sends a Mach IPC message to the server, and passes `mach_thread_self()` as the handler thread id:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/crash_generation/crash_generation_client.cc#49

`CrashGenerationServer::WaitForOneMessage` (in the chrome process) receives that message and pulls out the mach ports:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/crash_generation/crash_generation_server.cc#110

At this point `crashing_thread == handler_thread` because the content process used `mach_thread_self()` for both of them. It instantiates a `MinidumpGenerator`, passing `handler_thread` into the constructor:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/crash_generation/crash_generation_server.cc#122

and then calls `MinidumpGenerator::SetExceptionInformation`, passing in the exception information and `crashing_thread`.

Those two threads are stored in the `handler_thread_` and `exception_thread_` fields in `MinidumpGenerator`, respectively:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/minidump_generator.cc#97
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/minidump_generator.h#105


Still with me? OK. Here's where things break. in `MinidumpGenerator::WriteThreadListStream`, it enumerates the threads of the process and writes an entry in the thread list stream for each, but it *skips* the handler thread:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/minidump_generator.cc#998

Since the handler thread is also the crashing thread per above, we fail to write info about the thread we're most interested in there! I'm not really sure why that code skips the handler thread, given that it also has code to special-case getting the thread context from the handler thread when a context is provided from an exception, but I think this is just an unexpected interaction with the OOP code.

The simplest fix here might be to make `CrashGenerationClient::RequestDumpForException` send `MACH_PORT_NULL` as the handler thread, although I don't know if that would break things. Alternately, we could remove the code in `MinidumpGenerator::WriteThreadListStream` that skips the handler thread, assuming that doesn't break anything. We have plenty of tests that test in-process crashes, so I think if we get green tests we should be OK.

I'm guessing we either don't have a test for this specific case, or our test isn't thorough enough in what it's testing--we do get a valid minidump in this case, it's just missing a critical piece of information.

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> There's a Socorro bug here that needs to be filed, in that it's not
> displaying any thread stacks at all in this case.

Filed bug 1406121 on this bit.

Comment 6

[(not currently active) Ted Mielczarek]

Oh, related, I was not quite right in my assessment in comment 4 about when we made this code work--we did enable it ~3 years ago, but then regressed it shortly after, and it remained broken until mconley re-fixed it in bug 1350917 for Firefox 55.

So in Firefox 47-54 we wouldn't have seen this issue at all because we would not have caught these crashes(!).

Comment 7

[Sylvestre Ledru [:Sylvestre]]

OK, if I understand correctly:
* we always had these crashes
* we didn't detect them in the past
* we are now catching them thanks to bug 1350917
* this was also impacting 55 and previous release
* this is an improvement over what we had in the past
* there is no need to panic 
* this isn't a blocker to 57

Am I correct?

Comment 8

[(not currently active) Ted Mielczarek]

Not *exactly*.

(In reply to Sylvestre Ledru [:sylvestre] from comment #7)
> OK, if I understand correctly:
> * we always had these crashes
> * we didn't detect them in the past
> * we are now catching them thanks to bug 1350917
> * this was also impacting 55 and previous release

We did not detect these kinds of crashes on Mac prior to Firefox 55, this is correct.

> * this is an improvement over what we had in the past
> * there is no need to panic 
> * this isn't a blocker to 57

However, if this crash signature has spiked since Firefox 55, then there are in fact one or more new crashes that we need to fix. Without fixing *this* bug we don't have actionable data to fix these crashes, or even know whether it's one issue or multiple issues, or whether it's the same issue as crashes on other platforms...

Comment 9

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Tracked as a 57 blocker. Selena, is this something you can help with? This crash sign has been mentioned at the channel meeting twice already as a top crasher on OSX.

Comment 10

[Stephen A Pohl [:spohl]]

This is my top priority.

Comment 11

[Stephen A Pohl [:spohl]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2293ef4dc5a49af197a4ea07f9cb72582f04241

Comment 12

[Stephen A Pohl [:spohl]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c46f4cc59cfcf47b4601f26518495d04b09e472c

Comment 13

[Stephen A Pohl [:spohl]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> The simplest fix here might be to make
> `CrashGenerationClient::RequestDumpForException` send `MACH_PORT_NULL` as
> the handler thread, although I don't know if that would break things.

This failed pretty badly on try (comment 11).

> Alternately, we could remove the code in
> `MinidumpGenerator::WriteThreadListStream` that skips the handler thread,
> assuming that doesn't break anything. We have plenty of tests that test
> in-process crashes, so I think if we get green tests we should be OK.

This seemed to pass try (comment 12), but may have made things worse. I have been triggering crashes by sending `kill -s 6` to the plugin-container process. Before the patch, the following message was printed to the console:

2017-10-11 10:12:05: minidump_processor.cc:295: ERROR: Minidump indicated requesting thread 0x19613, not found in /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/A18814C7-8DD8-40C1-B1D4-4BBB7318FAB3.dmp

The associated crash report is: https://crash-stats.mozilla.com/report/index/257b7eb2-7859-46a5-a1fa-c0a0b0171011

If we stop skipping the handler thread as in the try build in comment 12, we start printing the following:

2017-10-11 10:06:59: minidump.cc:1425: ERROR: MinidumpThread has a memory region problem, 0x0+0x0, RVA 0x0x0
2017-10-11 10:06:59: minidump.cc:1472: ERROR: MinidumpThread cannot read context
2017-10-11 10:06:59: minidump_processor.cc:249: ERROR: No memory region for /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/AA4D239B-682A-4F3D-A7BB-FEF108293DE5.dmp:42/43 id 0x0
2017-10-11 10:06:59: stackwalker.cc:198: ERROR: Can't choose a stackwalker implementation without context
2017-10-11 10:06:59: minidump_processor.cc:280: ERROR: No stackwalker for /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/AA4D239B-682A-4F3D-A7BB-FEF108293DE5.dmp:42/43 id 0x0
2017-10-11 10:06:59: minidump_processor.cc:295: ERROR: Minidump indicated requesting thread 0x1107, not found in /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/AA4D239B-682A-4F3D-A7BB-FEF108293DE5.dmp

The associated crash report is: https://crash-stats.mozilla.com/report/index/0feaef05-3076-4639-aaeb-cc8740171011

I will continue to investigate this.

Comment 14

[(not currently active) Ted Mielczarek]

Oh, I think you missed a spot! You also need to remove this check:
https://hg.mozilla.org/try/file/c46f4cc59cfc/toolkit/crashreporter/breakpad-client/mac/handler/minidump_generator.cc#l1014

I think most of the error spew you're seeing there is because without that change you wound up with an empty entry at the end of the thread list stream, so the minidump reader is just failing to read that.

Comment 15

[Stephen A Pohl [:spohl]]

Ok, that fixed the extra console spew. Now we're back to the original error:

2017-10-11 15:42:11: minidump_processor.cc:295: ERROR: Minidump indicated requesting thread 0x1df2b, not found in /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/963D1F56-86D9-48B6-A212-34697E3E2CBB.dmp

And we're seemingly still unable to determine the crashing thread:

https://crash-stats.mozilla.com/report/index/d943bca4-aeac-4a3d-a2d7-635a50171011

Trying to look at the minidump next.

Comment 16

[Stephen A Pohl [:spohl]]

Attachment: minidump_dump dump

I used minidump_dump to dump the contents of one of these minidump files. The console printed the following error when I crashed the child process via `kill -s 6`:

2017-10-12 14:42:53: minidump_processor.cc:295: ERROR: Minidump indicated requesting thread 0x3d01b, not found in /Users/spohl/Library/Application Support/Firefox/Crash Reports/pending/E2443AC4-803D-4C40-9F5D-6F79B0AB1EAE.dmp

I will upload the minidump shortly. It can be accessed in Socorro here:

https://crash-stats.mozilla.com/report/index/544d8a9d-144e-41e6-b489-1c8fd0171012

Despite the console error message stating that the thread 0x3d01b could not be found in the minidump file, it does seem to appear in the dump file as thread[36]. Ted, does it appear so to you as well?

Comment 17

[Stephen A Pohl [:spohl]]

Attachment: dmp file

Comment 18

[Stephen A Pohl [:spohl]]

The console spew from running minidump_dump was:

MacBook-Pro:bug1405151 spohl$ ./minidump_dump E2443AC4-803D-4C40-9F5D-6F79B0AB1EAE.dmp > dump.txt
2017-10-12 14:45:22: minidump.cc:5008: INFO: Minidump opened minidump E2443AC4-803D-4C40-9F5D-6F79B0AB1EAE.dmp
2017-10-12 14:45:22: minidump.cc:5128: INFO: Minidump not byte-swapping minidump
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:2203: INFO: MinidumpModule could not determine version for /Users/spohl/Documents/objdir-ff-release/dist/Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
2017-10-12 14:45:22: minidump.cc:473: INFO: MinidumpContext: looks like AMD64 context
2017-10-12 14:45:22: minidump.cc:5771: INFO: GetStream: type 1197932546 not present
2017-10-12 14:45:22: minidump_dump.cc:149: INFO: minidump.GetAssertion() failed
2017-10-12 14:45:22: minidump.cc:3993: INFO: MinidumpMiscInfo size larger than expected 1368, skipping over the unknown part
2017-10-12 14:45:22: minidump.cc:5771: INFO: GetStream: type 16 not present
2017-10-12 14:45:22: minidump_dump.cc:181: ERROR: minidump.GetMemoryInfoList() failed
2017-10-12 14:45:22: minidump.cc:5771: INFO: GetStream: type 1129316353 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932550 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932551 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932549 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932548 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932547 not present
2017-10-12 14:45:22: minidump.cc:5726: INFO: SeekToStreamType: type 1197932553 not present
2017-10-12 14:45:22: minidump.cc:4980: INFO: Minidump closing minidump

Comment 19

[(not currently active) Ted Mielczarek]

Yeah, huh, running that dump through minidump_dump locally I see it listed in the logging output:
2017-10-12 15:38:47: minidump_processor.cc:201: INFO: Looking at thread bug1405151.dmp:36/44 id 0x3d01b

...but then I see the same error message you did:
2017-10-12 15:38:47: minidump_processor.cc:302: ERROR: Minidump indicated requesting thread 0x3d01b, not found in bug1405151.dmp

Comment 20

[(not currently active) Ted Mielczarek]

Oh, I found it. The minidump processor in Breakpad also has this:
https://chromium.googlesource.com/breakpad/breakpad/+/0924d424e444d57dd95c647652a11f2d655c11a0/src/processor/minidump_processor.cc#203

It skips the dump_thread because in the in-process case that's the thread calling MinidumpWriteDump, so it winds up with a weird stack. However, in this case `dump_thread == handler_thread`, so we're getting screwed up! This is why I was hoping we could simply pass `MACH_PORT_NULL` as the handler thread :-/.

Comment 21

[(not currently active) Ted Mielczarek]

Oh! Wait! I think your patch to use `MACH_PORT_NULL` changed the wrong place! Sorry, I should have been clearer there!

You changed the call to `CrashGenerationClient::RequestDumpForException`, which is sending the *crashing thread*:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc#379

What I intended was to change this line *in* `CrashGenerationClient::RequestDumpForException`:
https://dxr.mozilla.org/mozilla-central/rev/c97190c389c4cfef20fe55b4bacade95a36ae6ef/toolkit/crashreporter/breakpad-client/mac/crash_generation/crash_generation_client.cc#49

to set the *handler thread* to MACH_PORT_NULL.

Comment 22

[Stephen A Pohl [:spohl]]

Oh, sorry for misunderstanding. This seems to work locally. The following 6 crashes seem to appear with proper stacks in Socorro:

https://crash-stats.mozilla.com/report/index/f2bc5a6f-99db-4925-9abb-8b4370171012
https://crash-stats.mozilla.com/report/index/017d1133-8434-4c3f-a3cb-9de621171012
https://crash-stats.mozilla.com/report/index/50cc0063-ee53-4b90-b241-7c21e0171012
https://crash-stats.mozilla.com/report/index/c2aa2c0e-e33c-4128-b09d-1edc50171012
https://crash-stats.mozilla.com/report/index/518775cf-a5d7-40a5-bce3-60a850171012
https://crash-stats.mozilla.com/report/index/6edd1615-8f81-48dd-a264-cc5c00171012

Sending to try shortly.

Comment 23

[Stephen A Pohl [:spohl]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=239c9c73655923c1909bd88a59b35929ea97af80

Comment 24

[Stephen A Pohl [:spohl]]

Attachment: Patch

This seems to pass try (comment 23) and stacks seem to appear correctly in Socorro now. Shall we give this a shot?

Comment 25

[Gabriele Svelto [:gsvelto] (PTO)]

(In reply to Stephen A Pohl [:spohl] from comment #18)
> 2017-10-12 14:45:22: minidump.cc:3993: INFO: MinidumpMiscInfo size larger
> than expected 1368, skipping over the unknown part

This is most likely unrelated but weird. We already fixed this once and upstreamed the fix to breakpad, I'll have a look at this dump and figure out why the misc info field is still larger than expected.

Comment 26

[(not currently active) Ted Mielczarek]

Comment on attachment 8918010 [details] [diff] [review]
Patch

Review of attachment 8918010 [details] [diff] [review]:
-----------------------------------------------------------------

Excellent, thanks! Glad to see the simple fix actually works.

Comment 27

[Stephen A Pohl [:spohl]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/cafd9cdafeb71fe486fc078cc1ae16e8c2c75e01
Bug 1405151: Ensure that crashes appear correctly in Socorro in the case of SIGABRT crashes on macOS. r=ted

Comment 28

[Stephen A Pohl [:spohl]]

Comment on attachment 8918010 [details] [diff] [review]
Patch

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: A significant number of crashes would continue to be reported without usable crash stacks.
[Is this code covered by automated tests?]: Our test suite covers a lot of our crash reporting. We don't have a dedicated test for the issue in this bug yet, but we shouldn't hold up landing of this patch. I will file a separate bug to add a test case.
[Has the fix been verified in Nightly?]: In a local build off of current Nightly, yes.
[Needs manual test from QE? If yes, steps to reproduce]: Ideally, yes. Do the following about 5-10 times and make sure that Socorro displays correct crash stacks every time (as opposed to "Crash in EMPTY: no crashing thread identified; OK"):
1. Launch Firefox
2. Open Terminal
3. Type `ps ax | grep plugin`
4. Find the process id for the first plugin-container process
5. Type `kill -s 6` followed by a space and the process id from step 4.
6. Go to about:crashes in Firefox.
7. If the crash report is not submitted, click on it to submit. Otherwise, click on the entry for the crash that was triggered in step 5 (should be the top/last entry).
8. Socorro should open. Verify that a proper stack for the crashing thread is displayed and that the title does not say "Crash in EMPTY: no crashing thread identified; OK"
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: This is a one-line change in code that is currently broken.
[String changes made/needed]: none

Comment 29

[Stephen A Pohl [:spohl]]

Comment on attachment 8918010 [details] [diff] [review]
Patch

Note that we should also uplift bug 1350917 for this to be effective on ESR52.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: We need to have accurate and usable crash reports to understand if crashes may pose a security risk.
User impact if declined: A significant number of crashes would continue to be reported without usable crash stacks.
Fix Landed on Version: 58, uplift-approval to 57 is pending.
Risk to taking this patch (and alternatives if risky): Minimal. This is a one-line change in code that is currently broken.
String or UUID changes made by this patch: none

Comment 30

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8918010 [details] [diff] [review]
Patch

This fix will help us better root cause the content crashes on OS X, Beta57+

Comment 31

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d8aed408f78a

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/cafd9cdafeb7

Comment 33

[Bogdan Maris, Desktop QA]

Stephen, checking your steps from comment 28, I saw that after running 'ps ax | grep plugin' with Firefox 57 beta 11 I don't get any plugin-container processes so I can't actually verify it on beta

This is the output from terminal:
> 236   ??  Ss     0:00.09 /System/Library/Frameworks/CoreMediaIO.framework/Resources/VDC.plugin/Contents/Resources/VDCAssistant
> 3419 s000  S+    0:00.01 grep plugin

Do you know another way I can try and verify it on Fx 57 build?

Comment 34

[(not currently active) Ted Mielczarek]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #33)
> Stephen, checking your steps from comment 28, I saw that after running 'ps
> ax | grep plugin' with Firefox 57 beta 11 I don't get any plugin-container
> processes so I can't actually verify it on beta

Those steps might be missing a:
1.5. Navigate to a web URL (i.e. https://mozilla.org/ )

Firefox doesn't launch a content process until a content page gets loaded.

Comment 35

[Stephen A Pohl [:spohl]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #34)
> (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #33)
> > Stephen, checking your steps from comment 28, I saw that after running 'ps
> > ax | grep plugin' with Firefox 57 beta 11 I don't get any plugin-container
> > processes so I can't actually verify it on beta
> 
> Those steps might be missing a:
> 1.5. Navigate to a web URL (i.e. https://mozilla.org/ )
> 
> Firefox doesn't launch a content process until a content page gets loaded.

That's it precisely. Thanks, Ted!

Comment 36

[Bogdan Maris, Desktop QA]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #34)
> (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #33)
> > Stephen, checking your steps from comment 28, I saw that after running 'ps
> > ax | grep plugin' with Firefox 57 beta 11 I don't get any plugin-container
> > processes so I can't actually verify it on beta
> 
> Those steps might be missing a:
> 1.5. Navigate to a web URL (i.e. https://mozilla.org/ )
> 
> Firefox doesn't launch a content process until a content page gets loaded.

Thanks for that.

So I could reproduce the crash with signature [@ EMPTY: no crashing thread identified; OK ] using Fx 56.0 RC which is wontfix:
- bp-3475f6b5-3bdc-4d35-8152-d0ccc0171025
- bp-a518d82c-945f-48b3-977c-ee8d30171025

I verified that the signature for the fixed builds (Fx 57 beta 11 and latest Nightly 58.0a1) on macOS 10.13 is [@ google_breakpad::ReceivePort::WaitForMessage | google_breakpad::CrashGenerationClient::RequestDumpForException ]:
57 Beta
- bp-89148055-d2bc-4e9e-9d64-440e20171025
- bp-a700eb92-300b-490d-ade5-1acfe0171025
- bp-08d47a06-abba-40c8-bd82-41d7b0171025
- bp-1997f1ed-aad1-4d67-89c9-9d4cd0171025
- bp-4a2a3c8e-07fc-4270-b34c-601350171025
- bp-3ce5fbca-2785-4693-87b3-f765d0171025
- bp-72ed90d7-8b83-4881-96f3-93f3e0171025
- bp-9caec6d0-48c4-4225-a36f-83a140171025
- bp-dbd5c403-6db7-48b0-8157-cfa590171025
- bp-eabb7805-8fc3-4ad8-a2c6-eebe90171025

58 Nightly
- bp-ea04e55e-40fb-4414-a9db-994b90171025
- bp-f12d9d73-5287-49cd-8914-384070171025
- bp-60c113b1-33af-4a87-a9c2-a1af30171025
- bp-19498eb4-5272-4c40-8e77-074f80171025
- bp-eaed1cff-718d-4b67-88d2-a31d00171025
- bp-f6b1bf08-6342-43a2-9dec-015d40171025
- bp-915bd08e-6584-4ba4-85d0-35d5c0171025
- bp-0476925f-4463-4d34-8ca7-460b70171025
- bp-40a9d409-19c1-4f47-889e-8e2e00171025
- bp-c012734c-f4f8-41a8-9815-c94720171025

Also checking socorro, I can't see any more crashes for 57 beta 10 or 11. Based on that, I am closing this bug as verified fixed.
https://crash-stats.mozilla.com/signature/?product=Firefox&signature=EMPTY%3A%20no%20crashing%20thread%20identified%3B%20OK&date=%3E%3D2017-10-18T06%3A34%3A00.000Z&date=%3C2017-10-25T06%3A34%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-version&_sort=-date&page=2#summary

Comment 37

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, as I understand this, ESR52 is indeed affected (from https://bugzilla.mozilla.org/show_bug.cgi?id=1405151#c6 and the next few comments) but we may not be detecting the crashes.

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8918010 [details] [diff] [review]
Patch

Should land for esr52.5.0 along with the patch from bug 1350917.

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/f97879a59f5f

Comment 40

[Bogdan Maris, Desktop QA]

Also verified using Firefox 52.5.0esr on macOS 10.13.1.

bp-437d2fdf-0699-4a2e-8ec1-a7f4c0171109
bp-3690982e-ae09-4d0d-9023-a19600171109
bp-e04fc551-14cc-48a8-98de-026420171109
bp-35493adb-3059-437d-a041-5015d0171109
bp-62b11062-f4a8-4568-a3ad-2e9e70171109
bp-0c49a078-0a65-4e38-b47d-eb42f0171109
bp-f74b60e2-5e50-40aa-913b-cebe00171109
bp-9c4238ea-e2d7-4fe6-b6a1-c84a60171109
bp-b892e396-4994-4a1e-8f53-6df090171109
bp-69d90eda-73df-4c25-b184-6cce70171109