1556846 - [10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName]

Status: VERIFIED FIXED

(Core Graveyard :: Plug-ins, defect, P1)

Description

[Marcia Knous [:marcia]]

This bug is for crash report bp-d40ce24a-7d3a-4a14-ab28-63b350190604.

Seen while looking at nightly crash stats. There are a few different signatures that appear to have a similar stack. Most of them are single install crashes.

• https://bit.ly/2WojowR
• https://bit.ly/2KqqLwt

We had a somewhat similar signature back in the 58 timeframe: Bug 1419004

Top 10 frames of crashing thread:

0  @0x7fff6c9b47a2 
1  @0x7fff6c99474d 
2  @0x7fff6c9958c6 
3  @0x7fff6c9b3464 
4  @0x7fff6cc07c64 
5  @0x7fff6c99474d 
6  @0x7fff6c9958c6 
7  @0x7fff6cc07bba 
8  @0x7fff36104d5a 
9  @0x7fff6c99474d 

Comment 1

[Andrew McCreight [:mccr8]]

Stephen, it looks like you fixed a similar crash a few years ago (bug 1419004). Any chance you could take a look at this? Thanks.

Comment 2

[Andrew McCreight [:mccr8]]

This is the #2 top crash in the June 12 Nightlies, behind only bug 1558836, which was a pretty bad crash regression.

Comment 3

[Andrew McCreight [:mccr8]]

Adding some more related signatures. These two signature each had a dozen or so crash reports, but only from a single installation each.

Comment 4

[Andrew McCreight [:mccr8]]

Some more signatures.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Any idea why this would only be affecting Nightlies? Something sandbox-related maybe?

Comment 6

[Marcia Knous [:marcia]]

Maybe this should move to Core | AV? The process type is rdd (web).

Comment 7

[Stephen A Pohl [:spohl]]

I just installed 10.15 and should be able to take a look this week. Leaving n-i set.

Comment 8

[Marcia Knous [:marcia]]

This is currently the #7 overall top crash on nightly. spohl: have you had a chance to look at it yet? I can confirm there are crashes in both the first and second betas. Thanks.

Comment 9

[Stephen A Pohl [:spohl]]

No progress to report, but still looking into it. Keeping n-i set.

Comment 10

[Andrew McCreight [:mccr8]]

There are still tons of these crashes on the OSX Nightly, with various other signatures.

Comment 11

[Stephen A Pohl [:spohl]]

Investigations here are largely blocked by bug 1562684.

Comment 12

[Stephen A Pohl [:spohl]]

With the macOS 10.15 Beta 3 update, I'm not even able to build Firefox anymore because Python 2 is crashing. This will slow down investigations significantly.

Comment 13

[Marcia Knous [:marcia]]

I took a look at some of the correlations on nightly for the first signature (https://crash-stats.mozilla.org/signature/?signature=mozilla%3A%3Aplugins%3A%3APluginUtilsOSX%3A%3ASetProcessName).

(92.17% in signature vs 58.44% overall) Addon "[e-mail]" = true
(31.18% in signature vs 00.83% overall) Addon "[e-mail]" = true
(100.0% in signature vs 10.12% overall) Module "Backup" = true [203.33% vs 10.61% if startup_crash = null]
(100.0% in signature vs 07.48% overall) platform_pretty_version = OS X 10.15 [100.0% vs 73.90% if platform = Mac OS X]
(55.91% in signature vs 28.37% overall) Addon "[e-mail]" Version = 1.20.0 [55.91% vs 35.37% if process_type = rdd]

Interesting that there is a fairly high correlation to the amazon search engine.

Comment 14

[Haik Aftandilian [:haik]]

I think this is a sandboxing issue. I'm working on confirming that and will update the bug after I do some more testing.

While debugging why AV1 decoding is not working on 10.15, I found that the RDD process is crashing here in a SetProcessName() stack.

The message "BUG IN LIBDISPATCH: Unable to get the unique pid (size)" indicates the proc_pidinfo() syscall is failing and this is likely to be due to sandbox restrictions. This is one of the problems we hit with Widevine decoding on 10.15 on bug 1558924 and I didn't realize the SetProcessName() crash was similar.

https://hg.mozilla.org/mozilla-central/rev/83204e889b721c109c7296abd86b592553f825c5

(lldb) c
Process 50468 resuming
Process 50468 stopped
* thread #1, name = 'MainThread', queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00007fff6b66b762 libdispatch.dylib`_firehose_task_buffer_init + 266
libdispatch.dylib`_firehose_task_buffer_init:
->  0x7fff6b66b762 <+266>: ud2    
    0x7fff6b66b764 <+268>: cltq   
    0x7fff6b66b766 <+270>: leaq   0x15010(%rip), %rcx       ; "BUG IN LIBDISPATCH: Unable to get the unique pid (size)"
    0x7fff6b66b76d <+277>: movq   %rcx, 0x2ca484dc(%rip)    ; gCRAnnotations + 8
Target 0: (plugin-container) stopped.
(lldb) bt
* thread #1, name = 'MainThread', queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
  * frame #0: 0x00007fff6b66b762 libdispatch.dylib`_firehose_task_buffer_init + 266
    frame #1: 0x00007fff6b64b60e libdispatch.dylib`_dispatch_client_callout + 8
    ...
    stack frames omitted to save space
    ...
    frame #30: 0x00000001185138d2 XUL`mozilla::plugins::PluginUtilsOSX::SetProcessName(char const*) + 130
    frame #31: 0x000000011814cf16 XUL`mozilla::RDDParent::Init(int, char const*, MessageLoop*, IPC::Channel*) + 214
    frame #32: 0x000000011814ec2d XUL`mozilla::RDDProcessImpl::Init(int, char**) + 445
    frame #33: 0x000000011a06b669 XUL`XRE_InitChildProcess(int, char**, XREChildData const*) + 4297
    frame #34: 0x000000010aac5f07 plugin-container`main + 103
    frame #35: 0x00007fff6b69bc49 libdyld.dylib`start + 1

Comment 15

[Haik Aftandilian [:haik]]

Disassembly of libdispatch.dylib`_firehose_task_buffer_init confirms the message BUG IN LIBDISPATCH: Unable to get the unique pid (size) is used after a call to the proc_pidinfo syscall fails.

Adding (allow process-info-pidinfo (target self)) to the utility sandbox policy (used by the RDD process) avoids the crash and allows AV1 decoding to work in my limited testing so far.

Comment 16

[Haik Aftandilian [:haik]]

Attachment: Bug 1556846 - [10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName] r?spohl

Allow limited access to the proc_pidinfo() syscall from the Mac utility process sandbox.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c3b98f220f4f
[10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName] r=spohl

Comment 18

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/c3b98f220f4f

Comment 19

[Haik Aftandilian [:haik]]

I'll request uplift once this has been on Nightly for a bit longer. We will want the fix on 68 and 69.

Comment 20

[Marcia Knous [:marcia]]

This is looking good on nightly so far. The main signature in this bug has not been seen since the fix landed, and I am not seeing any of the single installation Mac crashes in my nightly crash triage in the last few days :)

Comment 21

[Haik Aftandilian [:haik]]

Comment on attachment 9077025 [details]
Bug 1556846 - [10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName] r?spohl

Beta/Release Uplift Approval Request

• User impact if declined: Some AV1 media decoding will fail on macOS 10.15 due to the Remote Data Decoder (RDD) process crashing every time it launches.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: On macOS 10.15 Beta, visit https://bitmovin.com/demos/av1 and ensure the AV1 embedded video plays.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The fix is small, limited to macOS, and just adds a new process sandbox rule needed on 10.15 that is unlikely to cause problems with earlier macOS versions.
• String changes made/needed: None

Comment 22

[Haik Aftandilian [:haik]]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #20)

This is looking good on nightly so far. The main signature in this bug has not been seen since the fix landed, and I am not seeing any of the single installation Mac crashes in my nightly crash triage in the last few days :)

Thanks for verifying that, Marcia.

Comment 23

[ovidiu boca[:Ovidiu]]

Attachment: Screenshot 2019-07-16 at 14.35.34.png

I tested this on iMac and MacBook Pro OS X 10.15 Beta (19A501i) with the latest FF Nightly 70.0a1(2019-07-16) and I can't reproduce any crash but there is a problem, the video doesn't start and there is an error displayed, please see the attached document.
Haik, can you please take a look at this? Thanks

Comment 24

[Marcia Knous [:marcia]]

I just tested on my 10.15 machine using https://bitmovin.com/demos/av1 and the latest nightly, and the video does play fine for me. I am running (19A501i) as well.

Comment 25

[Haik Aftandilian [:haik]]

(In reply to ovidiu boca[:Ovidiu] from comment #23)

Created attachment 9078380 [details]
Screenshot 2019-07-16 at 14.35.34.png

I tested this on iMac and MacBook Pro OS X 10.15 Beta (19A501i) with the latest FF Nightly 70.0a1(2019-07-16) and I can't reproduce any crash but there is a problem, the video doesn't start and there is an error displayed, please see the attached document.
Haik, can you please take a look at this? Thanks

That's the error you should see without the fix. The RDD process is crashing, but it doesn't cause a crashed tab page like a content process crash would.

Once you have the fix, you should not see that error and instead should see the video playing.

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9077025 [details]
Bug 1556846 - [10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName] r?spohl

Fixes an RDD crash on OSX 10.15. Approved for 69.0b6.

Comment 27

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6ab5b8214d16

Comment 28

[Julien Cristau [:jcristau]]

Comment on attachment 9077025 [details]
Bug 1556846 - [10.15] Crash in [@ mozilla::plugins::PluginUtilsOSX::SetProcessName] r?spohl

crash fix for macos 10.15, approved for 68.0.1 and 68.1esr

Comment 29

[ovidiu boca[:Ovidiu]]

Tested again on MacBook OS X 10.15 Beta (19A512f) with FF Nightly 70.0aa1(2019-07-17) and the error is still there.

Comment 30

[Stephen A Pohl [:spohl]]

(In reply to ovidiu boca[:Ovidiu] from comment #29)

Tested again on MacBook OS X 10.15 Beta (19A512f) with FF Nightly 70.0aa1(2019-07-17) and the error is still there.

Is there some kind of macOS version and Firefox version combination that plays this successfully for you?

Comment 31

[ovidiu boca[:Ovidiu]]

For example, Mac 10.14 with FF Nightly 70.0a1(2019-07-17) is working for me.

Comment 32

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-release/rev/f0aaf22db57d

Comment 33

[Ryan VanderMeulen [:RyanVM]]

Per discussion with jcristau, we're uplifting this to 68.0.1esr also to maintain parity with the non-ESR 68.0.1 release and hopefully avoid some confusion.

Comment 34

[Haik Aftandilian [:haik]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #33)

Per discussion with jcristau, we're uplifting this to 68.0.1esr also to maintain parity with the non-ESR 68.0.1 release and hopefully avoid some confusion.

Thanks for catching that. ESR 68 should have been in my uplift request. This fix is needed in ESR 68 because the bug affects all Firefox versions with the RDD process enabled (65+) run on macOS 10.15 and we'll be supporting ESR 68 on 10.15.

Comment 35

[Ryan VanderMeulen [:RyanVM]]

default (68.1esr): https://hg.mozilla.org/releases/mozilla-esr68/rev/63288163530510b1393a90b3a1f196c196c88fc8
FIREFOX_ESR_68_0_X_RELBRANCH (68.0.1esr): https://hg.mozilla.org/releases/mozilla-esr68/rev/6825ce124f4a7a9a2240eaa655657e05d3550832

Comment 36

[Rares Doghi, Desktop QA]

Hi everyone, after further testing this issue I can safely say that this issue no longer occurs on Esr 68.0.1 or Release candidate 68.0.1, the video starts playing without issues.

It DOES however still occur on Beta 69.0b5 as well as our latest Nightly 70.0a1 (2019-07-18).

Comment 37

[Haik Aftandilian [:haik]]

(In reply to Rares Doghi from comment #36)

It DOES however still occur on Beta 69.0b5 as well as our latest Nightly 70.0a1 (2019-07-18).

I'll look into this. I upgraded to the newest macOS Beta yesterday (19A512f) and am seeing this problem too with Beta and Nightly.

Comment 38

[Marcia Knous [:marcia]]

(In reply to Haik Aftandilian [:haik] from comment #37)

(In reply to Rares Doghi from comment #36)

It DOES however still occur on Beta 69.0b5 as well as our latest Nightly 70.0a1 (2019-07-18).

I'll look into this. I upgraded to the newest macOS Beta yesterday (19A512f) and am seeing this problem too with Beta and Nightly.

Same here - using (19A512f) the video is now giving me an error on the latest nightly :(.

Comment 39

[Haik Aftandilian [:haik]]

Mozregression run on macOS 10.15 Beta 4 (19A512f)

$ mozregression --good 2019-07-10 --bad 2019-07-18

led me to https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5671bf0f8a214d690b4754aab8e6304102c1b791&tochange=368e9ae19115fce103b410d423d78076d59db7e7 indicating bug 1560368 caused this regression.

But bug 1560368 is only on 70 which wouldn't explain the issue on Beta.

I'll file a new bug post details there.

Comment 40

[Andrew McCreight [:mccr8]]

There's already a bug on the regression from bug 1560368. I'll see if I can find it...

Comment 41

[Andrew McCreight [:mccr8]]

Bug 1566540 is what I was thinking of.

Comment 42

[Michael Froman [:mjf]]

(In reply to Haik Aftandilian [:haik] from comment #39)

Mozregression run on macOS 10.15 Beta 4 (19A512f)

$ mozregression --good 2019-07-10 --bad 2019-07-18

led me to https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5671bf0f8a214d690b4754aab8e6304102c1b791&tochange=368e9ae19115fce103b410d423d78076d59db7e7 indicating bug 1560368 caused this regression.

But bug 1560368 is only on 70 which wouldn't explain the issue on Beta.

I'll file a new bug post details there.

Haik, I'm not exactly sure what the question here is for me. What I've gathered from reading through the 2 bugs (this one and Bug 1566540) is that there is possibly a sandbox issue related only to RDD and macOS 10.15? Please let me know what I can do to help. I don't have a 10.15 machine here, but if necessary I can probably track down a spare drive to install the 10.15 beta on.

Comment 43

[Haik Aftandilian [:haik]]

(In reply to Michael Froman [:mjf] from comment #42)

Haik, I'm not exactly sure what the question here is for me. What I've gathered from reading through the 2 bugs (this one and Bug 1566540) is that there is possibly a sandbox issue related only to RDD and macOS 10.15? Please let me know what I can do to help. I don't have a 10.15 machine here, but if necessary I can probably track down a spare drive to install the 10.15 beta on.

Sorry for not being clear. Originally, due to the mozregression result, the crash looked like a possible regression caused by bug 1560368. Now, after looking at the crash some more, I think it could be a new 10.15 sandboxing issue exposed by the RDD changes in bug 1560368. I'll work on bug 1566540 assuming it is a sandboxing issue.

Comment 44

[Ryan VanderMeulen [:RyanVM]]

(In reply to Rares Doghi from comment #36)

Hi everyone, after further testing this issue I can safely say that this issue no longer occurs on Esr 68.0.1 or Release candidate 68.0.1, the video starts playing without issues.

It DOES however still occur on Beta 69.0b5 as well as our latest Nightly 70.0a1 (2019-07-18).

Are we able to call this verified for 69/70 now?

Comment 45

[ovidiu boca[:Ovidiu]]

Hi Ryan, we tested this issue on Mac OS X 10.15 Beta 5 but we are blocked by bug 1570451, however, we manage to test this issue using an older OS 10.15 Beta version and with the Latest Nightly 70.0a1 (2019-08-15) and Beta 69.0b14 the issue is no longer reproducible.
So to answer your question, yes we can call this verified for 69/70.

Comment 46

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This bug has been identified as part of a pilot on determining root causes of blocking and dot release drivers.

It needs a root-cause set for it. Please see the list at https://docs.google.com/document/d/1FFEGsmoU8T0N8R9kk-MXWptOPtXXXRRIe4vQo3_HgMw/.

Add the root cause as a whiteboard tag in the form [rca - <cause> ] and remove the rca-needed keyword.

If you have questions, please contact :tmaity.