1273372 - [EME] Crash in mozilla::gmp::GMPChild::ProcessingError

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect, P1)

Description

[Chris Pearce [:cpearce (Not reading bugmail)]]

This bug was filed from the Socorro interface and is 
report bp-0d607c08-d6fc-4bb8-a49a-97a0d2160515.
=============================================================

We're still seeing a low volume of crashes in  mozilla::gmp::GMPChild::ProcessingError.

I can reproduce this crash if I have my Firefox profile dir behind a junction on my D: drive.

i.e.:

mklink /J c:\Users\cpearce\Desktop\d-firefox d:\firefox
# .. unzip a Nightly into ~/Desktop\d-firefox
cd ~/Desktop/d-firefox/firefox/
./firefox --profile profile --no-remote

# Netflix fails with  F7702-1003-8053000B error.

Looking in about:crashes, I see this crash report:

https://crash-stats.mozilla.com/report/index/eb2527e0-7466-42c3-b0ce-fb0a82160517

This is similar (or pretty much the same as) bug 1236680.

There's 3 crashes in 48, and one of those was me.

We had twenty-something crashes in 45, and none in 46. Presumably because users who had this in 45 gave up watching Netflix in Firefox.

Comment 1

[Bob Owen (:bobowen)]

(In reply to Chris Pearce (:cpearce) from comment #0)

> mklink /J c:\Users\cpearce\Desktop\d-firefox d:\firefox
> # .. unzip a Nightly into ~/Desktop\d-firefox
> cd ~/Desktop/d-firefox/firefox/
> ./firefox --profile profile --no-remote

In bug 1236680, we only specifically allowed the case where the Users folder had been made a junction point.
Instructions for doing this were on sites like lifehacker.com, so probably a reasonable number of people might have done this.

The sandbox deliberately doesn't allow rules for paths that include junction points, so I didn't want to just resolve anything.
That would also be more work than just checking and resolving the Users folder.

> This is similar (or pretty much the same as) bug 1236680.

Wouldn't anything that returns false during GMPChild::AnswerStartPlugin (or anything else I think) cause a similar crash.
Maybe when these return false we should annotate the crash report to pinpoint the problem.
(Although it looks like they were all load failures due to the sandbox, see below.)
 
> There's 3 crashes in 48, and one of those was me.

I think these were possibly all yours, they were within five minutes of each other and most of the meta data looks very similar.

> We had twenty-something crashes in 45, and none in 46. Presumably because
> users who had this in 45 gave up watching Netflix in Firefox.

Bug 1236680 only got uplifted to 46.
It got rejected for Beta uplift to 45 as it was getting late in the cycle and it was only a small number of crashes.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'mozilla::gmp::GMPChild::ProcessingError':
 - nightly (version 50): 88 crashes from 2016-06-06.
 - aurora  (version 49): 49 crashes from 2016-06-07.
 - beta    (version 48): 6 crashes from 2016-06-06.
 - release (version 47): 59 crashes from 2016-05-31.
 - esr     (version 45): 1 crash from 2016-04-07.

Crash volume on the last weeks:
             Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - nightly         15         19         35          5          0          5          5
 - aurora           0         27          3          6          3          2          1
 - beta             1          0          4          1          0          0          0
 - release         11          0         22          4          3          6          5
 - esr              0          0          0          0          0          0          0

Affected platforms: Windows, Mac OS X

Comment 3

[Milan Sreckovic [:milan] (needinfo for best results)]

Anthony, comment 2 non-withstanding, this appears to be the top XP crash, and overly represented with e10s - although unclear that it has anything to do with e10s, because it does show up in both.  We're worried about this blocking e10s on XP rollout.

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Milan Sreckovic [:milan] from comment #3)
> Anthony, comment 2 non-withstanding, this appears to be the top XP crash,
> and overly represented with e10s - although unclear that it has anything to
> do with e10s, because it does show up in both.  We're worried about this
> blocking e10s on XP rollout.

Milan: GMPChild is the IPDL child actor for the GMP child process. So these are not crashes in our chrome or content process, they are crashes in the GMP process. So I don't think they need to block the e10s rollout on WinXP.

I'll look into this, as it's alarming that we're seeing so many of these on WinXP. The GMPs that are crashing here are supposed to be disabled on WinXP!

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

There are a few web pages which recommend people flip the pref to enable unencrypted decoding via the Adobe GMP decoder:

http://www.msfn.org/board/topic/175591-enable-mp4-h264-aac-html5-video-in-firefox-on-windows-xp-without-flash/
http://forums.mozillazine.org/viewtopic.php?f=38&t=3010403


Based on reading those, there's a community of people who are enabling unencrypted decoding via the Adobe GMP decoder, and it's mostly working for them 90% of the time.

I'd guess the crashes we're seeing are those users who have preffed this on.

Anthony: what shall we do about this?

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Chris Pearce (:cpearce) from comment #4)
> ...
> Milan: GMPChild is the IPDL child actor for the GMP child process. So these
> are not crashes in our chrome or content process, they are crashes in the
> GMP process. So I don't think they need to block the e10s rollout on WinXP.

Good news, thanks!


(In reply to Chris Pearce (:cpearce) from comment #5)
> ...
> I'd guess the crashes we're seeing are those users who have preffed this on.

Heh, we were running into this often enough that we landed bug 1235437 - to get the values of "preference that could get you into unsupported paths" in the crash reports.  Comes in handy.

Comment 7

[yanfzhen]

Hi all 

Today use firefox 48, I crash mozilla::gmp::GMPChild::ProcessingError, pls check the crash report: https://crash-stats.mozilla.com/report/index/3b23b5bc-55f3-412d-993e-a0dfe2160818

Comment 8

[AlexKarlovich]

@cpearce Hi Chris,

Seems the issue https://bugzilla.mozilla.org/show_bug.cgi?id=1286685 "Bug 1286685 - Widevine CDM: MediaKeySystemAccess.createMediaKeys returns a promise that doesn't resolve"

could be related to current issue, because data in crash reports is the same: "Firefox 47.0.1 Crash Report [@ mozilla::gmp::GMPChild::ProcessingError ]"

FF 47.1 https://crash-stats.mozilla.com/report/index/8cfe623d-d5a1-45e6-9cb6-d8e332160726

FF nightly 50.0a1 2016-07-25: https://crash-stats.mozilla.com/report/index/7d31c9d0-f08f-448b-9343-71fba2160726

Comment 9

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

I'm going to leave this bug with Chris.

Comment 10

[Chris Pearce [:cpearce (Not reading bugmail)]]

Hi Alex. I'm about to go on leave, so won't be able to look at this for a week, but would you be able to test whether 32bit Firefox also has the same issue on your system? Can you also test whether setting MOZ_DISABLE_GMP_SANDBOX environment variable to 1 makes Widevine work? Thanks!

Comment 11

[AlexKarlovich]

(In reply to PTO until Sep 5 NZ time; Chris Pearce (:cpearce) from comment #10)
> Hi Alex. I'm about to go on leave, so won't be able to look at this for a
> week, but would you be able to test whether 32bit Firefox also has the same
> issue on your system? Can you also test whether setting
> MOZ_DISABLE_GMP_SANDBOX environment variable to 1 makes Widevine work?
> Thanks!

Chris,

FF 48.0.2 64 bit, Win 10 64 bit: Widevine playback successfully starts with MOZ_DISABLE_GMP_SANDBOX = 1!

Comment 12

[Chris Pearce [:cpearce (Not reading bugmail)]]

Bob: Any ideas here? Alex here is having the Widevine CDM load on x64 Firefox only if the sandbox is disabled.

Could this be somehow caused by the DLL manifest?

Comment 13

[Bob Owen (:bobowen)]

(In reply to Chris Pearce (:cpearce) from comment #12)
> Bob: Any ideas here? Alex here is having the Widevine CDM load on x64
> Firefox only if the sandbox is disabled.
> 
> Could this be somehow caused by the DLL manifest?

It's possible.

Alex - is your Firefox profile directory in a different location to the default? On a network drive for example.

Comment 14

[Chris Pearce [:cpearce (Not reading bugmail)]]

Bob, see Alex's response in bug 1286685 comment 6; "No, home user directories stored at my hard drive.". There are some crash reports there with the GMPChild::ProcessingError signature.

Bob: Is there any logging Alex can collect to help us debug this?

Comment 15

[Bob Owen (:bobowen)]

(In reply to Chris Pearce (:cpearce) from comment #14)
> Bob, see Alex's response in bug 1286685 comment 6; "No, home user
> directories stored at my hard drive.". There are some crash reports there
> with the GMPChild::ProcessingError signature.
> 
> Bob: Is there any logging Alex can collect to help us debug this?

Not in an opt build, at least not for the sandbox.

So if it's the StartPlugin message failing due to the sandbox it must be returning here:
https://hg.mozilla.org/mozilla-central/file/tip/dom/media/gmp/GMPLoader.cpp#l163

Or potentially one of the other |return false| below.

Usually it is the load and has either been because:
* the path to the DLL has been rejected in the sandbox rule set-up, given the answers this would be a new issue. We could add logging when the addition of the rule fails (filed bug 1301034), but to get more details of why it failed would be in the chromium sandbox code I think.
* the sandbox interferes with the DLL load, normally this has been due to the side-by-side assembly manifest. In that case it fails in the OS code, so I don't think we can add any logging for this.

Comment 16

[AlexKarlovich]

(In reply to PTO until Sep 19 NZT Chris Pearce (:cpearce) from comment #14)
> Bob, see Alex's response in bug 1286685 comment 6; "No, home user
> directories stored at my hard drive.". There are some crash reports there
> with the GMPChild::ProcessingError signature.
> 
> Bob: Is there any logging Alex can collect to help us debug this?

Gents,

My FF profiles is stored at:
"C:\Users\Aliaksandr_Karlovich\AppData\Roaming\Mozilla\Firefox\Profiles\mv95aykk.VeryNewUser\gmp-widevinecdm\1.4.8.866"

And the same behavior If move FF user profile to any folder on disk, e.g.:
"D:\FF-Profiles\d-profile\gmp-widevinecdm\1.4.8.866" - the same issue:
https://crash-stats.mozilla.com/report/index/da8126b9-a753-472f-b3e3-8fa2b2160914


Please just ping me if I can provide any additional logging.

Regards,

Sasha.

Comment 17

[Bob Owen (:bobowen)]

Hi Alex,

I've added some logging if the path to the CDM isn't being added to the rules correctly.

Could you install the latest Nightly and test with the following two environment variables set before you start Firefox:

MOZ_LOG="SandboxBroker:1,sync"
MOZ_LOG_FILE="%TEMP%\bug1273372sb.txt"

If there is a problem with this part of the sandbox set-up then it will log to that file in your temp directory.

Comment 18

[AlexKarlovich]

(In reply to Bob Owen (:bobowen) from comment #17)
> Hi Alex,
> 
> I've added some logging if the path to the CDM isn't being added to the
> rules correctly.
> 
> Could you install the latest Nightly and test with the following two
> environment variables set before you start Firefox:
> 
> MOZ_LOG="SandboxBroker:1,sync"
> MOZ_LOG_FILE="%TEMP%\bug1273372sb.txt"
> 
> If there is a problem with this part of the sandbox set-up then it will log
> to that file in your temp directory.

Hi Bob,

FF Nightly was updated: 52.0a1 2016-09-20 (64 bit)
Environment variables: http://prntscr.com/ckld02

File "bug1273372sb.txt" was created but it is empty, size is 0 bit: http://prntscr.com/cklc64

Issue with WidevineCDM is not resolved:
https://crash-stats.mozilla.com/report/index/23648397-a88b-4d7b-87de-db6ad2160921


Regards,

Alex.

Comment 19

[Bob Owen (:bobowen)]

So, that should rule out the first bullet from comment 15.

Chris - do we know when we might get a version without the side-by-side assembly manifest?

Comment 20

[Chris Pearce [:cpearce (Not reading bugmail)]]

I'll follow up with Google. Not sure how long it will take to get a new CDM without the side-by-side manifest embedded.

Comment 21

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: 32bit widevinecdm.dll without manifest

Alex: Would you be able to try something for us? Can you try copying this file into your gmp-widevinecdm directory in your Firefox profile? Just overwrite the old DLL. Then try loading a page that uses Widevine, and let us know if it still crashes.

I've removed the side-by-side manifest from this DLL. Testing this would help us validate the theory that this failure is caused by the presence of the side-by-side manifest in the Widevine DLL.

Comment 22

[Chris Pearce [:cpearce (Not reading bugmail)]]

Bob: Are you sure this is caused by the side-by-side assembly manifest? I've removed the manifest from the Widevine CDM 903 DLL (using Visual Studio), and I still can't load the DLL with USER_LOCKDOWN security level. It's loading fine for me with USER_RESTRICTED as we currently are doing.

I also don't think the DLL is signed; `signtool.exe verify widevinecdm.dll` doesn't find any signatures in the DLL. So our previous theory that the signature was broken when we removed the manifest manually doesn't add up I think.

I expect we'll still need the Widevine CDM produced by Google to not have the manifest in order to get down to USER_LOCKDOWN anyway, but could there be some other failure there? I don't see any sandbox logging noting why the sandbox is blocking the load of the DLL without the manifest. Any ideas?

I haven't asked Google to host a DLL without the manifest yet, as I want to try to verify that this fixes the problem (waiting for Alex to confirm), and I want to ensure that we don't need to ask for anything else.

Comment 23

[Bob Owen (:bobowen)]

(In reply to Chris Pearce (:cpearce) from comment #22)
> Bob: Are you sure this is caused by the side-by-side assembly manifest? I've
> removed the manifest from the Widevine CDM 903 DLL (using Visual Studio),
> and I still can't load the DLL with USER_LOCKDOWN security level. It's
> loading fine for me with USER_RESTRICTED as we currently are doing.

No, as I said in comment 15, it is just one of the two things that we've seen cause problems in the past and it doesn't look like it is the first one.

On 64-bit I too can remove the manifest and it still seems to load, this breaks 32-bit where I've tried it before, and as you say this doesn't seem to help with running at USER_LOCKDOWN.
I'll see if I can spot anything else when debugging 64-bit.

> I also don't think the DLL is signed; `signtool.exe verify widevinecdm.dll`
> doesn't find any signatures in the DLL. So our previous theory that the
> signature was broken when we removed the manifest manually doesn't add up I
> think.

Both of the 32-bit and 64-bit versions that get downloaded to my profiles have signatures, which gets broken when the side-by-side assembly is removed.
Odd that it doesn't appear to interfere with the loading for 64-bit.

Comment 24

[Bob Owen (:bobowen)]

(In reply to Bob Owen (:bobowen) from comment #23)

> ... and as you say this doesn't seem
> to help with running at USER_LOCKDOWN.

Not sure what I did before, but now I do seem to be able to get this running at USER_LOCKDOWN, with the manifest removed.

Comment 25

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Bob Owen (:bobowen) from comment #24)
> (In reply to Bob Owen (:bobowen) from comment #23)
> 
> > ... and as you say this doesn't seem
> > to help with running at USER_LOCKDOWN.
> 
> Not sure what I did before, but now I do seem to be able to get this running
> at USER_LOCKDOWN, with the manifest removed.

In 32bit or 64bit?

You're right about removing the manifest breaking the signature, but I have trouble believing that the presence of the authenticode signature would interact with the sandbox security level; based on some googling, the signature seems mainly to affect how Windows Smart Screen prompts to execute stuff downloaded from the internet. Or computers which are configured to run only signed executables.

Comment 26

[Bob Owen (:bobowen)]

(In reply to Chris Pearce (:cpearce) from comment #25)
> (In reply to Bob Owen (:bobowen) from comment #24)
> > (In reply to Bob Owen (:bobowen) from comment #23)
> > 
> > > ... and as you say this doesn't seem
> > > to help with running at USER_LOCKDOWN.
> > 
> > Not sure what I did before, but now I do seem to be able to get this running
> > at USER_LOCKDOWN, with the manifest removed.
> 
> In 32bit or 64bit?

64-bit, if I remove the manifest for 32-bit, the load fails even with the sandbox turned off.

I don't think the broken signature is affecting the sandbox.
Two separate but linked things:

* manifest removal, which affects signature and for 32-bit causes the load to fail (possibly down to the signature, but may not be).

* for 64-bit, dll still loads after manifest is removed and it then appears to be fine loading at USER_LOCKDOWN, suggesting that it is the manifest that is causing that problem.

Whether this is related to Alex's problem, is still unconfirmed.

Alex - would you mind trying with the 64-bit CDM that Chris provided in comment 21, thanks.

Comment 27

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: 64bit widevinecdm.dll with manifest removed

Here's the 64bit Widevine CDM with the manifest removed.

Alex, can you try this in your 64bit Firefox?

Comment 28

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Bob Owen (:bobowen) from comment #26)
> Alex - would you mind trying with the 64-bit CDM that Chris provided in
> comment 21, thanks.

The first CDM I uploaded was 32 bit, I've now uploaded the 64 bit one with manifest removed in comment 27.

Comment 29

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Screenshot: secpol.msc Executable Rules

Alex, can you try something to help us debug this? Can you try opening your Start menu, search for secpol.msc, run that, and open Application Control Policies > AppLocker > Executable Rules, and see if there's any rules in there? I wonder if your system is somehow configured to not allow the CDM binary to load from it's location.

Screenshot of secpol.msc showing this screen attached.

Widevine is working fien in Chrome for you I take it?

Comment 30

[AlexKarlovich]

(In reply to Chris Pearce (:cpearce) from comment #27)
> Created attachment 8796063 [details]
> 64bit widevinecdm.dll with manifest removed
> 
> Here's the 64bit Widevine CDM with the manifest removed.
> 
> Alex, can you try this in your 64bit Firefox?

Checked with specified attach on FF 64bit, the same result, Widevine has crashed.

Yep, Chrome has no such problems to play Widevine assets.

Comment 31

[AlexKarlovich]

Attachment: Screenshot of AppLocker Rules

No rules here (at AppLocker->Executable Rules)

But I can add that this issue constantly reproduced on our Corporate User Machines (Window 7 & 10) (at least 10 computers affected)

Checked on three personal windows computers - Widevine assets playback is well.

Playback under Chrome is fine on all machines.

Comment 32

[Bob Owen (:bobowen)]

(In reply to AlexKarlovich from comment #31)

Thanks Alex.

I'm wondering if the loaded dependencies, before the sandbox is locked down are different on those machines and it then attempts to load new ones with the CDM, which would break.

I'll file another bug, to change the sandbox logging that we do have to log with MOZ_LOG (if I can), so we can see them in opt builds.
It doesn't cover everything that the sandbox might block, but at least it's something.

Might be Monday, before I get to it, setting ni to remind me.

Comment 33

[Bob Owen (:bobowen)]

Hi Alex,

The updated logging has landed in Nightly.
So, if you could update to the latest Nightly and this time run with the following environment variables set before you start Nightly:

NSPR_LOG_MODULES="SandboxTarget:5,sync"
NSPR_LOG_FILE="%TEMP%\bug1273372sb.txt"
MOZ_WIN_SANDBOX_LOGGING=1

This time it should also create files like this: bug1273372sb.txt.child-*

One of these should have messages about loading widevinecdm.dll.
Please could you attach that to the bug.

Thanks for your help with this.

Comment 34

[AlexKarlovich]

Attachment: log.zip

Comment 35

[AlexKarlovich]

Comment on attachment 8800240 [details]
log.zip

Hi Bob,


Created log files are empty at "C:/Windows/Temp" for some reasons: http://prntscr.com/csy356

Locate log files to disk D:
http://prntscr.com/csy5c9

Please see created log files in attach.


Crash report:
https://crash-stats.mozilla.com/report/index/6f1fb967-fa9b-497f-8f54-b30a02161012

FF version: 52.0a1 2016-10-12 (64-bit)

WidevineCDM is native (not modified, e.g. not from attach in #Comment27)



Regards,

Alex.

Comment 36

[Bob Owen (:bobowen)]

Thanks Alex,

Looks like the following registry key is getting blocked during the load:
\Registry\Machine\System\CurrentControlSet\Control\Srp\GP\

Under here is one of the places where AppLocker rules are stored.
So, it looks like it's trying to read the rules anyway.

With that in mind I'll see if I can reproduce, I'll also create a build with that key allowed.

Comment 37

[Chris Peterson [:cpeterson]]

I think I just hit this crash when trying to load the Shaka Player demo page in 64-bit Nightly 52. After dismissing a couple info bar notifications about enabling DRM and downloading CDMs, the Adobe Primetime CDM crashed:

http://shaka-player-demo.appspot.com/demo/

bp-33ad1061-60ef-4917-8d2f-525202161012

Comment 38

[Bob Owen (:bobowen)]

(In reply to Bob Owen (:bobowen) from comment #36)
> Thanks Alex,
> 
> Looks like the following registry key is getting blocked during the load:
> \Registry\Machine\System\CurrentControlSet\Control\Srp\GP\
> 
> Under here is one of the places where AppLocker rules are stored.
> So, it looks like it's trying to read the rules anyway.
> 
> With that in mind I'll see if I can reproduce, I'll also create a build with
> that key allowed.

Having read a bit more about AppLocker I suspect we're going to have to add more that just that rule.
I think the best course of action is to reproduce.

Unfortunately AppLocker is only fully available on Enterprise versions, which I don't have a VM for currently.
So I need to wait for my MSDN licence to get renewed (thanks for the lack of warning about that MS!). Shouldn't take too long.

Comment 39

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Peterson [:cpeterson] from comment #37)
> I think I just hit this crash when trying to load the Shaka Player demo page
> in 64-bit Nightly 52. After dismissing a couple info bar notifications about
> enabling DRM and downloading CDMs, the Adobe Primetime CDM crashed:
> 
> http://shaka-player-demo.appspot.com/demo/
> 
> bp-33ad1061-60ef-4917-8d2f-525202161012

That's very interesting. I note that eme-adobe.dll is missing from the modules list. So it would seem that in your case we somehow tried to start the CDM even though it's not installed.

Comment 40

[Bob Owen (:bobowen)]

I can reproduce, but only if I have "DLL Rules" configured in AppLocker.
Do you have any configured?

Any rules seem to cause a problem.

Comment 41

[AlexKarlovich]

(In reply to Bob Owen (:bobowen) from comment #40)
> I can reproduce, but only if I have "DLL Rules" configured in AppLocker.
> Do you have any configured?
> 
> Any rules seem to cause a problem.

Hi Bob,

Strange thing:
No rules via UI: https://bug1273372.bmoattachments.org/attachment.cgi?id=8796128
But Count is 165 in register: http://prntscr.com/d51jb4


So could be any way to fix this issue?

Comment 42

[Bob Owen (:bobowen)]

(In reply to AlexKarlovich from comment #41)
> (In reply to Bob Owen (:bobowen) from comment #40)
> > I can reproduce, but only if I have "DLL Rules" configured in AppLocker.
> > Do you have any configured?
> > 
> > Any rules seem to cause a problem.
> 
> Hi Bob,
> 
> Strange thing:
> No rules via UI:
> https://bug1273372.bmoattachments.org/attachment.cgi?id=8796128
> But Count is 165 in register: http://prntscr.com/d51jb4
> 
> 
> So could be any way to fix this issue?

Hmm, that is odd.
I've been tied up with other things, but hopefully I can look at this early next week and try and find acceptable rules to add to the sandbox. It will require a couple of changes to the chromium sandbox code as well.

I'll let you know as soon as I have something that you can test with.

Comment 43

[Bob Owen (:bobowen)]

Finally had time to look at this, here's a try push with some changes that work for me on Win7 and Win10 Enterprise:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=86883cf14d26107640b0ef0b489a48336d04a433

Alex, can you try out those builds, thanks.

Comment 44

[AlexKarlovich]

(In reply to Bob Owen (:bobowen) from comment #43)
> Finally had time to look at this, here's a try push with some changes that
> work for me on Win7 and Win10 Enterprise:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=86883cf14d26107640b0ef0b489a48336d04a433
> 
> Alex, can you try out those builds, thanks.

Hi Bob,

Thanks a lot, seems the issue was resolved.

The next flow was tested:
1. Just to be sure that bug still exists in Nightly 53.0a1(2016-11-27) open shaka demo page (http://shaka-player-demo.appspot.com/demo/): http://prnt.sc/dcp0w6
2. Delete FF Nightly.
3. Install custom version (firefox-53.0a1.en-US.win64.installer.exe) from: https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-86883cf14d26107640b0ef0b489a48336d04a433/try-win64/
4. Check again - no errors on shaka demo page regarding widevine CDM: http://prntscr.com/dcpm68


Just a small note:
On custom version of Nightly FF something wrong with media source extensions, our playback is failed with the next errors in console: http://prntscr.com/dcppoi ("Shaka Error MEDIA.MEDIA_SOURCE_OPERATION_THREW (InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable"). And no errors appeared at "about:crashes"

On FF 50.0 playback of the same asset is good (with MOZ_DISABLE_GMP_SANDBOX=1 flag).

But seems that is another story :)



Guys, thanks all for your effort, great work!


Sasha.

Comment 45

[Chris Pearce [:cpearce (Not reading bugmail)]]

Thanks Bob! This was a tough one. Can we get this landed. :)

Comment 46

[Bob Owen (:bobowen)]

Attachment: Part 1: Backout change to allow network drives in sandbox rules

Comment 47

[Bob Owen (:bobowen)]

Attachment: Part 2: Re-apply change to allow network drives in sandbox rules with non-file device fix

Comment 48

[Bob Owen (:bobowen)]

Attachment: Part 2: Re-apply change to allow network drives in sandbox rules with non-file device fix

Comment 49

[Bob Owen (:bobowen)]

Attachment: Part 3: Add KEY_WOW64_64Key and KEY_WOW64_32KEY to the Chromium sandbox allowed registry read flags

Comment 50

[Bob Owen (:bobowen)]

Attachment: Part 4: Add AppLocker rules to GMP sandbox policy

Comment 51

[Bob Owen (:bobowen)]

Attachment: Part 2 as if done without the backout

Comment 52

[Bob Owen (:bobowen)]

Comment on attachment 8817222 [details] [diff] [review]
Part 1: Backout change to allow network drives in sandbox rules

r=me - This is just a backout of the original changes, so I don't think this needs a review.

Comment 53

[Bob Owen (:bobowen)]

Comment on attachment 8817225 [details] [diff] [review]
Part 2: Re-apply change to allow network drives in sandbox rules with non-file device fix

This re-applies the network drive changes with some changes, so that I don't break adding \Device\* rules.

I've also uploaded "Part 2 as if done without the backout", which as it states just contains the extra changes on top of current m-c.

Comment 54

[Bob Owen (:bobowen)]

Comment on attachment 8817226 [details] [diff] [review]
Part 3: Add KEY_WOW64_64Key and KEY_WOW64_32KEY to the Chromium sandbox allowed registry read flags

I don't think there should be an issue allowing the flags for read access.

Comment 55

[Bob Owen (:bobowen)]

Just removing this needinfo now.

Thanks for all your help in tracking down the problem.

Comment 56

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6fc10f2c30c9
Part 1: Backout change to allow network drives in sandbox rules. r=backout
https://hg.mozilla.org/integration/mozilla-inbound/rev/c70d06fa5302
Part 2: Re-apply change to allow network drives in sandbox rules with non-file device fix. r=aklotz
https://hg.mozilla.org/integration/mozilla-inbound/rev/d24db55deb85
Part 3: Add KEY_WOW64_64Key and KEY_WOW64_32KEY to the Chromium sandbox allowed registry read flags. r=aklotz
https://hg.mozilla.org/integration/mozilla-inbound/rev/9174d825a6ee
Part 4: Add AppLocker rules to GMP sandbox policy. r=aklotz

Comment 57

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/6fc10f2c30c9
https://hg.mozilla.org/mozilla-central/rev/c70d06fa5302
https://hg.mozilla.org/mozilla-central/rev/d24db55deb85
https://hg.mozilla.org/mozilla-central/rev/9174d825a6ee

Comment 58

[Sylvestre Ledru [:Sylvestre]]

Is it something you are considering uplifting to at least 52/aurora? Thanks

Comment 59

[Bob Owen (:bobowen)]

Comment on attachment 8817222 [details] [diff] [review]
Part 1: Backout change to allow network drives in sandbox rules

(In reply to Sylvestre Ledru [:sylvestre] [PTO Jan 3rd] from comment #58)
> Is it something you are considering uplifting to at least 52/aurora? Thanks

Yes to 51 I think and I've just checked that it applies cleanly to beta, so should be OK for aurora as well.

Approval Request Comment
[Feature/Bug causing the regression]:
Will have been an issue since launch of GMP process sandbox.

[User impact if declined]:
Users on Enterprise versions of Windows, may be unable to use GMP/EME if also using AppLocker.

[Is this code covered by automated tests?]:
GMP/EME has automated tests.

[Has the fix been verified in Nightly?]:
It was verified on a try build.

[Needs manual test from QE? If yes, steps to reproduce]:
No.

[List of other uplifts needed for the feature/fix]:
None.

[Is the change risky?]:
Small risk.

[Why is the change risky/not risky?]:
This change includes a small change to the chromium sandbox code to fix an issue previously introduced, that meant we couldn't add the rules needed for AppLocker.

[String changes made/needed]:
No.

Comment 60

[Bob Owen (:bobowen)]

Comment on attachment 8817225 [details] [diff] [review]
Part 2: Re-apply change to allow network drives in sandbox rules with non-file device fix

See comment 59.

Comment 61

[Bob Owen (:bobowen)]

Comment on attachment 8817226 [details] [diff] [review]
Part 3: Add KEY_WOW64_64Key and KEY_WOW64_32KEY to the Chromium sandbox allowed registry read flags

See comment 59.

Comment 62

[Bob Owen (:bobowen)]

Comment on attachment 8817227 [details] [diff] [review]
Part 4: Add AppLocker rules to GMP sandbox policy

See comment 59.

Comment 63

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8817222 [details] [diff] [review]
Part 1: Backout change to allow network drives in sandbox rules

Should allow better video playback for enterprise users. 
Let's try this on aurora and beta (51.0b11) -- but if there are any problems in landing on beta or we see regressions, please back out from 51 right away since we are close to release.

Comment 64

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/a3caf184f1d196576442bdd7dc315b18113035a0
https://hg.mozilla.org/releases/mozilla-aurora/rev/c3c9fdf4606a0fae9df2dbc20df4101314d52eb5
https://hg.mozilla.org/releases/mozilla-aurora/rev/40c74f5059db401c512937c1fd6406e02bf371a6
https://hg.mozilla.org/releases/mozilla-aurora/rev/741296a2d92b93348fe573b2b5884d6421ba4282

Comment 65

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/5339b68d8a2a96e1f6c90544fc32e1408536671e
https://hg.mozilla.org/releases/mozilla-beta/rev/00780d8c7b9421d4d3e714dc9b00cbeb469c9c41
https://hg.mozilla.org/releases/mozilla-beta/rev/a9402f818972a194d452af23452744a9441e077e
https://hg.mozilla.org/releases/mozilla-beta/rev/4284001e0ba8c1c1d02301f15ca596b55f971d37

Comment 66

[(inactive) Jim Chen [:jchen] [:darchons]]

We're still seeing "mozilla::gmp::GMPChild::ProcessingError" crashes (e.g. bp-40e049bb-0993-49a3-88ad-215712170111). Any ideas?

Comment 67

[Bob Owen (:bobowen)]

(In reply to Jim Chen [:jchen] [:darchons] from comment #66)
> We're still seeing "mozilla::gmp::GMPChild::ProcessingError" crashes (e.g.
> bp-40e049bb-0993-49a3-88ad-215712170111). Any ideas?

Not really, this is a fairly generic error which could have a number of causes.

Very often in the past it has been due to the sandbox blocking the load of the DLL, so these could be similar problems.
Without more specific bug reports, that have usually come from users, it is difficult to diagnose.
This is because the load is blocked by the OS and we just fail and crash later on.

I've filed bug 1330930, to look into adding more information to the crash reports to try and work out where and how these are failing.