1286480 - [10.12] Widevine CDM always crashes on Amazon since upgrade to macOS Sierra

Status: VERIFIED FIXED

(Core :: Audio/Video: GMP, defect, P1)

Description

[Sören Hentzschel]

Attachment: screenshot

Every time I try to watch something on Amazon Prime the Widevine CDM crashes. It never crashed on OS X 10.11, it happens since the upgrade to macOS Sierra (public beta). It's reproducible on stable Firefox 47.0.1 and current Firefox Nightly 50.

Comment 1

[Marcia Knous [:marcia]]

(In reply to Sören Hentzschel from comment #0)
> Created attachment 8770435 [details]
> screenshot
> 
> Every time I try to watch something on Amazon Prime the Widevine CDM
> crashes. It never crashed on OS X 10.11, it happens since the upgrade to
> macOS Sierra (public beta). It's reproducible on stable Firefox 47.0.1 and
> current Firefox Nightly 50.

Sören: Can you please submit a crash report and then add it to the bug? Thanks!

Comment 2

[Sören Hentzschel]

Where can I find the crash report? There is no crash report on about:crashes after clicking the button "submit a crash report".

Comment 3

[Chris Peterson [:cpeterson]]

I don't see any Widevine crash reports submitted from macOS 10.12.x users:

https://is.gd/3rsROR

Comment 4

[Maire Reavy [:mreavy]]

Chris -- Do you agree this is a P1?  If so, do you want to take it or find someone to take it?  Or ask Anthony to find someone?  Thanks.

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Peterson [:cpeterson] from comment #3)
> I don't see any Widevine crash reports submitted from macOS 10.12.x users:
> 
> https://is.gd/3rsROR

You need to search for "gmp plugin" "exists", rather than "product" "has terms" "firefox". i.e.:"

http://bit.ly/2aafdez

Comment 6

[123joseffg]

I can confirm that this is a bug. Having the same issue myself.

Comment 7

[Chris Pearce [:cpearce (Not reading bugmail)]]

Haik: Can you reproduce this on MacOS Sierra Beta? Is it a sandboxing issue?

Comment 8

[Chris Pearce [:cpearce (Not reading bugmail)]]

Haik: I installed macOS Sierra on a spare mac mini we had lying around the office, and I if I spoof my useragent to Chrome's and try to play Netflix (so that Netflix serves us Widevine content) I see the CDM crash.

I see this logged to the console, right before the crash:

Assertion failed: (skylight_handle != NULL), function __get_skylight_handle_block_invoke, file /Library/Caches/com.apple.xbs/Sources/Quartz2D/Quartz2D-1070/CoreGraphics/Services/Connection/SLShim.c, line 26.
2016-08-11 04:23:55.484976 UTC - [GMPThread]: D/GMP GMPDecryptorParent[12325e860]::ActorDestroy(reason=4)

The ActorDestroy is from the GMP NSPR_LOG, and is the GMP code detecting the abnormal shutdown (i.e. crash).

The same CDM version is working in Chrome on Sierra.

Is this a sandbox difference?

Comment 9

[Haik Aftandilian [:haik]]

Yes, it's related to the plugin sandbox. On Sierra, plugin-container is trying to read from /System/Library/PrivateFrameworks/SkyLight.framework/SkyLight. There shouldn't be any harm in allow read access to OS X system executable code there. I did some quick tests with that modification and Widevine worked on a few Amazon videos I tested.

Comment 10

[Haik Aftandilian [:haik]]

Attachment: Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

This 1-line patch allows /System/Library/PrivateFrameworks/* to be read from the plugin sandbox. Would like to investigate a bit more before asking for review. The change applies to all plugin types, not just Widevine, and it's specific to Sierra because it's safe and other plugin types may end up needing it on Sierra.

Comment 11

[Chris Pearce [:cpearce (Not reading bugmail)]]

We should uplift the fix here, as 10.12 is reportedly being released in northern hemisphere autumn. i.e. soon.

Comment 12

[Haik Aftandilian [:haik]]

Attachment: Crash report whch includes the stack trace of thread causing the crash

Added a crash report from Sierra showing the stack trace. This just adds some more context and confirms the read (a stat call in this case) is coming from macOS CoreGraphics.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/10e99226d837
[10.12] Widevine CDM always crashes on Amazon since upgrade to macOS Sierra. r=gcp

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/10e99226d837

Comment 15

[Haik Aftandilian [:haik]]

Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Approval Request Comment
[Feature/regressing bug #]:
This is a new bug reproducible only on betas of macOS Sierra, expected to release in the coming months.

[User impact if declined]:
macOS Sierra users will not be able to stream Widevine videos from sites light Amazon.com. The browser will report that the plugin process has crashed.

[Describe test coverage new/current, TreeHerder]:
Manual tests of viewing Widevine videos using Nightly builds that have the fix.

[Risks and why]: 
This is a low risk change because it just involves a small change to OS X plugin sandbox. The change loosens the plugin sandbox in a minor way, adding permission for the plugin process to read from /System/Library/PrivateFrameworks which contains Apple-private library code.

[String/UUID change made/needed]:
N/A

Comment 16

[Haik Aftandilian [:haik]]

Attachment: (Firefox 48 patch) Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Approval Request Comment (Same as Aurora/Beta patch justification.)

[Feature/regressing bug #]:
This is a new bug reproducible only on betas of macOS Sierra, expected to release in the coming months.

[User impact if declined]:
macOS Sierra users will not be able to stream Widevine videos from sites light Amazon.com. The browser will report that the plugin process has crashed.

[Describe test coverage new/current, TreeHerder]:
Manual tests of viewing Widevine videos using Nightly builds that have the fix.

[Risks and why]: 
This is a low risk change because it just involves a small change to OS X plugin sandbox. The change loosens the plugin sandbox in a minor way, adding permission for the plugin process to read from /System/Library/PrivateFrameworks which contains Apple-private library code.

[String/UUID change made/needed]:
N/A

Comment 17

[Haik Aftandilian [:haik]]

If macOS Sierra is going to release before Firefox 49, we will want this fix in Firefox 48 so that Firefox users on Sierra have working Widevine playback.

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Florin, this is something we should add to our sign off testing for MacOS Sierra. Thanks!

Comment 19

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

This is a core scenario, uplifting to Aurora sooner so this gets stabilized faster, Aurora50+

Comment 20

[Chris Pearce [:cpearce (Not reading bugmail)]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/947cb99af8fe

Comment 21

[Andrei Vaida [:avaida]]

We'll make sure this is included in our test coverage for macOS 10.11.

Comment 22

[Sylvestre Ledru [:Sylvestre]]

It is too late to be part of 48.0.1.
However, with e10s being pushed to much more users today, I am expecting that we will do a 48.0.2 release to address some potential regressions, we could take this patch as ride along.

Comment 23

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Taking this for the beta 6 build next Monday.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1ffc259ce08b

Comment 25

[Alexandru Simonca [:asimonca], Release Desktop QA]

Hello, 
I have verified that the fix was applied to the following builds:
  - Firefox Beta 49.0b6 (id: 20160822111414)
  - Firefox DeveloperEdition 50.0a2 (id: 20160822004020)
  - Firefox Nightly 51.0a1 (id: 20160822030424).
There are no longer crashes of the WIdevine CDM plugin on any of these builds. I am setting the flags accordingly and the status to VERIFIED FIXED.

Comment 26

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8782164 [details] [diff] [review]
(Firefox 48 patch) Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

We are going to release 49 next week and sierra has not been released. So, no need to do a 48 dot release for this

Comment 27

[Markus Stange [:mstange]]

The release date for sierra has been announced: It's September 20th, the same day that we want to release 49 according to the revised schedule.

Comment 28

[Craig]

I have also posted here: https://support.mozilla.org/en-US/questions/1153608

I seem to be having an extremely similar issue. Single upgrading to 50.1 Widevine always crashes on Amazon and Netflix. I just upgraded to 51 hoping it would fix the issue and I'm still getting constant crashes. I'm not sure what else to try. It is only Widevine that is crashing, other browsing behaves as normal.

Comment 29

[Haik Aftandilian [:haik]]

(In reply to Craig from comment #28)
> I have also posted here: https://support.mozilla.org/en-US/questions/1153608
> 
> I seem to be having an extremely similar issue. Single upgrading to 50.1
> Widevine always crashes on Amazon and Netflix. I just upgraded to 51 hoping
> it would fix the issue and I'm still getting constant crashes. I'm not sure
> what else to try. It is only Widevine that is crashing, other browsing
> behaves as normal.

Hi, thanks for reporting this.

1) Do you have an external monitor attached? If so, could you try without that? There's another Widevine failure (bug 1309891) that triggers with an external monitor and I'm wondering if this is related.

2) Otherwise, a good first step for debugging would be to check in the OS X Console app for any messages that pop up when you're trying to view Widevine content. Have Console open before you try to view the content that causes the crash (because I find the logging unreliable otherwise.)

That's Console from /Applications/Utilities.

If you filter on "plugin-container", you'll see log entries from the plugin process (and the web content process if multi-process Firefox (e10s) is enabled for you.) Sierra logs a lot of messages in Console so you could easily miss it.

Also try filtering on "sandbox" separately.

3) Could you attach the output from "about:support"?

4) Lastly, what do you see that tells you only the Widevine CDM crashed and not content?

Thanks.

Comment 30

[Craig]

Hi Haik,
Thank you for your response. The answers to your questions are as follows:

1) No external monitor is attached. This is just direct from my MacBook Pro

2) This is a Console output surrounding a forced crash (attempting to play a Netflix title)
default	10:37:18.414567 +0000	plugin-container	subsystem: com.apple.coreaudio, category: ac, enable_level: 0, persist_level: 0, default_ttl: 0, info_ttl: 0, debug_ttl: 0, generate_symptoms: 0, enable_oversize: 0, privacy_setting: 2, enable_private_data: 0
default	10:37:18.415525 +0000	plugin-container	subsystem: com.apple.coreaudio, category: drc, enable_level: 0, persist_level: 0, default_ttl: 0, info_ttl: 0, debug_ttl: 0, generate_symptoms: 0, enable_oversize: 0, privacy_setting: 2, enable_private_data: 0
default	10:37:18.421542 +0000	plugin-container	ACMP4AACBaseDecoder.cpp:352:Initialize: (0x7fb5e4232640) Input format:  2 ch,  48000 Hz, 'aach' (0x00000000) 0 bits/channel, 0 bytes/packet, 2048 frames/packet, 0 bytes/frame
default	10:37:18.421717 +0000	plugin-container	ACMP4AACBaseDecoder.cpp:355:Initialize: (0x7fb5e4232640) Output format:  2 ch,  48000 Hz, Float32, inter
default	10:37:19.601487 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:21.466308 +0000	kernel	IO80211Interface::updateReport _peerManager is missing

default	10:37:23.017438 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:23.018525 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 20, EGID: 20
default	10:37:23.020181 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:23.045740 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:30.623253 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:30.679981 +0000	kernel	AMFI: allowing exception handler for 'plugin-container' (17045) because it is handling itself.
default	10:37:32.094788 +0000	CommCenter	#watchdog #I Callback Watchdog: checkin 7434
default	10:37:32.095067 +0000	CommCenter	#watchdog #I Server Watchdog: checkin 7434
default	10:37:33.069148 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:33.069971 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 20, EGID: 20
default	10:37:33.071153 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:33.093426 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20

3) about:support output
{
  "application": {
    "name": "Firefox",
    "osVersion": "Darwin 16.4.0",
    "version": "51.0.1",
    "buildID": "20170125094131",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0",
    "safeMode": false,
    "updateChannel": "release",
    "supportURL": "https://support.mozilla.org/1/firefox/51.0.1/Darwin/en-US/",
    "numTotalWindows": 1,
    "numRemoteWindows": 1,
    "remoteAutoStart": true,
    "autoStartStatus": 1
  },
  "modifiedPreferences": {
    "accessibility.typeaheadfind.flashBar": 0,
    "browser.cache.frecency_experiment": 1,
    "browser.cache.disk.smart_size.first_run": false,
    "browser.cache.disk.smart_size.use_old_max": false,
    "browser.cache.disk.capacity": 358400,
    "browser.cache.disk.filesystem_reported": 1,
    "browser.download.importedFromSqlite": true,
    "browser.places.smartBookmarksVersion": 8,
    "browser.sessionstore.upgradeBackup.latestBuildID": "20170125094131",
    "browser.startup.homepage_override.buildID": "20170125094131",
    "browser.startup.homepage_override.mstone": "51.0.1",
    "browser.tabs.remote.autostart.2": true,
    "browser.tabs.drawInTitlebar": false,
    "browser.urlbar.lastSuggestionsPromptDate": 20170114,
    "browser.urlbar.daysBeforeHidingSuggestionsPrompt": 0,
    "dom.push.userAgentID": "30f5ef1d8ecf4fc586ba6059a8f725f3",
    "extensions.lastAppVersion": "51.0.1",
    "font.internaluseonly.changed": true,
    "media.gmp-manager.buildID": "20170125094131",
    "media.gmp.storage.version.observed": 1,
    "media.benchmark.vp9.versioncheck": 1,
    "media.gmp-gmpopenh264.lastUpdate": 1484137798,
    "media.gmp-widevinecdm.lastUpdate": 1484137799,
    "media.gmp-gmpopenh264.version": "1.6",
    "media.benchmark.vp9.fps": 210,
    "media.gmp-gmpopenh264.abi": "x86_64-gcc3-u-i386-x86_64",
    "media.gmp-manager.lastCheck": 1485599849,
    "media.gmp-widevinecdm.version": "1.4.8.903",
    "media.gmp-widevinecdm.abi": "x86_64-gcc3-u-i386-x86_64",
    "network.cookie.prefsMigrated": true,
    "network.predictor.cleaned-up": true,
    "places.database.lastMaintenance": 1485507323,
    "places.history.expiration.transient_current_max_pages": 106842,
    "plugin.importedState": true,
    "plugin.disable_full_page_plugin_for_types": "application/pdf",
    "services.sync.declinedEngines": "forms,history,passwords,prefs,addons,adblockplus",
    "services.sync.lastPing": 1485515774,
    "services.sync.lastSync": "Sat Jan 28 2017 10:38:16 GMT+0000 (GMT)",
    "services.sync.numClients": 2,
    "services.sync.engine.prefs": false,
    "services.sync.engine.addons": false,
    "services.sync.engine.passwords": false,
    "services.sync.engine.prefs.modified": false,
    "services.sync.engine.history": false,
    "storage.vacuum.last.places.sqlite": 1484167903,
    "storage.vacuum.last.index": 1
  },
  "lockedPreferences": {},
  "javaScript": {
    "incrementalGCEnabled": true
  },
  "accessibility": {
    "isActive": false,
    "forceDisabled": 0
  },
  "libraryVersions": {
    "NSPR": {
      "minVersion": "4.13.1",
      "version": "4.13.1"
    },
    "NSS": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSUTIL": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSSSL": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSSMIME": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    }
  },
  "userJS": {
    "exists": false
  },
  "crashes": {
    "submitted": [
      {
        "id": "bp-430f9387-e690-44e2-84fb-495d82170125",
        "date": 1485343168000,
        "pending": false
      }
    ],
    "pending": 4
  },
  "sandbox": {},
  "extensions": [
    {
      "name": "Adblock Plus",
      "version": "2.8.2",
      "isActive": true,
      "id": "{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}"
    },
    {
      "name": "Application Update Service Helper",
      "version": "1.0",
      "isActive": true,
      "id": "[e-mail]"
    },
    {
      "name": "Multi-process staged rollout",
      "version": "1.7",
      "isActive": true,
      "id": "[e-mail]"
    },
    {
      "name": "Pocket",
      "version": "1.0.5",
      "isActive": true,
      "id": "[e-mail]"
    },
    {
      "name": "Reddit Enhancement Suite",
      "version": "5.2.2",
      "isActive": true,
      "id": "jid1-xUfzOsOFlzSOXg@jetpack"
    },
    {
      "name": "Web Compat",
      "version": "1.0",
      "isActive": true,
      "id": "[e-mail]"
    },
    {
      "name": "Adobe Acrobat - Create PDF",
      "version": "1.2",
      "isActive": false,
      "id": "[e-mail]"
    }
  ],
  "experiments": [],
  "graphics": {
    "numTotalWindows": 1,
    "numAcceleratedWindows": 1,
    "windowLayerManagerType": "OpenGL",
    "windowLayerManagerRemote": true,
    "supportsHardwareH264": "Yes",
    "currentAudioBackend": "audiounit",
    "adapterDescription": "",
    "adapterVendorID": "0x10de",
    "adapterDeviceID": "0x0fe9",
    "adapterRAM": "",
    "adapterDrivers": "",
    "driverVersion": "",
    "driverDate": "",
    "webglRenderer": "NVIDIA Corporation -- NVIDIA GeForce GT 750M OpenGL Engine",
    "webgl2Renderer": "NVIDIA Corporation -- NVIDIA GeForce GT 750M OpenGL Engine",
    "info": {
      "AzureCanvasBackend": "skia",
      "AzureCanvasAccelerated": 1,
      "AzureFallbackCanvasBackend": "none",
      "AzureContentBackend": "skia",
      "ApzWheelInput": 1,
      "TileHeight": 1024,
      "TileWidth": 1024
    },
    "featureLog": {
      "features": [
        {
          "name": "HW_COMPOSITING",
          "description": "Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "OPENGL_COMPOSITING",
          "description": "OpenGL Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        }
      ],
      "fallbacks": []
    },
    "crashGuards": []
  }
}

4) http://i.imgur.com/ZT2Mac2.png

Comment 31

[Haik Aftandilian [:haik]]

Moved discussion of Craig's issue over to a new bug: 1335282.