[10.12] Widevine CDM always crashes on Amazon since upgrade to macOS Sierra

VERIFIED FIXED in Firefox 49

Status

()

P1
normal
Rank:
10
VERIFIED FIXED
2 years ago
2 years ago

People

(Reporter: cadeyrn, Assigned: haik)

Tracking

(Blocks: 2 bugs, {crash})

Trunk
mozilla51
crash
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(platform-rel ?, firefox48+ wontfix, firefox49 verified, firefox50 verified, firefox51 verified)

Details

(Whiteboard: [platform-rel-Amazon][platform-rel-AmazonVideo])

Attachments

(4 attachments)

(Reporter)

Description

2 years ago
Created attachment 8770435 [details]
screenshot

Every time I try to watch something on Amazon Prime the Widevine CDM crashes. It never crashed on OS X 10.11, it happens since the upgrade to macOS Sierra (public beta). It's reproducible on stable Firefox 47.0.1 and current Firefox Nightly 50.
(In reply to Sören Hentzschel from comment #0)
> Created attachment 8770435 [details]
> screenshot
> 
> Every time I try to watch something on Amazon Prime the Widevine CDM
> crashes. It never crashed on OS X 10.11, it happens since the upgrade to
> macOS Sierra (public beta). It's reproducible on stable Firefox 47.0.1 and
> current Firefox Nightly 50.

Sören: Can you please submit a crash report and then add it to the bug? Thanks!
Flags: needinfo?(cadeyrn)
Summary: Widevine CDM always crashes on Amazon since upgrade to macOS Sierra → [10.12] Widevine CDM always crashes on Amazon since upgrade to macOS Sierra
(Reporter)

Comment 2

2 years ago
Where can I find the crash report? There is no crash report on about:crashes after clicking the button "submit a crash report".
Flags: needinfo?(cadeyrn)
I don't see any Widevine crash reports submitted from macOS 10.12.x users:

https://is.gd/3rsROR
Blocks: 1222845
Keywords: crash
Priority: -- → P1
Chris -- Do you agree this is a P1?  If so, do you want to take it or find someone to take it?  Or ask Anthony to find someone?  Thanks.
Rank: 10
Flags: needinfo?(cpearce)
(In reply to Chris Peterson [:cpeterson] from comment #3)
> I don't see any Widevine crash reports submitted from macOS 10.12.x users:
> 
> https://is.gd/3rsROR

You need to search for "gmp plugin" "exists", rather than "product" "has terms" "firefox". i.e.:"

http://bit.ly/2aafdez
platform-rel: --- → ?
Whiteboard: [platform-rel-Amazon][platform-rel-AmazonVideo]

Comment 6

2 years ago
I can confirm that this is a bug. Having the same issue myself.
Haik: Can you reproduce this on MacOS Sierra Beta? Is it a sandboxing issue?
Flags: needinfo?(cpearce) → needinfo?(haftandilian)
Flags: needinfo?(cpearce)
Haik: I installed macOS Sierra on a spare mac mini we had lying around the office, and I if I spoof my useragent to Chrome's and try to play Netflix (so that Netflix serves us Widevine content) I see the CDM crash.

I see this logged to the console, right before the crash:

Assertion failed: (skylight_handle != NULL), function __get_skylight_handle_block_invoke, file /Library/Caches/com.apple.xbs/Sources/Quartz2D/Quartz2D-1070/CoreGraphics/Services/Connection/SLShim.c, line 26.
2016-08-11 04:23:55.484976 UTC - [GMPThread]: D/GMP GMPDecryptorParent[12325e860]::ActorDestroy(reason=4)

The ActorDestroy is from the GMP NSPR_LOG, and is the GMP code detecting the abnormal shutdown (i.e. crash).

The same CDM version is working in Chrome on Sierra.

Is this a sandbox difference?
(Assignee)

Comment 9

2 years ago
Yes, it's related to the plugin sandbox. On Sierra, plugin-container is trying to read from /System/Library/PrivateFrameworks/SkyLight.framework/SkyLight. There shouldn't be any harm in allow read access to OS X system executable code there. I did some quick tests with that modification and Widevine worked on a few Amazon videos I tested.
Flags: needinfo?(haftandilian)
(Assignee)

Comment 10

2 years ago
Created attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

This 1-line patch allows /System/Library/PrivateFrameworks/* to be read from the plugin sandbox. Would like to investigate a bit more before asking for review. The change applies to all plugin types, not just Widevine, and it's specific to Sierra because it's safe and other plugin types may end up needing it on Sierra.
Assignee: nobody → haftandilian
Status: NEW → ASSIGNED
We should uplift the fix here, as 10.12 is reportedly being released in northern hemisphere autumn. i.e. soon.
Flags: needinfo?(cpearce)
(Assignee)

Updated

2 years ago
Attachment #8779992 - Flags: review?(gpascutto)
(Assignee)

Comment 12

2 years ago
Created attachment 8780337 [details]
Crash report whch includes the stack trace of thread causing the crash

Added a crash report from Sierra showing the stack trace. This just adds some more context and confirms the read (a stat call in this case) is coming from macOS CoreGraphics.
Attachment #8779992 - Attachment is patch: true
Attachment #8779992 - Attachment mime type: text/x-patch → text/plain
Attachment #8779992 - Flags: review?(gpascutto) → review+
(Assignee)

Updated

2 years ago
Keywords: checkin-needed
(Assignee)

Updated

2 years ago
status-firefox48: --- → affected
status-firefox49: --- → affected
status-firefox51: --- → affected

Comment 13

2 years ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/10e99226d837
[10.12] Widevine CDM always crashes on Amazon since upgrade to macOS Sierra. r=gcp
Keywords: checkin-needed

Comment 14

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/10e99226d837
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox51: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
(Assignee)

Comment 15

2 years ago
Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Approval Request Comment
[Feature/regressing bug #]:
This is a new bug reproducible only on betas of macOS Sierra, expected to release in the coming months.

[User impact if declined]:
macOS Sierra users will not be able to stream Widevine videos from sites light Amazon.com. The browser will report that the plugin process has crashed.

[Describe test coverage new/current, TreeHerder]:
Manual tests of viewing Widevine videos using Nightly builds that have the fix.

[Risks and why]: 
This is a low risk change because it just involves a small change to OS X plugin sandbox. The change loosens the plugin sandbox in a minor way, adding permission for the plugin process to read from /System/Library/PrivateFrameworks which contains Apple-private library code.

[String/UUID change made/needed]:
N/A
Attachment #8779992 - Flags: approval-mozilla-release?
Attachment #8779992 - Flags: approval-mozilla-beta?
Attachment #8779992 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 16

2 years ago
Created attachment 8782164 [details] [diff] [review]
(Firefox 48 patch) Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Approval Request Comment (Same as Aurora/Beta patch justification.)

[Feature/regressing bug #]:
This is a new bug reproducible only on betas of macOS Sierra, expected to release in the coming months.

[User impact if declined]:
macOS Sierra users will not be able to stream Widevine videos from sites light Amazon.com. The browser will report that the plugin process has crashed.

[Describe test coverage new/current, TreeHerder]:
Manual tests of viewing Widevine videos using Nightly builds that have the fix.

[Risks and why]: 
This is a low risk change because it just involves a small change to OS X plugin sandbox. The change loosens the plugin sandbox in a minor way, adding permission for the plugin process to read from /System/Library/PrivateFrameworks which contains Apple-private library code.

[String/UUID change made/needed]:
N/A
Attachment #8782164 - Flags: approval-mozilla-release?
(Assignee)

Updated

2 years ago
Attachment #8779992 - Flags: approval-mozilla-release?
(Assignee)

Comment 17

2 years ago
If macOS Sierra is going to release before Firefox 49, we will want this fix in Firefox 48 so that Firefox users on Sierra have working Widevine playback.
Hi Florin, this is something we should add to our sign off testing for MacOS Sierra. Thanks!
Flags: qe-verify+
Flags: needinfo?(florin.mezei)
Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

This is a core scenario, uplifting to Aurora sooner so this gets stabilized faster, Aurora50+
Attachment #8779992 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+

Comment 20

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/947cb99af8fe
status-firefox50: affected → fixed
We'll make sure this is included in our test coverage for macOS 10.11.
Flags: needinfo?(florin.mezei) → needinfo?(andrei.vaida)
QA Contact: alexandru.simonca
It is too late to be part of 48.0.1.
However, with e10s being pushed to much more users today, I am expecting that we will do a 48.0.2 release to address some potential regressions, we could take this patch as ride along.
tracking-firefox48: --- → +
Comment on attachment 8779992 [details] [diff] [review]
Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

Taking this for the beta 6 build next Monday.
Attachment #8779992 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment 24

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/1ffc259ce08b
status-firefox49: affected → fixed
Hello, 
I have verified that the fix was applied to the following builds:
  - Firefox Beta 49.0b6 (id: 20160822111414)
  - Firefox DeveloperEdition 50.0a2 (id: 20160822004020)
  - Firefox Nightly 51.0a1 (id: 20160822030424).
There are no longer crashes of the WIdevine CDM plugin on any of these builds. I am setting the flags accordingly and the status to VERIFIED FIXED.
Status: RESOLVED → VERIFIED
status-firefox49: fixed → verified
status-firefox50: fixed → verified
status-firefox51: fixed → verified
Flags: qe-verify+
Flags: needinfo?(andrei.vaida)
Comment on attachment 8782164 [details] [diff] [review]
(Firefox 48 patch) Allows /System/Library/PrivateFrameworks/ to be read from the from the plugin sandbox

We are going to release 49 next week and sierra has not been released. So, no need to do a 48 dot release for this
Attachment #8782164 - Flags: approval-mozilla-release? → approval-mozilla-release-
status-firefox48: affected → wontfix
The release date for sierra has been announced: It's September 20th, the same day that we want to release 49 according to the revised schedule.

Comment 28

2 years ago
I have also posted here: https://support.mozilla.org/en-US/questions/1153608

I seem to be having an extremely similar issue. Single upgrading to 50.1 Widevine always crashes on Amazon and Netflix. I just upgraded to 51 hoping it would fix the issue and I'm still getting constant crashes. I'm not sure what else to try. It is only Widevine that is crashing, other browsing behaves as normal.
(Assignee)

Comment 29

2 years ago
(In reply to Craig from comment #28)
> I have also posted here: https://support.mozilla.org/en-US/questions/1153608
> 
> I seem to be having an extremely similar issue. Single upgrading to 50.1
> Widevine always crashes on Amazon and Netflix. I just upgraded to 51 hoping
> it would fix the issue and I'm still getting constant crashes. I'm not sure
> what else to try. It is only Widevine that is crashing, other browsing
> behaves as normal.

Hi, thanks for reporting this.

1) Do you have an external monitor attached? If so, could you try without that? There's another Widevine failure (bug 1309891) that triggers with an external monitor and I'm wondering if this is related.

2) Otherwise, a good first step for debugging would be to check in the OS X Console app for any messages that pop up when you're trying to view Widevine content. Have Console open before you try to view the content that causes the crash (because I find the logging unreliable otherwise.)

That's Console from /Applications/Utilities.

If you filter on "plugin-container", you'll see log entries from the plugin process (and the web content process if multi-process Firefox (e10s) is enabled for you.) Sierra logs a lot of messages in Console so you could easily miss it.

Also try filtering on "sandbox" separately.

3) Could you attach the output from "about:support"?

4) Lastly, what do you see that tells you only the Widevine CDM crashed and not content?

Thanks.
Flags: needinfo?(cmandrews85)

Comment 30

2 years ago
Hi Haik,
Thank you for your response. The answers to your questions are as follows:

1) No external monitor is attached. This is just direct from my MacBook Pro

2) This is a Console output surrounding a forced crash (attempting to play a Netflix title)
default	10:37:18.414567 +0000	plugin-container	subsystem: com.apple.coreaudio, category: ac, enable_level: 0, persist_level: 0, default_ttl: 0, info_ttl: 0, debug_ttl: 0, generate_symptoms: 0, enable_oversize: 0, privacy_setting: 2, enable_private_data: 0
default	10:37:18.415525 +0000	plugin-container	subsystem: com.apple.coreaudio, category: drc, enable_level: 0, persist_level: 0, default_ttl: 0, info_ttl: 0, debug_ttl: 0, generate_symptoms: 0, enable_oversize: 0, privacy_setting: 2, enable_private_data: 0
default	10:37:18.421542 +0000	plugin-container	ACMP4AACBaseDecoder.cpp:352:Initialize: (0x7fb5e4232640) Input format:  2 ch,  48000 Hz, 'aach' (0x00000000) 0 bits/channel, 0 bytes/packet, 2048 frames/packet, 0 bytes/frame
default	10:37:18.421717 +0000	plugin-container	ACMP4AACBaseDecoder.cpp:355:Initialize: (0x7fb5e4232640) Output format:  2 ch,  48000 Hz, Float32, inter
default	10:37:19.601487 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:21.466308 +0000	kernel	IO80211Interface::updateReport _peerManager is missing

default	10:37:23.017438 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:23.018525 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 20, EGID: 20
default	10:37:23.020181 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:23.045740 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:30.623253 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:30.679981 +0000	kernel	AMFI: allowing exception handler for 'plugin-container' (17045) because it is handling itself.
default	10:37:32.094788 +0000	CommCenter	#watchdog #I Callback Watchdog: checkin 7434
default	10:37:32.095067 +0000	CommCenter	#watchdog #I Server Watchdog: checkin 7434
default	10:37:33.069148 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 0, EGID: 0
default	10:37:33.069971 +0000	opendirectoryd	Client: <private>, UID: 0, EUID: 0, GID: 20, EGID: 20
default	10:37:33.071153 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20
default	10:37:33.093426 +0000	opendirectoryd	Client: <private>, UID: 501, EUID: 501, GID: 20, EGID: 20

3) about:support output
{
  "application": {
    "name": "Firefox",
    "osVersion": "Darwin 16.4.0",
    "version": "51.0.1",
    "buildID": "20170125094131",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0",
    "safeMode": false,
    "updateChannel": "release",
    "supportURL": "https://support.mozilla.org/1/firefox/51.0.1/Darwin/en-US/",
    "numTotalWindows": 1,
    "numRemoteWindows": 1,
    "remoteAutoStart": true,
    "autoStartStatus": 1
  },
  "modifiedPreferences": {
    "accessibility.typeaheadfind.flashBar": 0,
    "browser.cache.frecency_experiment": 1,
    "browser.cache.disk.smart_size.first_run": false,
    "browser.cache.disk.smart_size.use_old_max": false,
    "browser.cache.disk.capacity": 358400,
    "browser.cache.disk.filesystem_reported": 1,
    "browser.download.importedFromSqlite": true,
    "browser.places.smartBookmarksVersion": 8,
    "browser.sessionstore.upgradeBackup.latestBuildID": "20170125094131",
    "browser.startup.homepage_override.buildID": "20170125094131",
    "browser.startup.homepage_override.mstone": "51.0.1",
    "browser.tabs.remote.autostart.2": true,
    "browser.tabs.drawInTitlebar": false,
    "browser.urlbar.lastSuggestionsPromptDate": 20170114,
    "browser.urlbar.daysBeforeHidingSuggestionsPrompt": 0,
    "dom.push.userAgentID": "30f5ef1d8ecf4fc586ba6059a8f725f3",
    "extensions.lastAppVersion": "51.0.1",
    "font.internaluseonly.changed": true,
    "media.gmp-manager.buildID": "20170125094131",
    "media.gmp.storage.version.observed": 1,
    "media.benchmark.vp9.versioncheck": 1,
    "media.gmp-gmpopenh264.lastUpdate": 1484137798,
    "media.gmp-widevinecdm.lastUpdate": 1484137799,
    "media.gmp-gmpopenh264.version": "1.6",
    "media.benchmark.vp9.fps": 210,
    "media.gmp-gmpopenh264.abi": "x86_64-gcc3-u-i386-x86_64",
    "media.gmp-manager.lastCheck": 1485599849,
    "media.gmp-widevinecdm.version": "1.4.8.903",
    "media.gmp-widevinecdm.abi": "x86_64-gcc3-u-i386-x86_64",
    "network.cookie.prefsMigrated": true,
    "network.predictor.cleaned-up": true,
    "places.database.lastMaintenance": 1485507323,
    "places.history.expiration.transient_current_max_pages": 106842,
    "plugin.importedState": true,
    "plugin.disable_full_page_plugin_for_types": "application/pdf",
    "services.sync.declinedEngines": "forms,history,passwords,prefs,addons,adblockplus",
    "services.sync.lastPing": 1485515774,
    "services.sync.lastSync": "Sat Jan 28 2017 10:38:16 GMT+0000 (GMT)",
    "services.sync.numClients": 2,
    "services.sync.engine.prefs": false,
    "services.sync.engine.addons": false,
    "services.sync.engine.passwords": false,
    "services.sync.engine.prefs.modified": false,
    "services.sync.engine.history": false,
    "storage.vacuum.last.places.sqlite": 1484167903,
    "storage.vacuum.last.index": 1
  },
  "lockedPreferences": {},
  "javaScript": {
    "incrementalGCEnabled": true
  },
  "accessibility": {
    "isActive": false,
    "forceDisabled": 0
  },
  "libraryVersions": {
    "NSPR": {
      "minVersion": "4.13.1",
      "version": "4.13.1"
    },
    "NSS": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSUTIL": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSSSL": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    },
    "NSSSMIME": {
      "minVersion": "3.28.1",
      "version": "3.28.1"
    }
  },
  "userJS": {
    "exists": false
  },
  "crashes": {
    "submitted": [
      {
        "id": "bp-430f9387-e690-44e2-84fb-495d82170125",
        "date": 1485343168000,
        "pending": false
      }
    ],
    "pending": 4
  },
  "sandbox": {},
  "extensions": [
    {
      "name": "Adblock Plus",
      "version": "2.8.2",
      "isActive": true,
      "id": "{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}"
    },
    {
      "name": "Application Update Service Helper",
      "version": "1.0",
      "isActive": true,
      "id": "aushelper@mozilla.org"
    },
    {
      "name": "Multi-process staged rollout",
      "version": "1.7",
      "isActive": true,
      "id": "e10srollout@mozilla.org"
    },
    {
      "name": "Pocket",
      "version": "1.0.5",
      "isActive": true,
      "id": "firefox@getpocket.com"
    },
    {
      "name": "Reddit Enhancement Suite",
      "version": "5.2.2",
      "isActive": true,
      "id": "jid1-xUfzOsOFlzSOXg@jetpack"
    },
    {
      "name": "Web Compat",
      "version": "1.0",
      "isActive": true,
      "id": "webcompat@mozilla.org"
    },
    {
      "name": "Adobe Acrobat - Create PDF",
      "version": "1.2",
      "isActive": false,
      "id": "web2pdfextension@web2pdf.adobedotcom"
    }
  ],
  "experiments": [],
  "graphics": {
    "numTotalWindows": 1,
    "numAcceleratedWindows": 1,
    "windowLayerManagerType": "OpenGL",
    "windowLayerManagerRemote": true,
    "supportsHardwareH264": "Yes",
    "currentAudioBackend": "audiounit",
    "adapterDescription": "",
    "adapterVendorID": "0x10de",
    "adapterDeviceID": "0x0fe9",
    "adapterRAM": "",
    "adapterDrivers": "",
    "driverVersion": "",
    "driverDate": "",
    "webglRenderer": "NVIDIA Corporation -- NVIDIA GeForce GT 750M OpenGL Engine",
    "webgl2Renderer": "NVIDIA Corporation -- NVIDIA GeForce GT 750M OpenGL Engine",
    "info": {
      "AzureCanvasBackend": "skia",
      "AzureCanvasAccelerated": 1,
      "AzureFallbackCanvasBackend": "none",
      "AzureContentBackend": "skia",
      "ApzWheelInput": 1,
      "TileHeight": 1024,
      "TileWidth": 1024
    },
    "featureLog": {
      "features": [
        {
          "name": "HW_COMPOSITING",
          "description": "Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "OPENGL_COMPOSITING",
          "description": "OpenGL Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        }
      ],
      "fallbacks": []
    },
    "crashGuards": []
  }
}

4) http://i.imgur.com/ZT2Mac2.png
Flags: needinfo?(cmandrews85)
(Assignee)

Comment 31

2 years ago
Moved discussion of Craig's issue over to a new bug: 1335282.
You need to log in before you can comment on or make changes to this bug.