Closed Bug 1306314 Opened 8 years ago Closed 8 years ago

Persistent CDM storage broken for Widevine

Categories

(Core :: Audio/Video: Playback, defect, P1)

All
Other
defect

Tracking

()

VERIFIED FIXED
mozilla53
Tracking Status
firefox50 --- wontfix
firefox51 --- verified
firefox52 --- fixed
firefox53 --- verified

People

(Reporter: kieron.allsop, Assigned: cpearce)

References

(Blocks 1 open bug)

Details

(Whiteboard: [specification][type:bug])

Attachments

(5 files)

What did you do?
================
We are attempting to use setServerCertificate in section 5.1 Methods of the current EME W3C Candidate Recommendations by making use of the Queue a "message" Event algorithm (section 6.6.1) to obfuscate the device private data, in this case the private data is in the form of the Widevine Provider Client Token (PCT) which is generated server side.

1. Load this page
   https://urm.latens.com/hislop2
   (modified Shaka v.2 demo player which for a custom asset has had the player configuration altered to set persistentStateRequired = true)

2. Select custom asset from the drop down menu (last entry).

3. Apply these settings
	Username:               will
        Account:                will
        Portal UUID:            xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        Custom manifest:        https://urm.latens.com/content/dashoutput/out.mpd
        Custom license server:  https://urm.latens.com/WvLicenseProxy

        Note: a list of provisioned Portal UUID is supplied below. 
	
4. Select Load.

5. Pause playback.

6. Select Load.

What happened?
==============
On second and subsequent loads there is no server certificate request/response happening and therefore the CDM is unable to pass the persisted application data to the Widevine proxy.

What should have happened?
==========================
After the initial playback, all subsequent playback attempts should result in the CDM making two requests to the proxy for each key it needs

1.)A request for the server certificate (public key) that is used to obfuscate the device private data (normally this is much shorter in length than the regular key request). Base64 encoded this equals CAQ=

2.)The key request itself with additional data.
On second and subsequent loads there should be a server certificate request which would enable playback. 

This is the behaviour in Chrome browser using the same CDM version.

Is there anything else we should know?
======================================
Related to Bug 1278198 - [Widevine] Use of EME defined 'persistentState' attribute not supported by the CDM on Firefox beta47

Provisioned Portal UUIDs (only one can be used per pysical device)

a99ed906-8646-11e6-ae22-56b6b6499611
a99edc80-8646-11e6-ae22-56b6b6499611
a99ee216-8646-11e6-ae22-56b6b6499611
a99ee3e2-8646-11e6-ae22-56b6b6499611
a99ee590-8646-11e6-ae22-56b6b6499611
a99ee720-8646-11e6-ae22-56b6b6499611
a99ee8e2-8646-11e6-ae22-56b6b6499611
a99eea72-8646-11e6-ae22-56b6b6499611
a99eefcc-8646-11e6-ae22-56b6b6499611
a99ef170-8646-11e6-ae22-56b6b6499611
a99ef328-8646-11e6-ae22-56b6b6499611
a99ef54e-8646-11e6-ae22-56b6b6499611
a99ef6de-8646-11e6-ae22-56b6b6499611
a99ef864-8646-11e6-ae22-56b6b6499611
a99efa12-8646-11e6-ae22-56b6b6499611
a99eff9e-8646-11e6-ae22-56b6b6499611
a99f0142-8646-11e6-ae22-56b6b6499611
a99f02c8-8646-11e6-ae22-56b6b6499611
a99f0444-8646-11e6-ae22-56b6b6499611
a99f05e8-8646-11e6-ae22-56b6b6499611
Component: BrowserCompat → Untriaged
Product: Mozilla Developer Network → Firefox
Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Hi,

I also have this issue - has this been confirmed as a bug yet ?

Karl.
Added Joey from the Google Widevine team
I need to clarify one point.  In your bug report, you say you are using setServerCertificate, but you also say you are expecting a server certificate request.  These two are mutually exclusive.  If you call setServerCertificate, there is no need to request one from the server.

Are you actually using setServerCertificate?  Or are you only setting persistentStateRequired = true?
Hi,

Its the latter, sorry for the confusion.  The issue is that the (expected) automatically generated server certificate request event does not occur when persistentStateRequired = true
Does this work as you expect in Chrome?  If you set persistentStateRequired = true in Chrome, do you get a server certificate request?  What happens in Chrome if persistentStateRequired = false?
With persistentStateRequired = false the behaviour on both Chrome and Firefox is that our Proxy will reject the license request due to the fact that we cannot track/manage the device, this is custom logic.

On Chrome we see the expected behaviour - in that once we create and pass a provider_client_token to the CDM it will always send a server certificate request before each subsequent license request so that it can embed the token securely.

On Firefox, whilst we can now create and pass back a provider_client_token (in beta version) - we never get a server certificate request prior to any further license requests and as such we don't get the token we created passed back to us in said license request.  We are assuming that it is related to the fact the the CDM can not encrypt the token. The behaviour this causes is that we create a new provider_client_token (for the same device), which results in undefined behaviour in our device management subsystems.
Looks like a difference between the Firefox & Chrome integrations of the CDM, but I am not sure what the problem is.  CC'ing Chris.  Chris, please let me know if I can do anything to help.
Is this a regression in Firefox 52 Nightly? Do you see this bug in Firefox Release?

We're pushing to get the Web Platform Tests passing at the moment. We'll look at this once that's under control.
Blocks: EME
Priority: -- → P3
Chris,

I don't think this is a regression  - the issue was masked by https://bugzilla.mozilla.org/show_bug.cgi?id=1278198  since support for persistentState is a pre-requisite here.
There is not going to be a quick fix here.

The problem here is that Firefox has an assumption baked in that Widevine doesn't use persistent storage. We also can't handle running more than one CDM in the same process. So to solve for both of these constraints, we assign each CDM instance its own unique nodeId and storage container:

http://searchfox.org/mozilla-central/rev/f5c9e9a249637c9abd88754c8963ecb3838475cb/dom/media/gmp/GMPServiceParent.cpp#1379

This means we'll create a new CDM with a unique storage location every time we create a CDM. i.e. persistent state for Widevine actually is persistented, but not able to be read-back at a later time.

We can't handle running multiple instances of the Widevine CDM same-origin concurrently because the Gecko Media Plugins API, which we adapt the Widevine CDM's implementation of the Chromium CDM API to, has separate interfaces for decoding and decryption, whereas the Chromium interface has one interface that does both.

We can't easily associate the decoding GMP interface to the corresponding decryption interface with the one Chromium CDM interface, so we just disallow running more than one CDM per process:

http://searchfox.org/mozilla-central/rev/f5c9e9a249637c9abd88754c8963ecb3838475cb/dom/media/gmp/widevine-adapter/WidevineAdapter.cpp#97

So the fix is to re-architect our plugin harness so that it natively supports the Chromium CDM interface, rather than Adapting the CDM to our GMP APIs. I had wanted to do that anyway so that we could remove some excess copies of data. Here's another reason why we need to do that sooner rather than later.
Thanks for the detailed update Chris.

We are working with an operator who wishes to launch their service on Firefox. The content owners insist on device controls/management as part of the DRM system, which is currently blocking for us...

Can you perhaps provide a rough estimate of when the work required might be completed 

Thanks,
Karl.
I'd hope to get this fixed in Firefox 53, which ships on approximately 2017-04-18. 
https://wiki.mozilla.org/RapidRelease/Calendar
Thanks for the further update Chris.  We appreciate your help on this.
Hi Chris,
   Many thanks for your feedback and work on this issue to date. Given your last update around estimated timings(hopefully) for a fix for this problem, is it possible or likely that NPAPI support for Silverlight would remain past the March 2017 date that Benjamin Smedberg states here https://blog.mozilla.org/futurereleases/2016/07/20/reducing-adobe-flash-usage-in-firefox/ ?

Thanks
Davy
Flags: needinfo?(benjamin)
We plan to disable NPAPI support in Firefox 52 (bug 1269807), so it would be good to have this Widevine bug fixed in 52 or earlier so streamers can switch from Silverlight to Widevine. cpearce suggests fixing in 51 so streamers have time to test Widevine before 52 ships.

Changing priority from P3 to P1 because we need to fix this right away if we plan to uplift to Beta 51 (which starts next week).
Blocks: npapi-eol
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: P3 → P1
Summary: Server Certificate for Widevine is not supported Firefox beta52 → Implement setServerCertificate persistent storage for Widevine
Will try to split out a minimal fix that can be uplift, rather than tying fixing this to a large rearchitecturing of the CDM harness.
Flags: needinfo?(benjamin)
Summary: Implement setServerCertificate persistent storage for Widevine → Persistent CDM storage broken for Widevine
Many thanks Chris and Chris - appreciate the effort in prioritizing this. We'll look to retest once this is available in a beta/nightly build and give feedback from our point of view.

Thanks 
Davy
Assignee: nobody → cpearce
Comment on attachment 8810244 [details]
Bug 1306314 - Add an ID to GMPDecryptor instances, reflect that on CDMProxy.

https://reviewboard.mozilla.org/r/92586/#review92614

r+ with typos fixed, and a question:

::: dom/media/eme/CDMProxy.h:1
(Diff revision 1)
>  /* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

In commit description:
- 'who' -> 'whom'
- 'actor id' -> 'actor ID' (for consistency, unless lowercase 'id' is intended in this case)

::: dom/media/gmp/GMPDecryptorChild.cpp:76
(Diff revision 1)
> +  // The ID of this decryptor is the IPDL actor ID. Note it's unique inside
> +  // the child process, but not necessaily across all gecko processes.

"not [unique] across processes" -- interesting information, but what are we doing about it? Are potential clashes handled elsewhere, or just deemed rare enough that it's not worth bothering?

::: dom/media/gmp/GMPDecryptorChild.cpp:77
(Diff revision 1)
>  GMPDecryptorChild::Init(GMPDecryptor* aSession)
>  {
>    MOZ_ASSERT(aSession);
>    mSession = aSession;
> +  // The ID of this decryptor is the IPDL actor ID. Note it's unique inside
> +  // the child process, but not necessaily across all gecko processes.

'necessaily' -> 'necessarily'
Attachment #8810244 - Flags: review?(gsquelart) → review+
(In reply to Gerald Squelart [:gerald] from comment #23)
> Comment on attachment 8810244 [details]
> Bug 1306314 - Add an ID to GMPDecryptor instances, reflect that on CDMProxy.
> ::: dom/media/gmp/GMPDecryptorChild.cpp:76
> (Diff revision 1)
> > +  // The ID of this decryptor is the IPDL actor ID. Note it's unique inside
> > +  // the child process, but not necessaily across all gecko processes.
> 
> "not [unique] across processes" -- interesting information, but what are we
> doing about it? Are potential clashes handled elsewhere, or just deemed rare
> enough that it's not worth bothering?

We're not going to get conflicts because we segregate the actors via nodeid, and nodeids are separated by origin. So we shouldn't get ids from different processes being used together, as we segregate the nodeids already.
Comment on attachment 8810245 [details]
Bug 1306314 - Pass decryptor ID to GMPVideoDecoder constructor.

https://reviewboard.mozilla.org/r/92588/#review92616
Attachment #8810245 - Flags: review?(gsquelart) → review+
Comment on attachment 8810246 [details]
Bug 1306314 - Pipe decryptor ID through to WidevineAdapter.

https://reviewboard.mozilla.org/r/92590/#review92618
Attachment #8810246 - Flags: review?(gsquelart) → review+
Comment on attachment 8810247 [details]
Bug 1306314 - Use decryptor ID in WidevineAdapter to link decryptors with decoders.

https://reviewboard.mozilla.org/r/92592/#review92620
Attachment #8810247 - Flags: review?(gsquelart) → review+
Comment on attachment 8810248 [details]
Bug 1306314 - Allow Widevine CDM process to contain multiple CDM instances and to have persistent storage.

https://reviewboard.mozilla.org/r/92594/#review92622
Attachment #8810248 - Flags: review?(gsquelart) → review+
Comment on attachment 8810244 [details]
Bug 1306314 - Add an ID to GMPDecryptor instances, reflect that on CDMProxy.

https://reviewboard.mozilla.org/r/92586/#review92626

::: dom/media/gmp/GMPDecryptorChild.cpp:76
(Diff revision 1)
> +  // The ID of this decryptor is the IPDL actor ID. Note it's unique inside
> +  // the child process, but not necessaily across all gecko processes.

I'll add a comment explaining why this is OK.
Pushed by cpearce@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9e9727bdc484
Add an ID to GMPDecryptor instances, reflect that on CDMProxy. r=gerald
https://hg.mozilla.org/integration/autoland/rev/9343cb73c218
Pass decryptor ID to GMPVideoDecoder constructor. r=gerald
https://hg.mozilla.org/integration/autoland/rev/e2a5702d96b0
Pipe decryptor ID through to WidevineAdapter. r=gerald
https://hg.mozilla.org/integration/autoland/rev/d37e28309560
Use decryptor ID in WidevineAdapter to link decryptors with decoders. r=gerald
https://hg.mozilla.org/integration/autoland/rev/5c5600e1f30f
Allow Widevine CDM process to contain multiple CDM instances and to have persistent storage. r=gerald
Sorry had to back out for bustage like https://treeherder.mozilla.org/logviewer.html#?job_id=6561209&repo=autoland#L15076
Flags: needinfo?(cpearce)
Pushed by cpearce@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b0c612e8723
Add an ID to GMPDecryptor instances, reflect that on CDMProxy. r=gerald
https://hg.mozilla.org/integration/autoland/rev/893b62158c45
Pass decryptor ID to GMPVideoDecoder constructor. r=gerald
https://hg.mozilla.org/integration/autoland/rev/e183a3804fbb
Pipe decryptor ID through to WidevineAdapter. r=gerald
https://hg.mozilla.org/integration/autoland/rev/6a563a098351
Use decryptor ID in WidevineAdapter to link decryptors with decoders. r=gerald
https://hg.mozilla.org/integration/autoland/rev/b856b7987192
Allow Widevine CDM process to contain multiple CDM instances and to have persistent storage. r=gerald
Chris

Can you give me a link to the fixed version of Nightly so that I may test against our systems.

Thanks, Kieron
(In reply to Kieron Allsop from comment #46)
> Chris
> 
> Can you give me a link to the fixed version of Nightly so that I may test
> against our systems.
> 
> Thanks, Kieron

Daily Nightly is here: https://nightly.mozilla.org/
Kieron, Davy: Please can you download the latest Nightly from https://nightly.mozilla.org/ and test that? If you can confirm that this works, I'll uplift fixes so that they get ahead and into a version of Firefox which will ship sooner (51 hopefully).
Flags: needinfo?(cpearce)
Chris, I can confirm that the latest Nightly works as expected in Windows 7 Professional SP1 32-bit, Windows 10 Professional 64-bit, and OSX 10.11.
Comment on attachment 8810244 [details]
Bug 1306314 - Add an ID to GMPDecryptor instances, reflect that on CDMProxy.

Approval Request Comment
[Feature/regressing bug #]: Widevine persistent state support
[User impact if declined]: Some sites that require "persistent state" for EME sessions will think they can use persistent state but actually fail to be able to use it, and thus fail to playback. Without this, some sites won't be able to find a non-Silverlight DRM playback solution which satisfies their streaming contracts' robustness requirements before we remove Silverlight support in 52. So this patch makes persistent state actually work for Widevine EME.
[Describe test coverage new/current, TreeHerder]: We have partial coverage here. It's hard to get complete coverage, as it would require having a licensed Widevine server.
[Risks and why]: Low; I've tried hard to minimize the changes required to make this work.
[String/UUID change made/needed]: None
Attachment #8810244 - Flags: approval-mozilla-beta?
Attachment #8810244 - Flags: approval-mozilla-aurora?
Comment on attachment 8810245 [details]
Bug 1306314 - Pass decryptor ID to GMPVideoDecoder constructor.

Approval Request Comment
[Feature/regressing bug #]: Widevine persistent state support
[User impact if declined]: Some sites that require "persistent state" for EME sessions will think they can use persistent state but actually fail to be able to use it, and thus fail to playback. Without this, some sites won't be able to find a non-Silverlight DRM playback solution which satisfies their streaming contracts' robustness requirements before we remove Silverlight support in 52. So this patch makes persistent state actually work for Widevine EME.
[Describe test coverage new/current, TreeHerder]: We have partial coverage here. It's hard to get complete coverage, as it would require having a licensed Widevine server.
[Risks and why]: Low; I've tried hard to minimize the changes required to make this work.
[String/UUID change made/needed]: None
Attachment #8810245 - Flags: approval-mozilla-beta?
Attachment #8810245 - Flags: approval-mozilla-aurora?
Comment on attachment 8810246 [details]
Bug 1306314 - Pipe decryptor ID through to WidevineAdapter.

Approval Request Comment
[Feature/regressing bug #]: Widevine persistent state support
[User impact if declined]: Some sites that require "persistent state" for EME sessions will think they can use persistent state but actually fail to be able to use it, and thus fail to playback. Without this, some sites won't be able to find a non-Silverlight DRM playback solution which satisfies their streaming contracts' robustness requirements before we remove Silverlight support in 52. So this patch makes persistent state actually work for Widevine EME.
[Describe test coverage new/current, TreeHerder]: We have partial coverage here. It's hard to get complete coverage, as it would require having a licensed Widevine server.
[Risks and why]: Low; I've tried hard to minimize the changes required to make this work.
[String/UUID change made/needed]: None
Attachment #8810246 - Flags: approval-mozilla-beta?
Attachment #8810246 - Flags: approval-mozilla-aurora?
Comment on attachment 8810247 [details]
Bug 1306314 - Use decryptor ID in WidevineAdapter to link decryptors with decoders.

Approval Request Comment
[Feature/regressing bug #]: Widevine persistent state support
[User impact if declined]: Some sites that require "persistent state" for EME sessions will think they can use persistent state but actually fail to be able to use it, and thus fail to playback. Without this, some sites won't be able to find a non-Silverlight DRM playback solution which satisfies their streaming contracts' robustness requirements before we remove Silverlight support in 52. So this patch makes persistent state actually work for Widevine EME.
[Describe test coverage new/current, TreeHerder]: We have partial coverage here. It's hard to get complete coverage, as it would require having a licensed Widevine server.
[Risks and why]: Low; I've tried hard to minimize the changes required to make this work.
[String/UUID change made/needed]: None
Attachment #8810247 - Flags: approval-mozilla-beta?
Attachment #8810247 - Flags: approval-mozilla-aurora?
Comment on attachment 8810248 [details]
Bug 1306314 - Allow Widevine CDM process to contain multiple CDM instances and to have persistent storage.

Approval Request Comment
[Feature/regressing bug #]: Widevine persistent state support
[User impact if declined]: Some sites that require "persistent state" for EME sessions will think they can use persistent state but actually fail to be able to use it, and thus fail to playback. Without this, some sites won't be able to find a non-Silverlight DRM playback solution which satisfies their streaming contracts' robustness requirements before we remove Silverlight support in 52. So this patch makes persistent state actually work for Widevine EME.
[Describe test coverage new/current, TreeHerder]: We have partial coverage here. It's hard to get complete coverage, as it would require having a licensed Widevine server.
[Risks and why]: Low; I've tried hard to minimize the changes required to make this work.
[String/UUID change made/needed]: None
Attachment #8810248 - Flags: approval-mozilla-beta?
Attachment #8810248 - Flags: approval-mozilla-aurora?
Comment on attachment 8810244 [details]
Bug 1306314 - Add an ID to GMPDecryptor instances, reflect that on CDMProxy.

Since we plan to disable NPAPI support in Firefox 52 and in order to give streamers more time to switch from Silverlight to Widevine, we can take these patches in 51 early beta and see how it goes. Beta51 and Aurora52+. Should be in 51 beta 2.
Attachment #8810244 - Flags: approval-mozilla-beta?
Attachment #8810244 - Flags: approval-mozilla-beta+
Attachment #8810244 - Flags: approval-mozilla-aurora?
Attachment #8810244 - Flags: approval-mozilla-aurora+
Attachment #8810245 - Flags: approval-mozilla-beta?
Attachment #8810245 - Flags: approval-mozilla-beta+
Attachment #8810245 - Flags: approval-mozilla-aurora?
Attachment #8810245 - Flags: approval-mozilla-aurora+
Attachment #8810246 - Flags: approval-mozilla-beta?
Attachment #8810246 - Flags: approval-mozilla-beta+
Attachment #8810246 - Flags: approval-mozilla-aurora?
Attachment #8810246 - Flags: approval-mozilla-aurora+
Attachment #8810247 - Flags: approval-mozilla-beta?
Attachment #8810247 - Flags: approval-mozilla-beta+
Attachment #8810247 - Flags: approval-mozilla-aurora?
Attachment #8810247 - Flags: approval-mozilla-aurora+
Attachment #8810248 - Flags: approval-mozilla-beta?
Attachment #8810248 - Flags: approval-mozilla-beta+
Attachment #8810248 - Flags: approval-mozilla-aurora?
Attachment #8810248 - Flags: approval-mozilla-aurora+
has problems to apply to beta

warning: conflicts while merging dom/media/gmp/GMPService.h! (edit, then use 'hg resolve --mark')
abort: unresolved conflicts, can't continue
(use 'hg resolve' and 'hg graft --continue')
Flags: needinfo?(cpearce)
Flags: needinfo?(cpearce)
Kieran Allsop, Karl Gallagher, this issue should be fixed in the next Firefox 51 beta build which is released Tuesday this week.

Firefox 51 ships to our release population on approximately 2017-01-24.

It would be great if you can download a Firefox 51 beta build on or after Wednesday and verify that persistent storage works as expected.

You can download Firefox beta here:
https://www.mozilla.org/en-US/firefox/channel/desktop/
Flags: qe-verify+
QA Contact: ciprian.georgiu
I will provide feedback on this on Wednesday Chris.

Thanks,
Karl.
Chris.

This issued is confirmed fixed in Firefox 51 beta.

Tested on 

OSX 10.10
Windows 7 32-bit
Windows 10 64-bit
Confirmed here in Firefox 51 beta on OSX 10.11.6.

We will be testing this across platforms and will feedback if needed.

Thanks for this as it gives us a chance to do a good amount of testing before NPAPI is dropped.
Karl, Chris can you please confirm that this issue is fixed across platforms, in Firefox 52 (latest Aurora) and 53 (latest Nightly) as well? Thanks in advance.
Based on comment 49 and comment 62, I think we're good here.
It looks like I missed comment 42, sorry...
In this case, I will update the status flag for 51 and 53 and close this issue as verified fixed.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: