1300069 - Playback of encrypted audio track fails

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[Sander Saares]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36

Steps to reproduce:

1) Go to http://axinom-dash-v7.axtest.net/
2) Select either of the players (for comparison - repros with both)
3) From the dropdown, select "Multi-DRM" (Shaka Player) or "v7 encrypted single key (1080p)".
4) Press Load.
5) Observe playback failure.

We believe this video conforms to various relevant specifications (ISOBMFF, DASH, DASH-IF IOP, CENC, also draft of CMAF) in all important dimensions. More details about the videos (including required DRM configuration to reproduce with other players) can be found at https://github.com/Axinom/dash-test-vectors

By trying various permutations on the source data, I was able to achieve successful playback if:

1. A Widevine-specific PSSH box was added to the audio track initialization segment.
2. The audio codec was changed to AAC-LC.
3. The audio track was marked as not using subsample encryption.

These permutations also required changes in tooling versions, so it is possible that hidden variables remain undiscovered. What I can only say for sure is that the video in question above appears to be valid in all important dimensions, yet does not successfully play back.

The issue only appears to affect audio tracks.

Chrome and Edge are also affected to a lesser degree, with successful playback requiring fewer permutations to the data.

I realize that the presence of 3rd party players can make diagnostics difficult, as it is a layer between the browser and the actual data. Still, I know of no better repro case. I am happy to provide any additional data and to try to constrain the scenario using alternative playback mechanisms if you can suggest some ways to narrow down the problem.


Actual results:

No playback starts.

With Shaka Player: DRM.REQUESTED_KEY_SYSTEM_CONFIG_UNAVAILABLE ()
With dash.js player: no obvious error in console but playback also does not start.


Expected results:

Playback should have started, with both video and audio present.

Comment 1

[Sander Saares]

Some additional rationale and notes:

* The Widevine PSSH box is in all cases provided to EME, so it should not be required in the initialization segment (fed to MSE - an independent API).
* The content is HE-AAC v2 and the clear variant appears to play successfully. I was not able to get encrypted HE-AAC v2 to play in any permutation of the data.
* The audio track uses "no-op" subsample encryption, with all the data actually encrypted (signaling zero bytes of clear data). This usage is in conformance with CENC spec.

Comment 2

[Jean-Yves Avenard [:jya]]

http://www.w3.org/TR/2014/WD-encrypted-media-20140828/cenc-format.html#init-data

"Initialization Data is always one or more concatenated 'pssh' boxes. The CDM must be able to examine multiple 'pssh' boxes in the Initialization Data to find a 'pssh' box that it supports."

Pssh must be in the init segment.

Comment 3

[Jean-Yves Avenard [:jya]]

http://www.w3.org/TR/2014/WD-encrypted-media-20140828/cenc-format.html#init-data

"Initialization Data is always one or more concatenated 'pssh' boxes. The CDM must be able to examine multiple 'pssh' boxes in the Initialization Data to find a 'pssh' box that it supports."

Pssh must be in the init segment.

Comment 4

[Jean-Yves Avenard [:jya]]

What exactly is this bug about?

Is your bug about playback not starting when the init segment doesn't contain the pssh but is contained in each of the media segment?

If so this bug will be closed as invalid. As per spec, the "encrypted" event will only be fired if the init segment contains encryption info. For MSE to know it's encrypted, it must be told in the init segment.

You'll have the same issue with chrome.

For he-aac v2, what mimetype are you using? could you attach separately here the init segment you are using for each stream?

Comment 5

[Sander Saares]

The bug is about spec-conforming content not playing in Firefox. The reasons for it not playing are difficult for me to determine but in the issue description I provide the results of my investigations.

Let's turn to the topic of the Initialization Data. While it is true that an "encrypted" event could only be raised in response to a PSSH box found in the stream, it is not necessary for an "encrypted" event to be raised in order to begin encrypted content playback - the players linked above provide Initialization Data using the generateRequest() function defined by EME. The players will do this for the referenced content regardless of whether any "encrypted" event is raised!

Therefore, the browser is always supplied Initialization Data and the preconditions for encrypted content playback are met.

Note that I am not a player developer so my understanding of these players is limited and it is possible that my understanding of their behavior diverges from reality. However, I do not at present have any reason to suspect so.

It is unclear to me what role "https://www.w3.org/TR/2014/WD-encrypted-media-20140828/cenc-format.html#bib-CENC-1ST" plays in the context of EME. I do not see any references from EME to this spec. Can you reference something that states that content should conform to this in order to be played back, if that was the intent behind your referencing this document?

Regardless of whether this document has any relevance to EME, the sentence you quoted says that Initialization Data is within one or more PSSH boxes. This constraint is satisfied by the content referenced in this issue - It is simply that these PSSH boxes are not in the initialization segment (or, indeed, in any segment!). The player provides them separately of the media stream fed via MSE. However, I can find no specification that places such a requirement (for PSSH boxes to be in the media stream fed to the browser via MSE).

I can confirm that Chrome does not have any such issue with the PSSH box - though it also fails to play this content due to the subsample encryption signaling (removal of that signaling was the only permutation required to achieve successful playback in Chrome, as far as I could determine).

For HE-AAC v2, the MIME type passed to the player is "audio/mp4" and the codecs string is "mp4a.40.29". I assume the players pass this on to MSE as-is. I will attach the initialization segments for all 3 audio tracks present in the referenced video.

Comment 6

[Sander Saares]

Attachment: Init segment for audio track with id 15 (lang=en)

Comment 7

[Sander Saares]

Attachment: Init segment for audio track with id 16 (lang=en-AU)

Comment 8

[Sander Saares]

Attachment: Init segment for audio track with id 17 (lang=et-ET)

Comment 9

[Chris Pearce [:cpearce (Not reading bugmail)]]

I'll take a look.

Comment 10

[Chris Pearce [:cpearce (Not reading bugmail)]]

Firefox assumes that a stream is encrypted only if it sees a PSSH box in the init segment. Otherwise it doesn't know that it needs to setup a decryption pipeline.

There's probably another box in the MP4 stream we can rely on instead to detect encryption.

I don't see anything in the EME spec that says that there must be a PSSH box in the MSE init segment for CENC content.

Note the EME "initialization data" and the MSE concept of "init segment" are two separate things.

We need to change the MediaFormatReader and/or MP4Demuxer and friends to handle detecting CENC content via something other than the presence of the PSSH box.

Comment 11

[Sander Saares]

Common Encryption defines the "tenc" (Track Encryption) box as mandatory for encrypted tracks - that would likely be the best indicator to use here.

Comment 12

[Sander Saares]

We applied a hotfix to the test content to fix an error: the saiz box incorrectly claimed that the sample aux info size was 10, where in fact it should have been 8. This is now corrected.

It was the reason the content failed to play back on Chrome (now works fine). This may also have been a contributing factor preventing correct playback on Firefox.

No user-visible change in Firefox behavior, though - the video still fails to play in Firefox, even if the PSSH box is added to the audio tracks.

Comment 13

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Bug 1300069 - Consider a media encrypted if it contains a track with crypto metadata, rather than only if the media contains encryption init data.

Some encrypted MP4 streams don't contain PSSH boxes in their MOOV boxes, but
they still have tracks with valid TENC boxes which we still parse. So we need
to consider media encrypted if any of its tracks have crypto meta data, rather
than only if the media has crypto init data.

The WebM demuxer's crypto init data is all the tracks' crypto meta data, so
WebM isn't broken by this change.

Review commit: https://reviewboard.mozilla.org/r/81578/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/81578/

Comment 14

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Bug 1300069 - Rename MediaDataDemuxer::GetCrypto to GetCryptoInitData to better reflect its purpose.

Review commit: https://reviewboard.mozilla.org/r/81580/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/81580/

Comment 15

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Bug 1300069 - Remove TrackBuffersManager::mIsEncrypted.

It's write only, so there's no point storing this, and it's not accurate
anyway, as it's actually tracking whether there's encrypted init data in
the media, not whether its got encrypted tracks.

Review commit: https://reviewboard.mozilla.org/r/81582/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/81582/

Comment 16

[Jean-Yves Avenard [:jya]]

Comment on attachment 8795578 [details]
Bug 1300069 - Consider a media encrypted if it contains a track with crypto metadata, rather than only if the media contains encryption init data.

https://reviewboard.mozilla.org/r/81578/#review80146

::: dom/media/MediaFormatReader.h:108
(Diff revision 1)
>  
>    void SetVideoBlankDecode(bool aIsBlankDecode) override;
>  
>  private:
>  
> -  bool HasVideo() { return mVideo.mTrackDemuxer; }
> +  bool HasVideo() const { return mVideo.mTrackDemuxer; }

ideally, const change should be a separate patch

Comment 17

[Jean-Yves Avenard [:jya]]

Comment on attachment 8795579 [details]
Bug 1300069 - Rename MediaDataDemuxer::GetCrypto to GetCryptoInitData to better reflect its purpose.

https://reviewboard.mozilla.org/r/81580/#review80148

I don't see much value in this commit. While it may clarify a particular use. The type it returns make it even more confusing; and the objects it uses do not line up either.

e.g. I don't see how renaming the function clarifies anything

::: dom/media/MediaDataDemuxer.h:71
(Diff revision 1)
>    virtual bool IsSeekableOnlyInBufferedRanges() const { return false; }
>  
> -  // Returns the media's crypto information, or nullptr if media isn't
> -  // encrypted.
> -  virtual UniquePtr<EncryptionInfo> GetCrypto()
> +  // Returns the media's crypto init data, or nullptr if media doesn't
> +  // contain crypto init data. Note: the media may still have encrypted
> +  // tracks and be transmitting the init data out of band.
> +  virtual UniquePtr<EncryptionInfo> GetCryptoInitData()

Seeing that the object returned is EncryptionInfo, wouldn't it be more approprate to rename this GetCryptoInfo ?

With a comment that EncryptionInfo is the init data.

That or also rename the EncryptionInfo object, as it's now even more confusing

Comment 18

[Jean-Yves Avenard [:jya]]

Comment on attachment 8795580 [details]
Bug 1300069 - Remove TrackBuffersManager::mIsEncrypted.

https://reviewboard.mozilla.org/r/81582/#review80154

::: dom/media/mediasource/TrackBuffersManager.cpp:1076
(Diff revision 1)
>      }
>  #endif // MOZ_EME
>      info.mCrypto = *crypto;
>      // We clear our crypto init data array, so the MediaFormatReader will
>      // not emit an encrypted event for the same init data again.
>      info.mCrypto.mInitDatas.Clear();

Side note.
Wouldn't this makes

Comment 19

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 8795579 [details]
Bug 1300069 - Rename MediaDataDemuxer::GetCrypto to GetCryptoInitData to better reflect its purpose.

Discussed with jya on #media, we'll shelve renaming this, as it would complicate uplifting a fix. Will take this up in another bug.

Comment 20

[Chris Pearce [:cpearce (Not reading bugmail)]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c5c52f1817b51b15829f46ff54f421a00cca6759
Bug 1300069 - Consider a media encrypted if it contains a track with crypto metadata, rather than only if the media contains encryption init data. r=jya

https://hg.mozilla.org/integration/mozilla-inbound/rev/ccc856b06a2ddbfffd0119522c0cbd95ff88c4fe
Bug 1300069 - Remove TrackBuffersManager::mIsEncrypted. r=jya

Comment 21

[Chris Pearce [:cpearce (Not reading bugmail)]]

Must remember to uplift...

Comment 22

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c5c52f1817b5
https://hg.mozilla.org/mozilla-central/rev/ccc856b06a2d

Comment 23

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 8795578 [details]
Bug 1300069 - Consider a media encrypted if it contains a track with crypto metadata, rather than only if the media contains encryption init data.

Approval Request Comment
[Feature/regressing bug #]: EME video playback
[User impact if declined]: Some encrypted videos which are encoded with the "encryption init data" delivered out of band will fail to playback correctly.
[Describe test coverage new/current, TreeHerder]: We have lots of EME mochitests.
[Risks and why]: Low, the fix is a simple change to the logic as to which markers in the streams we use to determine whether a stream is encrypted.
[String/UUID change made/needed]: None.

Comment 24

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hello Sander, could you please verify this issue is fixed as expected on a latest Nightly build? Thanks!

Comment 25

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8795578 [details]
Bug 1300069 - Consider a media encrypted if it contains a track with crypto metadata, rather than only if the media contains encryption init data.

Media playback issues with some types of encrypted media, Aurora51+, Beta50+

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b311b5854c90
https://hg.mozilla.org/releases/mozilla-aurora/rev/50a2d49431a9

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/996a38715d67
https://hg.mozilla.org/releases/mozilla-beta/rev/d67b181d7d11

Comment 29

[Sander Saares]

I confirm that behavior conforms to expectations on latest Nightly build.