bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

EME video playback fails due to hitting CDM IPC shmem limit

RESOLVED FIXED in Firefox 56

Status

()

Core
Audio/Video: GMP
P1
normal
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: cpearce, Assigned: cpearce)

Tracking

55 Branch
mozilla56
Points:
---

Firefox Tracking Flags

(firefox56 fixed)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

(Assignee)

Description

a year ago
In some situations playback of EME video fails due to us hitting the limit on the number of shmems we pre-allocate for transferring video frames from the CDM to Firefox.

Telemetry shows that we do hit the 50-shmem limit in Beta sometimes (the 50-infinity line in the table):
https://mzl.la/2vO60Ab

This is affecting Beta and Nightly.

This failure appears to happen basically every time in Nightly while watching Amazon Prime Video. I believe this started happening more in Nightly after we pushed the 1.4.8.970 CDM update, as that has a new video decoder which will have new behaviour.

While playing First Contact (1997) on Amazon Prime Video, I see up to 75 (!) shmems being allocated. That means we're putting 75 video frames into the decoder before we're getting anything out. It would seem the new decoder pre-buffers a lot more frames than the old CDM's decoder.

We need to use a new strategy for managing the shmems we use for transferring video frames between the CDM and Firefox.
(Assignee)

Comment 1

a year ago
We should fix this in Beta 55 as this failure happens there, though thankfully not a huge amount.
status-firefox55: --- → affected
status-firefox56: --- → affected
tracking-firefox55: --- → ?
(Assignee)

Comment 2

a year ago
The problem is that we're underestimating the size of buffer required. We increase the number of shmems in our pool if we receive a video frame from the CDM which is not in a shmem. The CDM process allocates a frame which is not a shmem if it doesn't have a shmem big enough. When we switch bitrates, we purge the shmems, and estimate the size off shmem we'll need. The parent then sends the CDM process enough shmems of that size to fill the pool. When the CDM tries to allocate a frame it has a full pool of shmems of too small size, so it returns the frame in a non-shmem buffer. The parent sees this, and increases the pool size, purges all shmems, and resends the CDM process a (now larger) pool of shmems of the correct size

If we switch bitrates enough, we'll increase the pool size like this until we hit the pool size limit, whereupon we error.
(Assignee)

Comment 3

a year ago
(In reply to Chris Pearce (:cpearce) from comment #0)
> While playing First Contact (1997) on Amazon Prime Video, I see up to 75 (!)
> shmems being allocated. That means we're putting 75 video frames into the
> decoder before we're getting anything out. It would seem the new decoder
> pre-buffers a lot more frames than the old CDM's decoder.

For the record, my claim here that we're putting 75 frames into the decoder before getting output is incorrect; we are pre-allocating too many frames as I describe in comment 2.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 6

a year ago
mozreview-review
Comment on attachment 8889265 [details]
Bug 1383580 - Add an explicit message to increase CDM-Firefox shmem pool.

https://reviewboard.mozilla.org/r/160316/#review165590

::: commit-message-d544d:7
(Diff revision 1)
> +
> +The strategy we were using to expand the pool of shmems we use to shuffle video
> +frames between the CDM and content processses is to increase the size of the
> +pool if the content process receives a video frame in a non-shmem. However the
> +CDM process will send a frame in a non-shmem if none of the shmems in the pool
> +are bug enough to fit the frame the CDM produces. This happens if we

'bug' -> 'big'
Attachment #8889265 - Flags: review?(gsquelart) → review+

Comment 7

a year ago
mozreview-review
Comment on attachment 8889266 [details]
Bug 1383580 - Pad estimated CDM frame sizes.

https://reviewboard.mozilla.org/r/160318/#review165594
Attachment #8889266 - Flags: review?(gsquelart) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 10

a year ago
Pushed by cpearce@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/25b1e61ea394
Add an explicit message to increase CDM-Firefox shmem pool. r=gerald
https://hg.mozilla.org/integration/autoland/rev/8ac6475952a1
Pad estimated CDM frame sizes. r=gerald

Comment 11

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/25b1e61ea394
https://hg.mozilla.org/mozilla-central/rev/8ac6475952a1
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox56: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
(Assignee)

Comment 12

a year ago
Must remember to uplift.
Flags: needinfo?(cpearce)
(Assignee)

Comment 13

a year ago
Actually, I think we can get away without this in 55. We only hit this case regularly in Nightly because we the new 970 Widevine CDM doesn't request buffers of the size we expect, but the 903 CDM in 33 does. So we won't hit this degenerate case.

The telemetry shows that while we do somehow hit the limit in beta 55, we do so in 0% of cases. So we're probably ok without this change.
status-firefox55: affected → ---
tracking-firefox55: ? → ---
Flags: needinfo?(cpearce)
You need to log in before you can comment on or make changes to this bug.