2080876 - Amazon Prime video playbacks fails in Firefox 158+ (release: Oct 13)

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Harshit reported he cannot play Amazon Prime videos in Firefox Nightly anymore. Others could reproduce the issue, release seems unaffected. He was still able to get the expected behavior ca. 12 hours ago.

Bisection by Claas points to bug 2051510 which landed a month ago. The change should be in the content served from Prime to clients.

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 2080876 - Log GMP service initialization and MediaCapabilities decoder support queries. r?#media-playback-reviewers!

This let us see the race clearly, but it's good anyway.

Comment 2

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 2080876 - Avoid XPCOM service lookup in HaveGMPFor. r?#media-playback-reviewers!

A background lookup can do GMP service creation and wait for the main
thread, while the main thread waits for that pending service. Use the
direct GMP singleton getter to break this cycle.

The drawback is that the direct getter synchronously dispatches to the
main thread for every background lookup, even when the singleton already
exists. XPCOM previously returned its cached service without that hop.

Unfixed Prime Video log, with prefixes shortened:
14:42:19.614316 [MediaPDecoder #1] HaveGMPFor: api=decode-video, mainThread=false
14:42:19.614331 [MediaPDecoder #1] GMPServiceCreateHelper::GetOrCreate: waiting for main thread
14:42:19.618142 [Main Thread] HaveGMPFor: api=chromium-cdm11-host4, mainThread=true
Neither lookup returns.

With the fix, main arrives after the background wait but creates the
singleton, allowing the background lookup to resume:
14:40:12.549437 [Main Thread] GMPServiceCreateHelper: created singleton=0x1129c4100
14:40:12.558611 [MediaPDecoder #1] GMPServiceCreateHelper::GetOrCreate: main thread returned 0x1129c4100

Comment 3

[BugBot [:suhaib / :marco]]

:pehrsons, since you are the author of the regressor, bug 2051510, could you take a look?

For more information, please visit BugBot documentation.

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 2080876 - Avoid deadlocking EME requests against GMP service initialization. r?#media-playback-reviewers!

A MediaCapabilities decoder probe can start creating the GMP service on a
background thread and wait for main to initialize it. If an EME request
then calls HaveGMPFor on main, it waits for that pending service creation
and prevents the initialization runnable from executing.

Fix this by moving EME's GMP support checks to the PDM thread pool, so main
remains free to initialize the service. Assert that HaveGMPFor runs off
main to catch callers that could reintroduce the deadlock.

To consume those checks asynchronously, return a promise from
GetKeySystemStatus and continue request processing on main when it
resolves. Adapt CDM installation retries to await the status too, and
ignore results for requests that have expired or been shut down. Filter
supported key systems off main before assembling their configurations on
main; DOM promise settlement also stays on main.

Comment 5

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Log GMP service initialization and MediaCapabilities decoder support queries.

This let us see the race clearly, but it's good anyway.

Original Revision: https://phabricator.services.mozilla.com/D332920

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Avoid deadlocking EME requests against GMP service initialization.

A MediaCapabilities decoder probe can start creating the GMP service on a
background thread and wait for main to initialize it. If an EME request
then calls HaveGMPFor on main, it waits for that pending service creation
and prevents the initialization runnable from executing.

Fix this by moving EME's GMP support checks to the PDM thread pool, so main
remains free to initialize the service. Assert that HaveGMPFor runs off
main to catch callers that could reintroduce the deadlock.

To consume those checks asynchronously, return a promise from
GetKeySystemStatus and continue request processing on main when it
resolves. Adapt CDM installation retries to await the status too, and
ignore results for requests that have expired or been shut down. Filter
supported key systems off main before assembling their configurations on
main; DOM promise settlement also stays on main.

Original Revision: https://phabricator.services.mozilla.com/D332938

Comment 7

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: Fail to watch Amazon Prime Videos.
• Code covered by automated testing?: no
• Fix verified in Nightly?: no
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing: Watch Amazon Prime Videos
• Risk associated with taking this patch: low
• Explanation of risk level: Not a user observable behavior change, it's about tweak our internal pipeline to ask an answer asynchronously to prevent deadlock.
• String changes made/needed?: No
• Is Android affected?: unknown

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/db6bd4c86de4
https://hg.mozilla.org/integration/autoland/rev/27c2b1d03084
Revert "Bug 2080876 - Avoid deadlocking EME requests against GMP service initialization. r=media-playback-reviewers,alwu" for causing gtest failures @GMPUtils.cpp.

Comment 9

[agoloman]

Backed out for causing gtest failures @GMPUtils.cpp.

• Backout link
• Push with failures
• Failure Log

Comment 10

[Alastor Wu [:alwu]]

I'm investigating that.

Comment 11

[Alastor Wu [:alwu]]

The patch D332938 does not fix all of the deadlock. It fixes only one case.

Why the deadlock occurs:

1. A background thread asks for the GMP service before the service exists.
2. The background thread starts to make the service. It waits for the main thread.
3. The main thread then asks for the same service.
4. The main thread waits for the background thread. The two threads stop.

Since bug 2051510, decodingInfo() makes the GMP service on a background thread. Thus, every main thread caller of HaveGMPFor can cause this deadlock.

D332938 moves only the EME check off the main thread. Other code calls HaveGMPFor on the main thread:

• new RTCPeerConnection(), through DefaultCodecPreferences.
• RTCRtpSender.getCapabilities() and RTCRtpReceiver.getCapabilities().
• The parent process, when it opens an MP4 file as a video document.

These callers caused the assertion failures and the backout in comment 9. The try push 42bbb745dc84 shows the same crashes in many media and WebRTC tests. I'm experimenting another fix now.

Comment 12

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Part 1: Answer GMP support queries in child processes from the cached capabilities. r?#media-playback-reviewers!

Child processes receive the list of installed GMPs from the parent
process, and the child GMP service only reads that list to report plugin
support. Querying through the service still created it lazily, and when
the first query came from a background thread, a main thread query made
at the same time could wait forever for that creation to finish, because
the creation itself needs the main thread.

Read the cached capability list directly in child processes, so support
queries no longer create the GMP service at all and can run on any
thread without blocking on the main thread. The parent process keeps
using its service, which scans the plugins on disk.

Comment 13

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Part 2: Add a test for EME requests racing GMP service creation from MediaCapabilities. r?#media-playback-reviewers!

Prime Video queries media capabilities and then requests key system
access a few milliseconds later, in a content process where the GMP
service does not exist yet. The capabilities query asked the GMP decoder
module for support on a background task queue, which started creating
the service and waited for the main thread. The key system request then
asked for the same service on the main thread and waited for the pending
creation, so neither thread made progress.

The test reproduces that sequence in a fresh content process. It blocks
the main thread between the two calls, so that the background query runs
first.

Comment 14

[Shawn Zivontsis]

(In reply to Alastor Wu [:alwu] from comment #11)

...
D332938 moves only the EME check off the main thread. Other code calls HaveGMPFor on the main thread:

• new RTCPeerConnection(), through DefaultCodecPreferences.
  ...

FWIW, I suspect this issue is preventing me from loading the Questrade login page (https://login.questrade.com/account/login) as well. A fingerprinting script on there seems to be calling mediaCapabilities.decodingInfo() followed by new RTCPeerConnection(), which hangs the page load

I was able to work around the issue either by setting media.peerconnection.enabled to false, or by adding the ublock rule ||echo.sterope.site^$domain=login.questrade.com.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/0a391bbac7f4
https://hg.mozilla.org/integration/autoland/rev/680d388dabe0
Part 1: Answer GMP support queries in child processes from the cached capabilities. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/880099cab927
https://hg.mozilla.org/integration/autoland/rev/62134d3c5112
Part 2: Add a test for EME requests racing GMP service creation from MediaCapabilities. r=media-playback-reviewers,jolin

Comment 16

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Part 1: Answer GMP support queries in child processes from the cached capabilities.

Read the cached GMP capability list directly when answering GMP support
queries in child processes, instead of getting the GMP service first.
The parent process sends this list to each content process, and the
child GMP service already answered from it, so the answers do not
change. Also clear the list under its lock at shutdown, because any
thread can now read it.

Getting the service created it lazily. When the first query came from a
background thread, a main thread query made at the same time could wait
forever for that creation to finish, because the creation itself needs
the main thread.

Late in shutdown, a child process can now answer from the cached list,
where it used to answer no because the service could no longer be
created.

Original Revision: https://phabricator.services.mozilla.com/D333074

Comment 17

[Alastor Wu [:alwu]]

Attachment: Bug 2080876 - Part 2: Add a test for EME requests racing GMP service creation from MediaCapabilities.

Add a browser test that queries media capabilities and then requests key
system access in a fresh content process that has not created the GMP
service yet. The test blocks the main thread between the two calls so
that the background support query runs first, and checks that both
requests resolve.

This is the sequence Prime Video uses on load. Before the fix, the
capabilities query started creating the GMP service on a background task
queue and waited for the main thread, while the key system request
waited on the main thread for that pending creation, so neither request
finished.

Original Revision: https://phabricator.services.mozilla.com/D333075

Comment 18

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: Fail to watch Amazon Prime Videos.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: no
• Needs manual QE testing?: yes
• Steps to reproduce for manual QE testing: Watch Amazon Prime Videos
• Risk associated with taking this patch: low
• Explanation of risk level: Its changes is based on the existing mechanism which has been tested for long while, and it's not a user observable behavior change.
• String changes made/needed?: No
• Is Android affected?: unknown

Comment 19

[Shawn Zivontsis]

FWIW, I did also experience the questrade logon issue I mentioned above on Android as well

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/f620c4f2ac22
https://hg.mozilla.org/mozilla-central/rev/7d308fd68335
Log GMP service initialization and MediaCapabilities decoder support queries. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/b84f2e9c41d3
https://hg.mozilla.org/mozilla-central/rev/8a061c2658ff
Avoid deadlocking EME requests against GMP service initialization. r=media-playback-reviewers,alwu

Comment 21

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/680d388dabe0
https://hg.mozilla.org/mozilla-central/rev/62134d3c5112

Comment 22

[Andreas Pehrson [:pehrsons]]

Could you confirm that latest Nightly fixes it for you? Make sure it's a 13 Oct build or newer. Thanks

Comment 23

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The Nightly build just started to build. It will be available in 2-3 hours.

Comment 24

[Shawn Zivontsis]

It appears to be working for me now, thank you!

I haven't received the new build on mobile yet, but after updating desktop I am now able to access the Questrade login page without any workarounds in place

Comment 25

[Timea Zsoldos [:tzsoldos], Desktop QA]

Reproduced this issue with Firefox nightly 160.0a1 build from 2026-10-12 on Ubuntu 26, macOS 13 and macOS 12.
Verified as fixed on latest nightly Firefox 160.0a1, build ID: 20261013094244 with Ubuntu 26, macOS 13 and macOS 12, Windows 11, the issue no longer persist, Amazon Prime videos loaded and played correctly, also tried Netflix, Bitmovin, HBOMax, Hulu, Shaka, Spotify and Disney+ videos worked.

Comment 26

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/b81611f67506
https://hg.mozilla.org/releases/mozilla-beta/rev/c10882ee2c5b
https://github.com/mozilla-firefox/firefox/commit/28d2701d93ac
https://hg.mozilla.org/releases/mozilla-beta/rev/413e3d1ce042

Comment 29

[Timea Zsoldos [:tzsoldos], Desktop QA]

Verified as fixed on Firefox 159.0b2 with Ubuntu 26, macOS and Windows 11, the issue no longer persist, Amazon Prime videos loaded and played correctly, also tried Netflix, Bitmovin, HBOMax, Hulu, Shaka, Spotify and Disney+ videos worked.

Comment 30

[Performance Sheriff Bot]

Perfherder has detected a browsertime performance change from push 62134d3c511273a73624a31f0bdaa23b0033f168.

No action is required from the author; this comment is provided for informational purposes only.

Improvement	Test	Platform	Options	Absolute values [old vs new]	Performance Profiles
77%	media-capabilities first-query-cold (doc)	linux2404-64-shippable	fission webrender	9.04 ms -> 2.08 ms	Before/After
74%	media-capabilities first-query-cold (doc)	macosx1470-64-shippable	fission webrender	2.90 ms -> 0.76 ms	Before/After

Need Help or Information?

If you have any questions, please reach out to [e-mail]. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.