Closed Bug 1505882 Opened 5 years ago Closed 5 years ago

Can't play videos on Amazon Prime Video ("video unavailable")


(Core :: Audio/Video: Playback, defect, P1)




Tracking Status
firefox-esr60 --- unaffected
firefox63 --- unaffected
firefox64 --- unaffected
firefox65 blocking verified


(Reporter: flod, Assigned: jya, NeedInfo)



(Keywords: regression)


(5 files)

Using the latest Nightly on macOS (65.0a1 (2018-11-08) (64 bit)) I get a "video unavailable" message on any video I try to play.

The same video works on Beta, and was working 24 hours ago on Nightly (2018-11-07 build, according to history updates).
Per Slack conversation, it seems to happen on Win10 too, but not on Linux.

The dialog displayed has a "Video Unavailable" title, and ""We're experiencing a problem playing this video. For assistance, please go to" text.

One of the errors I see in the console is

Media resource blob: could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005)
Details: virtual ipc::IPCResult mozilla::gmp::ChromiumCDMParent::RecvDecodeFailed(const uint32_t &): ChromiumCDMParent::RecvDecodeFailed
I tried to get a bisect on this, but I was hitting issues with builds failing which had worked previously and the like. I sorta-wonder if there's a change on Amazon's end at play here.
Component: Untriaged → Audio/Video: Playback
Flags: needinfo?(bvandyk)
Product: Firefox → Core
same here.
- I don't think we've made any Widevine path code changes in the last week. So don't think this would be fallout from that.
- We shouldn't have changed the CDM version in use, but to make sure, about:support should show `media.gmp-widevinecdm.version` at 4.10.1146.0.
- If the sandbox has changed, it's possible that could cause issue, though I'd think we'd fail earlier.

Will keep looking into this. I'm not immediately able to access Prime Video, but hope to be able to shortly. I'm currently outside the US, so may need some help if geoblocking ends up being blocker.

Sifting the pushlog in the mean time to see if anything stands out.
Flags: needinfo?(bvandyk)
I'm able to repro, and it appears the geoblocked content I have available should be sufficient to debug. I too saw the issue kick in when bumping to a recent build. I'm on the expected CDM. Investigating further now. P1, we'll want to resolve this in the current cycle.
Assignee: nobody → bvandyk
Priority: -- → P1
Clarifying bisection issues seen by others: Verified Media Path, which involves signing the CDM and Firefox binaries, can throw a wrench into the works for debugging these kinds of issues. It looks to me like failures when bisecting this are related to non-signed builds being rejected by Amazon. This is annoying as it means the smallest unit I've been able to bisect is between signed central builds, which I manually download from treeherder.

Based on doing such a bisection the issue appears to lie in this push: or that something changed in infra that's been impacting builds since then. This is in line with the findings in comment #4.

However, my local build, which is built from a fresh pull as of a couple of hours ago, still works. This would suggest that the problem is not solely based on the code in central, and may not be at all. Will continue to investigate and update the bug.
I believe this is a regression caused by Bug 1497951. This can be observed by using the signed builds from a try run where I've backed out those changes[0]. Francesco, could you please let me know if the signed Mac build there no longer suffers the issue?

My hypothesis is that we do not see the issue in local builds because we have 3 different states of builds when interacting with this issue:
- Non-release: includes local builds, not signed, and are somehow identified to Prime as being non-release (I don't know by what mechanism, though I'd like to find out). These work, but appear to be served low resolution content.
- Release, unsigned: built by Mozilla but not signed. These flat out don't work from what I see, and are what causes issues when we try bisect.
- Release, signed: built by Mozilla and signed. These work and can be served higher quality content than non-release.

I think we're seeing the issue only triggered by certain media which we're served only for signed builds.

Flags: needinfo?(francesco.lodolo)
Not incredibly familiar with Treeherder, hopefully I got the right one:
- Select "B" in "OS X Cross Compiled opt":
- Downloaded the target.dmg from artifacts.

Just in case I created a new profile.

Unfortunately the behavior is the same (video doesn't play). In the console I only see a bunch of "AbortError: The operation was aborted."
Flags: needinfo?(francesco.lodolo)
See Also: → 1506015
Coincidental timing between the bugs so it may be related.
FWIW, I tried to play the same site on my Macbook Pro using the latest nightly and it works fine. Spoofed the UA string to Android and still worked so I thought it was a GeckoView problem.
(In reply to Francesco Lodolo [:flod] from comment #9)
> Not incredibly familiar with Treeherder, hopefully I got the right one:
> - Select "B" in "OS X Cross Compiled opt":

Apologies, I should have given clearer instructions. The Bs builds are the singed ones.
OK. Downloaded and started from terminal

This one works fine.
Flags: needinfo?(bvandyk)
This is is what I did to reproduce the issue locally on my mac:

I added this to my .mozconfig
ac_add_options --with-branding=browser/branding/nightly
ac_add_options --enable-clang-plugin
ac_add_options --enable-dmd 
ac_add_options --enable-verify-mar 
mk_add_options 'export MOZILLA_OFFICIAL=1'
ac_add_options --enable-eme=widevine

Then, I copied into the Firefox compiled in:

the one from a signed build.

From that point on, signature is enforced, allowing to easily reproduce the widevine error.
keyframe was tracked in the CodecChangeMonitor in order to determine when to prepend the SPS/PPS to a sample for decoder using AnnexB format.
The assumption was that it was needed only once per the lifetime of the decoder (and indeed this is true with the Windows and Android decoder). However this causes issue with the Widevine H264 decoder and it will error on some content if the first sample passed following a flush doesn't contain a SPS/PPS.

Interestingly, this issue isn't observed with Netflix content or other Widevine sample videos. Only Amazon PrimeVideo content so far.

We instead use the global MediaChangeMonitor keyframe status that knows when the decoder has been drained or flushed.
This will avoid future bugs like bug 1506076.

Depends on D11556
Depends on D11558
ConvertSampleToAnnexB takes the sample's out of band extradata to prepend it to the data. It was theorically possible that the first sample would contain the SPS/PPS from the previous format.

Depends on D11559
Flags: needinfo?(bvandyk)
Blocks: 1497951
Keywords: regression
OS X Cross Compiled opt -> Bs works fine for me

Attachment #9024191 - Attachment description: Bug 1505882 - P3. Don't check the first sample's out of band extradata until after decoding the first frame. r?bryce → Bug 1505882 - P3. Don't check the sample's out of band extradata until after decoding the first frame. r?bryce
Pushed by
P1. Don't track keyframe in CodecChangeMonitor. r=bryce
P2. Assert that sample conversion required is either AVCC or AnnexB. r=bryce
P3. Don't check the sample's out of band extradata until after decoding the first frame. r=bryce
P4. Fix style. r=bryce
P5. Always ensure to use latest SPS/PPS when converting sample. r=bryce
Assignee: bvandyk → jyavenard
Flags: qe-verify+
Reproduced the issue on Win10 x64 on Nightly 65.0a1 20181108220756.

Checked on the latest Nightly 65.0a1 20181115100051 on macOS 10.12, Win10 x64 and ubuntu 16.04 and the issue is fixed, video playing on amazon prime is available and working.
Flags: qe-verify+

It's back. I have MacOS Catalina version 10.15.4. Prime video works on Apple's Safari browser but not on Firefox 75.0

(In reply to Joseph Otero from comment #25)

It's back. I have MacOS Catalina version 10.15.4. Prime video works on Apple's Safari browser but not on Firefox 75.0

Sounds like a new issue, could you create a new bug to track the issue you're seeing?

Flags: needinfo?(josephfotero)
You need to log in before you can comment on or make changes to this bug.