Can't play videos on Amazon Prime Video ("video unavailable")

VERIFIED FIXED in Firefox 65

Status

()

defect
P1
major
VERIFIED FIXED
6 months ago
5 months ago

People

(Reporter: flod, Assigned: jya)

Tracking

({regression})

Trunk
mozilla65
Unspecified
macOS
Points:
---

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox63 unaffected, firefox64 unaffected, firefox65blocking verified)

Details

Attachments

(5 attachments)

(Reporter)

Description

6 months ago
Using the latest Nightly on macOS (65.0a1 (2018-11-08) (64 bit)) I get a "video unavailable" message on any video I try to play.

The same video works on Beta, and was working 24 hours ago on Nightly (2018-11-07 build, according to history updates).
(Reporter)

Comment 1

6 months ago
Per Slack conversation, it seems to happen on Win10 too, but not on Linux.

The dialog displayed has a "Video Unavailable" title, and ""We're experiencing a problem playing this video. For assistance, please go to www.amazon.com/videohelp." text.

One of the errors I see in the console is

Media resource blob:https://www.primevideo.com/72377520-09bb-1941-bb53-be416af28dc1 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005)
Details: virtual ipc::IPCResult mozilla::gmp::ChromiumCDMParent::RecvDecodeFailed(const uint32_t &): ChromiumCDMParent::RecvDecodeFailed
I tried to get a bisect on this, but I was hitting issues with builds failing which had worked previously and the like. I sorta-wonder if there's a change on Amazon's end at play here.
Component: Untriaged → Audio/Video: Playback
Flags: needinfo?(bvandyk)
Product: Firefox → Core
(Assignee)

Comment 3

6 months ago
same here.
- I don't think we've made any Widevine path code changes in the last week. So don't think this would be fallout from that.
- We shouldn't have changed the CDM version in use, but to make sure, about:support should show `media.gmp-widevinecdm.version` at 4.10.1146.0.
- If the sandbox has changed, it's possible that could cause issue, though I'd think we'd fail earlier.

Will keep looking into this. I'm not immediately able to access Prime Video, but hope to be able to shortly. I'm currently outside the US, so may need some help if geoblocking ends up being blocker.

Sifting the pushlog in the mean time to see if anything stands out.
Flags: needinfo?(bvandyk)
I'm able to repro, and it appears the geoblocked content I have available should be sufficient to debug. I too saw the issue kick in when bumping to a recent build. I'm on the expected CDM. Investigating further now. P1, we'll want to resolve this in the current cycle.
Assignee: nobody → bvandyk
Priority: -- → P1
Clarifying bisection issues seen by others: Verified Media Path, which involves signing the CDM and Firefox binaries, can throw a wrench into the works for debugging these kinds of issues. It looks to me like failures when bisecting this are related to non-signed builds being rejected by Amazon. This is annoying as it means the smallest unit I've been able to bisect is between signed central builds, which I manually download from treeherder.

Based on doing such a bisection the issue appears to lie in this push: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=12afa29e9c8cde597314f072c6f0c1f81d681b40 or that something changed in infra that's been impacting builds since then. This is in line with the findings in comment #4.

However, my local build, which is built from a fresh pull as of a couple of hours ago, still works. This would suggest that the problem is not solely based on the code in central, and may not be at all. Will continue to investigate and update the bug.
I believe this is a regression caused by Bug 1497951. This can be observed by using the signed builds from a try run where I've backed out those changes[0]. Francesco, could you please let me know if the signed Mac build there no longer suffers the issue?

My hypothesis is that we do not see the issue in local builds because we have 3 different states of builds when interacting with this issue:
- Non-release: includes local builds, not signed, and are somehow identified to Prime as being non-release (I don't know by what mechanism, though I'd like to find out). These work, but appear to be served low resolution content.
- Release, unsigned: built by Mozilla but not signed. These flat out don't work from what I see, and are what causes issues when we try bisect.
- Release, signed: built by Mozilla and signed. These work and can be served higher quality content than non-release.

I think we're seeing the issue only triggered by certain media which we're served only for signed builds.

[0]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210717376
Flags: needinfo?(francesco.lodolo)
(Reporter)

Comment 9

6 months ago
Not incredibly familiar with Treeherder, hopefully I got the right one:
- Select "B" in "OS X Cross Compiled opt": https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210702635
- Downloaded the target.dmg from artifacts.

Just in case I created a new profile.

Unfortunately the behavior is the same (video doesn't play). In the console I only see a bunch of "AbortError: The operation was aborted."
Flags: needinfo?(francesco.lodolo)

Updated

6 months ago
See Also: → 1506015
Coincidental timing between the bugs so it may be related.
FWIW, I tried to play the same site https://twitch.tv/ninja on my Macbook Pro using the latest nightly and it works fine. Spoofed the UA string to Android and still worked so I thought it was a GeckoView problem.
(In reply to Francesco Lodolo [:flod] from comment #9)
> Not incredibly familiar with Treeherder, hopefully I got the right one:
> - Select "B" in "OS X Cross Compiled opt":

Apologies, I should have given clearer instructions. The Bs builds are the singed ones.
(Reporter)

Comment 12

5 months ago
OK. Downloaded and started from terminal
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210715055

This one works fine.
Flags: needinfo?(bvandyk)
(Assignee)

Comment 13

5 months ago
This is is what I did to reproduce the issue locally on my mac:

I added this to my .mozconfig
ac_add_options --with-branding=browser/branding/nightly
ac_add_options --enable-clang-plugin
ac_add_options --enable-dmd 
ac_add_options --enable-verify-mar 
mk_add_options 'export MOZILLA_OFFICIAL=1'
ac_add_options --enable-eme=widevine


Then, I copied into the Firefox Nightly.app compiled in:
Firefox Nightly.app/Content/MacOS/plugin-container.app

the one from a signed build.

From that point on, signature is enforced, allowing to easily reproduce the widevine error.
(Assignee)

Comment 14

5 months ago
keyframe was tracked in the CodecChangeMonitor in order to determine when to prepend the SPS/PPS to a sample for decoder using AnnexB format.
The assumption was that it was needed only once per the lifetime of the decoder (and indeed this is true with the Windows and Android decoder). However this causes issue with the Widevine H264 decoder and it will error on some content if the first sample passed following a flush doesn't contain a SPS/PPS.

Interestingly, this issue isn't observed with Netflix content or other Widevine sample videos. Only Amazon PrimeVideo content so far.

We instead use the global MediaChangeMonitor keyframe status that knows when the decoder has been drained or flushed.
(Assignee)

Comment 15

5 months ago
This will avoid future bugs like bug 1506076.

Depends on D11556
(Assignee)

Comment 17

5 months ago
Depends on D11558
(Assignee)

Comment 18

5 months ago
ConvertSampleToAnnexB takes the sample's out of band extradata to prepend it to the data. It was theorically possible that the first sample would contain the SPS/PPS from the previous format.

Depends on D11559
(Reporter)

Updated

5 months ago
Flags: needinfo?(bvandyk)
(Assignee)

Updated

5 months ago
Blocks: 1497951
Keywords: regression
(Reporter)

Comment 20

5 months ago
OS X Cross Compiled opt -> Bs works fine for me

about:buildconfig https://hg.mozilla.org/try/rev/75e762234e4dbec56a6cc5a46891df125689db69
Attachment #9024191 - Attachment description: Bug 1505882 - P3. Don't check the first sample's out of band extradata until after decoding the first frame. r?bryce → Bug 1505882 - P3. Don't check the sample's out of band extradata until after decoding the first frame. r?bryce

Comment 21

5 months ago
Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/69c386301fab
P1. Don't track keyframe in CodecChangeMonitor. r=bryce
https://hg.mozilla.org/integration/autoland/rev/f26fe4990bda
P2. Assert that sample conversion required is either AVCC or AnnexB. r=bryce
https://hg.mozilla.org/integration/autoland/rev/8dbde7a84d53
P3. Don't check the sample's out of band extradata until after decoding the first frame. r=bryce
https://hg.mozilla.org/integration/autoland/rev/4573b20405f3
P4. Fix style. r=bryce
https://hg.mozilla.org/integration/autoland/rev/eaef9f93b37c
P5. Always ensure to use latest SPS/PPS when converting sample. r=bryce
Assignee: bvandyk → jyavenard
Flags: qe-verify+

Comment 23

5 months ago
Reproduced the issue on Win10 x64 on Nightly 65.0a1 20181108220756.

Checked on the latest Nightly 65.0a1 20181115100051 on macOS 10.12, Win10 x64 and ubuntu 16.04 and the issue is fixed, video playing on amazon prime is available and working.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
(Assignee)

Updated

5 months ago
Duplicate of this bug: 1508554
You need to log in before you can comment on or make changes to this bug.