1505882 - Can't play videos on Amazon Prime Video ("video unavailable")

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[Francesco Lodolo [:flod]]

Using the latest Nightly on macOS (65.0a1 (2018-11-08) (64 bit)) I get a "video unavailable" message on any video I try to play.

The same video works on Beta, and was working 24 hours ago on Nightly (2018-11-07 build, according to history updates).

Comment 1

[Francesco Lodolo [:flod]]

Per Slack conversation, it seems to happen on Win10 too, but not on Linux.

The dialog displayed has a "Video Unavailable" title, and ""We're experiencing a problem playing this video. For assistance, please go to www.amazon.com/videohelp." text.

One of the errors I see in the console is

Media resource blob:https://www.primevideo.com/72377520-09bb-1941-bb53-be416af28dc1 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005)
Details: virtual ipc::IPCResult mozilla::gmp::ChromiumCDMParent::RecvDecodeFailed(const uint32_t &): ChromiumCDMParent::RecvDecodeFailed

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I tried to get a bisect on this, but I was hitting issues with builds failing which had worked previously and the like. I sorta-wonder if there's a change on Amazon's end at play here.

Comment 3

[Jean-Yves Avenard [:jya]]

same here.

Comment 4

[Francesco Lodolo [:flod]]

For me

65.0a1 (2018-11-07) (64 bit) doesn't work
Downloaded from: http://archive.mozilla.org/pub/firefox/nightly/2018/11/2018-11-07-22-01-28-mozilla-central-l10n/
Built from https://hg.mozilla.org/mozilla-central/rev/5836a60614764631436bf5030c5baa34c676c7a2

65.0a1 (2018-11-07) (64 bit) works
Downloaded from: http://archive.mozilla.org/pub/firefox/nightly/2018/11/2018-11-07-10-01-35-mozilla-central-l10n/
Built from https://hg.mozilla.org/mozilla-central/rev/070757a0160c8e6156cf6d8d567a0af280dc33e8

Comment 5

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

- I don't think we've made any Widevine path code changes in the last week. So don't think this would be fallout from that.
- We shouldn't have changed the CDM version in use, but to make sure, about:support should show `media.gmp-widevinecdm.version` at 4.10.1146.0.
- If the sandbox has changed, it's possible that could cause issue, though I'd think we'd fail earlier.

Will keep looking into this. I'm not immediately able to access Prime Video, but hope to be able to shortly. I'm currently outside the US, so may need some help if geoblocking ends up being blocker.

Sifting the pushlog in the mean time to see if anything stands out.

Comment 6

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I'm able to repro, and it appears the geoblocked content I have available should be sufficient to debug. I too saw the issue kick in when bumping to a recent build. I'm on the expected CDM. Investigating further now. P1, we'll want to resolve this in the current cycle.

Comment 7

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Clarifying bisection issues seen by others: Verified Media Path, which involves signing the CDM and Firefox binaries, can throw a wrench into the works for debugging these kinds of issues. It looks to me like failures when bisecting this are related to non-signed builds being rejected by Amazon. This is annoying as it means the smallest unit I've been able to bisect is between signed central builds, which I manually download from treeherder.

Based on doing such a bisection the issue appears to lie in this push: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=12afa29e9c8cde597314f072c6f0c1f81d681b40 or that something changed in infra that's been impacting builds since then. This is in line with the findings in comment #4.

However, my local build, which is built from a fresh pull as of a couple of hours ago, still works. This would suggest that the problem is not solely based on the code in central, and may not be at all. Will continue to investigate and update the bug.

Comment 8

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I believe this is a regression caused by Bug 1497951. This can be observed by using the signed builds from a try run where I've backed out those changes[0]. Francesco, could you please let me know if the signed Mac build there no longer suffers the issue?

My hypothesis is that we do not see the issue in local builds because we have 3 different states of builds when interacting with this issue:
- Non-release: includes local builds, not signed, and are somehow identified to Prime as being non-release (I don't know by what mechanism, though I'd like to find out). These work, but appear to be served low resolution content.
- Release, unsigned: built by Mozilla but not signed. These flat out don't work from what I see, and are what causes issues when we try bisect.
- Release, signed: built by Mozilla and signed. These work and can be served higher quality content than non-release.

I think we're seeing the issue only triggered by certain media which we're served only for signed builds.

[0]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210717376

Comment 9

[Francesco Lodolo [:flod]]

Not incredibly familiar with Treeherder, hopefully I got the right one:
- Select "B" in "OS X Cross Compiled opt": https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210702635
- Downloaded the target.dmg from artifacts.

Just in case I created a new profile.

Unfortunately the behavior is the same (video doesn't play). In the console I only see a bunch of "AbortError: The operation was aborted."

Comment 10

[Vicky Chin [:vchin]]

Coincidental timing between the bugs so it may be related.
FWIW, I tried to play the same site https://twitch.tv/ninja on my Macbook Pro using the latest nightly and it works fine. Spoofed the UA string to Android and still worked so I thought it was a GeckoView problem.

Comment 11

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

(In reply to Francesco Lodolo [:flod] from comment #9)
> Not incredibly familiar with Treeherder, hopefully I got the right one:
> - Select "B" in "OS X Cross Compiled opt":

Apologies, I should have given clearer instructions. The Bs builds are the singed ones.

Comment 12

[Francesco Lodolo [:flod]]

OK. Downloaded and started from terminal
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e6139b81579d4ffe0190af11731b721da0b67cc&selectedJob=210715055

This one works fine.

Comment 13

[Jean-Yves Avenard [:jya]]

This is is what I did to reproduce the issue locally on my mac:

I added this to my .mozconfig
ac_add_options --with-branding=browser/branding/nightly
ac_add_options --enable-clang-plugin
ac_add_options --enable-dmd 
ac_add_options --enable-verify-mar 
mk_add_options 'export MOZILLA_OFFICIAL=1'
ac_add_options --enable-eme=widevine


Then, I copied into the Firefox Nightly.app compiled in:
Firefox Nightly.app/Content/MacOS/plugin-container.app

the one from a signed build.

From that point on, signature is enforced, allowing to easily reproduce the widevine error.

Comment 14

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1505882 - P1. Don't track keyframe in CodecChangeMonitor. r?bryce

keyframe was tracked in the CodecChangeMonitor in order to determine when to prepend the SPS/PPS to a sample for decoder using AnnexB format.
The assumption was that it was needed only once per the lifetime of the decoder (and indeed this is true with the Windows and Android decoder). However this causes issue with the Widevine H264 decoder and it will error on some content if the first sample passed following a flush doesn't contain a SPS/PPS.

Interestingly, this issue isn't observed with Netflix content or other Widevine sample videos. Only Amazon PrimeVideo content so far.

We instead use the global MediaChangeMonitor keyframe status that knows when the decoder has been drained or flushed.

Comment 15

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1505882 - P2. Assert that sample conversion required is either AVCC or AnnexB. r?bryce!

This will avoid future bugs like bug 1506076.

Depends on D11556

Comment 16

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1505882 - P3. Don't check the sample's out of band extradata until after decoding the first frame. r?bryce

Depends on D11557

Comment 17

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1505882 - P4. Fix style. r?bryce

Depends on D11558

Comment 18

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1505882 - P5. Always ensure to use latest SPS/PPS when converting sample. r?bryce

ConvertSampleToAnnexB takes the sample's out of band extradata to prepend it to the data. It was theorically possible that the first sample would contain the SPS/PPS from the previous format.

Depends on D11559

Comment 19

[Jean-Yves Avenard [:jya]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=75e762234e4dbec56a6cc5a46891df125689db69

:flod you can try this build.

Comment 20

[Francesco Lodolo [:flod]]

OS X Cross Compiled opt -> Bs works fine for me

about:buildconfig https://hg.mozilla.org/try/rev/75e762234e4dbec56a6cc5a46891df125689db69

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/69c386301fab
P1. Don't track keyframe in CodecChangeMonitor. r=bryce
https://hg.mozilla.org/integration/autoland/rev/f26fe4990bda
P2. Assert that sample conversion required is either AVCC or AnnexB. r=bryce
https://hg.mozilla.org/integration/autoland/rev/8dbde7a84d53
P3. Don't check the sample's out of band extradata until after decoding the first frame. r=bryce
https://hg.mozilla.org/integration/autoland/rev/4573b20405f3
P4. Fix style. r=bryce
https://hg.mozilla.org/integration/autoland/rev/eaef9f93b37c
P5. Always ensure to use latest SPS/PPS when converting sample. r=bryce

Comment 22

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/69c386301fab
https://hg.mozilla.org/mozilla-central/rev/f26fe4990bda
https://hg.mozilla.org/mozilla-central/rev/8dbde7a84d53
https://hg.mozilla.org/mozilla-central/rev/4573b20405f3
https://hg.mozilla.org/mozilla-central/rev/eaef9f93b37c

Comment 23

[laszlo.bialis]

Reproduced the issue on Win10 x64 on Nightly 65.0a1 20181108220756.

Checked on the latest Nightly 65.0a1 20181115100051 on macOS 10.12, Win10 x64 and ubuntu 16.04 and the issue is fixed, video playing on amazon prime is available and working.

Comment 25

[Joseph Otero]

It's back. I have MacOS Catalina version 10.15.4. Prime video works on Apple's Safari browser but not on Firefox 75.0

Comment 26

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

(In reply to Joseph Otero from comment #25)

It's back. I have MacOS Catalina version 10.15.4. Prime video works on Apple's Safari browser but not on Firefox 75.0

Sounds like a new issue, could you create a new bug to track the issue you're seeing?