Amazon live streams do not load in Firefox 47 on OS X

VERIFIED FIXED in Firefox 47

Status

()

defect
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: cpeterson, Assigned: jya)

Tracking

({regression, site-compat})

47 Branch
mozilla49
Unspecified
All
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox46 unaffected, firefox47blocking fixed, firefox48+ fixed, firefox49+ verified)

Details

()

Attachments

(1 attachment)

Amazon reports:

We are seeing another issue with Firefox 47 and our live streams now. This is a serious regression that affects non-DRM content.

Firefox 47 on Windows is working fine, but on Mac, Firefox seems to be incorrectly extracting the duration of the media segments. This is an example dump of video.buffered ranges I am seeing:

[{“start":6039329845.866,"end":6039330124.433},{"start":6039331834.533,"end":6039332126.433},{"start":6039333836.533,"end":6039334128.433001},{"start":6039335838.533,"end":6039335840.533}]

You can see a live stream by going to http://www.amazon.com/gp/browse.html?node=12774562011&startStreamingOverride=1
Summary: Amazon live streams → Amazon live streams do not load in Firefox 47 on OS X
[Tracking Requested - why for this release]:

This is a regression in Firefox 47 that broken Amazon Video's live event streams. This looks like a regression from bug 1268868 or the backout of bug 1269325.

I bisected this regression to this range:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=369a5ee3a2880a4a98df3a00bf3db8d8f36b181b&tochange=e5a10bc7dac4ee2453d8319165c1f6578203eac7
Depends on: 1268868, 1269325
Oddly, the live stream fails in different Firefox versions on Mac and Windows:

* On Mac, Firefox 46 works, but 47/48/49 fail.
* On Windows, Firefox 46/47/48 work, but 49 fails.

I bisected the 48–49 regression on Windows to this pushlog, but there are no changesets that stand out, other than perhaps the SSE2 optimizations in bug 1271794. It's possible my bisection was inaccurate.

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=a018c36152cec4bcae7c873c6a6e4ad74d082b4b&tochange=9c21202a775c312ec9a013404b409f0a9a1c5b2f
OS: Mac OS X → All
Depends on: 1278019
[Tracking Requested - why for this release]:

The Windows part of this bug a regression in Firefox 49.
Blocks: 1268868, 1269325
No longer depends on: 1268868, 1269325
We had two potentials rounding issue occurring. The one causing the problem is that adding an int64 with a float is a float, and would be limited to 24bits mantissa.
The other, could be that rounding would occur if the segment duration was over 16s long, as that too would exceed the representation range as we using microseconds representation internally.

Review commit: https://reviewboard.mozilla.org/r/57734/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57734/
Attachment #8759921 - Flags: review?(cpeterson)
Comment on attachment 8759921 [details]
Bug 1278005: Fix rounding problems.

Approval Request Comment
[Feature/regressing bug #]: 1278005 
[User impact if declined]: Live streams will not play.
[Describe test coverage new/current, TreeHerder]: Manual test.
[Risks and why]: None, logic is the same, just extending the precision range.
[String/UUID change made/needed]: None
Attachment #8759921 - Flags: approval-mozilla-beta?
Attachment #8759921 - Flags: approval-mozilla-aurora?
Hi Chris, is there a way for more folks to verify this fix? Perhaps have Amazon test this from 6/4 or 6/5 Nightly. I think we should plan another round of full verification (non-DRM, Widevine, Adobe, etc.) on Amazon video streaming before uplifting this to do an RC3 Sunday am PST. I would like to be as cautious as possible. Hope you understand.
Flags: needinfo?(cpeterson)
This will block Fx47 release, also tracked for 48+
Will need to have a nightly including the fix though. Lately it's been taking about 4 days for the changes made on a Friday evening to make it to central. 

I have manually verified the change of course. 

We're only changed from 32 bits arithmetic to 64 bits one on a single operation.
Attachment #8759921 - Flags: review?(cpeterson)
I've reproduced the bug on my Mac with the test URL in bug 1278019. I've also confirmed that a local build with JYA's patch fixes the live stream at the same URL. 

Note that the fix isn't wrapped in a #ifdef MAC and affects all platforms, so we'll need more test coverage here.
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/f3740cfc4b04
Fix rounding problems. r=cpearce, a=tomcat
(In reply to Pulsebot from comment #12)
> Pushed by cbook@mozilla.com:
> https://hg.mozilla.org/mozilla-central/rev/f3740cfc4b04
> Fix rounding problems. r=cpearce, a=tomcat

backed this change out from m-i and direct landed this on mozilla-central to ensure its in today nightly build
Backout by cbook@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/515b840d20aa
Backed out changeset 3538bc8b05aa will land directly on m-c
(In reply to Pulsebot from comment #14)
> Backout by cbook@mozilla.com:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/515b840d20aa
> Backed out changeset 3538bc8b05aa will land directly on m-c

see comment #13 - its now direct on m-c
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
(In reply to Jean-Yves Avenard [:jya] from comment #8)
> Will need to have a nightly including the fix though. Lately it's been
> taking about 4 days for the changes made on a Friday evening to make it to
> central. 
> 
yeah sorry for this, its also because we don't work on weekends and so it can happen a friday change will land on mondays or so with a merge, because we need to make sure there are no other regressions and a pgo build. However in this case i catched this while checking mails today :) so its in any case now on the saturday nightly :)
(In reply to Jet Villegas (:jet) from comment #11)
> I've reproduced the bug on my Mac with the test URL in bug 1278019. I've
> also confirmed that a local build with JYA's patch fixes the live stream at
> the same URL. 
> 
> Note that the fix isn't wrapped in a #ifdef MAC and affects all platforms,
> so we'll need more test coverage here.

You don't want this wrapped in a #ifdef MAC
My guess is that the Windows compiler has a different behaviour than clang/gcc when it comes to floating point. That 49 behaves differently than 48 IMHO indicates the difference between VS2013 vs 2015. That it failed was the proper behaviour and playback working on Windows 47 shows non compliant C++ compilation.
(In reply to Jet Villegas (:jet) from comment #11)
> I've reproduced the bug on my Mac with the test URL in bug 1278019. I've
> also confirmed that a local build with JYA's patch fixes the live stream at
> the same URL. 
> 
This is amazing! :) Thank you so much for the verification Jet. This kind of sanity check on the fix is extremely timely and helpful.
Comment on attachment 8759921 [details]
Bug 1278005: Fix rounding problems.

Based on the release blocking nature, manual verification by Jet and Jean-Yves, let's uplift to Aurora48 and Beta47 so we have an RC3 build ready Monday Romania time for SoftVision (QA) sign offs.

Hi Jean-Yves, CPearce, Wes: Please go ahead and land it on moz-release.
Flags: needinfo?(wkocher)
Flags: needinfo?(jyavenard)
Flags: needinfo?(cpearce)
Attachment #8759921 - Flags: approval-mozilla-release+
Attachment #8759921 - Flags: approval-mozilla-beta?
Attachment #8759921 - Flags: approval-mozilla-aurora?
Attachment #8759921 - Flags: approval-mozilla-aurora+
Hi Florin, this is the only fix (that I know of so far) that will be included in RC3. Could we get some focused testing on video playback on Amazon, Youtube, Netflix, Facebook, Hulu (any other top sites) with various DRM combinations for RC3 sign offs? Thanks!
Flags: needinfo?(florin.mezei)
Uplift landed.
Flags: needinfo?(cpearce)
Can somebody please comment in this bug which specific bug/uplift caused this regression? I wasn't able to tell by looking at the previous comments (perhaps I missed something). Thanks!
In comment 1. There's not a single bug # that caused it.
Flags: needinfo?(jyavenard)
(In reply to Ritu Kothari (:ritu) from comment #24)
> Can somebody please comment in this bug which specific bug/uplift caused
> this regression? I wasn't able to tell by looking at the previous comments
> (perhaps I missed something). Thanks!

For Firefox 47 on OS X, bug 1268868 and/or the backout of bug 1269325. I wasn't able to narrow the regression range any more.

For Firefox 49 on Windows, it looks like SSE2 bug 1271794 and the upgrade to the VS2015 compiler might have made the OS X problem reproducible on Windows.
Flags: needinfo?(cpeterson)
seems the uplift is done, so nothing to do anymore for sheriffs ;)
Flags: needinfo?(wkocher)
(In reply to Ritu Kothari (:ritu) from comment #20)
> Hi Florin, this is the only fix (that I know of so far) that will be
> included in RC3. Could we get some focused testing on video playback on
> Amazon, Youtube, Netflix, Facebook, Hulu (any other top sites) with various
> DRM combinations for RC3 sign offs? Thanks!

Hey Ritu, this was our main focus during the 47.0-build3 sign off. There was nothing out of the ordinary uncovered - testing was performed on: amazon, youtube, netflix, vimeo, yahoo, dailymotion, twitch, youtube gaming, facebook live and hulu, with known issues encountered only (see http://mzl.la/1U9rkc0).

Let us know if there's anything else we can help with here.
Flags: needinfo?(florin.mezei)
(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #28)
> Hey Ritu, this was our main focus during the 47.0-build3 sign off. There was
> nothing out of the ordinary uncovered - testing was performed on: amazon,
> youtube, netflix, vimeo, yahoo, dailymotion, twitch, youtube gaming,
> facebook live and hulu, with known issues encountered only (see
> http://mzl.la/1U9rkc0).
> 
> Let us know if there's anything else we can help with here.

Andrei, can you please add Amazon live streaming to QA's media smoke sites? Who maintains the list of sites to test?

Amazon said we can use the following page to test live streaming:

http://www.amazon.com/gp/browse.html?node=12774562011
Flags: needinfo?(andrei.vaida)
Status: RESOLVED → VERIFIED
(In reply to Chris Peterson [:cpeterson] from comment #29)
> Andrei, can you please add Amazon live streaming to QA's media smoke sites?
> Who maintains the list of sites to test?
> 
> Amazon said we can use the following page to test live streaming:
> 
> http://www.amazon.com/gp/browse.html?node=12774562011

I'll make sure this is added to our media test suites, thanks for the heads-up! Leaving the ni? in place as a reminder.

Updating the list of websites we use in our various test suites is something we're always interested in, to keep our test coverage as relevant as possible. Feel free to ni? me whenever you think there's something specific we should be doing when testing these areas.
Blocks: 1279579
Update: amazon live streaming has been added to our media test suites.
Flags: needinfo?(andrei.vaida)
Version: unspecified → 47 Branch
You need to log in before you can comment on or make changes to this bug.