1425828 - Comcast video won't play (mixed OBJECT_SUBREQUESTS are blocked)

Status: RESOLVED WORKSFORME

(Core :: DOM: Security, defect, P3)

Description

[David Parks [:handyman]]

Steps To Reproduce:

1. Log into XFinity and go to your online DVR recordings at https://tv.xfinity.com/recordings
(this requires an account).
The page uses Flash so it will need to be enabled on xfinity.com.
2. Pick a recording and then click "Watch".

Expected Result:

After a brief spinner with the network logo, the recording starts playing.

Actual Result:

After a brief spinner with the network logo, Comcast shows a generic message about how you can still watch shows On Demand and that "Unfortunately, there's a problem with DVR playback."  There is no technical information in the error messages.

-----------------

mozregression shows that bug 1190623, "Add a pref to treat mixed OBJECT_SUBREQUESTS as active and block them", is when this started to fail.  Setting "security.mixed_content.block_object_subrequest" to false fixes the problem.

Comment 1

[Jim Mathies [:jimm]]

[Tracking Requested - why for this release]:
Regression in media playback from networking bug 1190623

Comment 2

[Tanvi Vyas[:tanvi]]

Does it work if you -
1) Click the lock icon
2) Right arrow into the detailed view
3) Click disable protection


If you could also send a copy of the errors and warnings in the console, that would be great.

I tried to reproduce this on a version of Nightly before 1190623, and I got different results.  I also can't get the video to play, but I get a different error message

Comment 3

[David Parks [:handyman]]

That does work.  Specifically, I try to play the recording, which brings up the error message.  I then temporarily disable protection and it sends me back to the DVR listing -- selecting the video again then plays it properly.

I'll post the logs from the success (ie after disabling protection) and failure (from before) runs.

Comment 4

[David Parks [:handyman]]

Attachment: Log from failed run

Comment 5

[David Parks [:handyman]]

Attachment: Log from successful run

Comment 6

[Tanvi Vyas[:tanvi]]

Can you try one more thing?  Go to about:config and set security.mixed_content.block_object_subrequest to false.  Then visit the page and see if the content loads.

If it does, then this is definitely a mixed object subrequest issue caused by bug 1190623.  The next question is what to do about it...

1) Continue blocking anyway, pushing the web forward.  Attempt to contact comcast and let them know that their mixed content is being blocked and they should fix this before Firefox 59 hits release.
2) Stop blocking object subrequests.
3) Continue blocking, make the UI to unblock more noticeable.
4) Determine if the content being loaded is actually mixed active or mixed passive (i.e. an image).  If mixed passive, see if there is an easy way to determine that and let the content load with degraded UI.

Also, we can see if the content is already available via https.  In which case, auto upgrading would help.

Comment 7

[David Parks [:handyman]]

Changing the setting to false does fix the issue.  Beyond that is going above my pay grade and head.  I'm happy to keep running experiments for you tho.

Comment 8

[Tanvi Vyas[:tanvi]]

Thanks David!

Jonathan, lets discuss and decide how to proceed.

Comment 9

[Jonathan Kingston [:jkt] he/him]

This feels like a webcompat issue personally: 1) from Comment 6.

The other options seem not so great:
2) I don't think we should stop blocking on the first regression here.
3) We could provide a first time onboarding to mixed active content like we do for tracking protection. I don't think anything else will work here.
4) I'm not sure if this will be possible, I also might not be able to debug this at all due to regional issues too.

It looks like they are on the HSTS preload list, I assume that the response however is a third party though? If it isn't however perhaps we aren't rewriting URLs within flash?

Questions:
1. HSTS doesn't ever apply to subresource loads right?
2. Would this instead be fixed if the website sent a Upgrade insecure CSP?
3. Should we consider a UIR header anyway for mixed active content? (perhaps just object subrequests?)
   - I think this would essentially be treating HSTS before blocking which I think Kate mentioned was removed from the standard?

We set out in our timeline to enable this for Firefox 60 stable, I suggest that we delay until this issue has been mitigated but keep enabled so that we can detect other breakage in early beta and nightly.

Comment 10

[Kate McKinley [:kmckinley, :Kate]]

We should probably contact Comcast to update their site / player. We should also remember that Flash is going away in 2020, and that Chrome already blocks much Flash content, including this.

(In reply to Jonathan Kingston [:jkt] from comment #9)
> This feels like a webcompat issue personally: 1) from Comment 6.
> 
> The other options seem not so great:
> 2) I don't think we should stop blocking on the first regression here.
> 3) We could provide a first time onboarding to mixed active content like we
> do for tracking protection. I don't think anything else will work here.

I think this is a bad idea as it opens up a way for a malicious site to degrade connections to mixed-content.

> 4) I'm not sure if this will be possible, I also might not be able to debug
> this at all due to regional issues too.

Not just regional, but you also need a Comcast connection and account.

> 
> It looks like they are on the HSTS preload list, I assume that the response
> however is a third party though? If it isn't however perhaps we aren't
> rewriting URLs within flash?

They may be using a (internal?) CDN not on the preload list, but even so, we process HSTS upgrades after mixed-content blocking. If you change the preference "security.mixed_content.use_hsts" to true, does that allow the content to load?

> 
> Questions:
> 1. HSTS doesn't ever apply to subresource loads right?
> 2. Would this instead be fixed if the website sent a Upgrade insecure CSP?

Yes

> 3. Should we consider a UIR header anyway for mixed active content? (perhaps
> just object subrequests?)

UIR is always evaluated before mixed-content because it is an active signal

>    - I think this would essentially be treating HSTS before blocking which I
> think Kate mentioned was removed from the standard?

It was never part of the standard, that was part of the HSTS priming experiment.

> 
> We set out in our timeline to enable this for Firefox 60 stable, I suggest
> that we delay until this issue has been mitigated but keep enabled so that
> we can detect other breakage in early beta and nightly.

Do we have telemetry on how many requests this represents?

If this were HTML5 video, the <video> and <audio> tags are optionally-blockable (passive) content, and would be allowed.

Comment 11

[Kate McKinley [:kmckinley, :Kate]]

https://theblog.adobe.com/adobe-flash-update/
https://blog.mozilla.org/futurereleases/2017/07/25/firefox-roadmap-flash-end-life/

Comment 12

[Jonathan Kingston [:jkt] he/him]

This sounds like a webcompat issue as Chrome is blocking the flash before it loads in the first place.

I actually would like to move to support whatever stricter blocking Chrome is doing on the Flash file in the first place, do we have more information on that?

Comment 13

[Tanvi Vyas[:tanvi]]

Hi David,
Another request... can you see what happens in Chrome?  Does the content play?  Thanks!

Comment 14

[David Parks [:handyman]]

Attachment: Log from Chrome

There is no issue with playback in Chrome.  Chrome is up to date: version 63.0.3239.108 (64-bit).  Console log is attached.

Comment 15

[David Parks [:handyman]]

Attachment: Log from Chrome

...fix line endings

Comment 16

[Jonathan Kingston [:jkt] he/him]

So it looks like the request URLs that Chrome is complaining about do actually work over HTTPS however they don't serve UIR headers as mentioned.

This type of request would be something we would allow if it were a HTML5 video. However I don't know if it makes sense to try and sniff this content to try and fix this (or if this is even possible).

It seems like we should be speaking with Comcast to fix this issue, as Kate pointed out Flash won't work in 2020.

Comment 17

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Jonathan Kingston [:jkt] from comment #16)
> It seems like we should be speaking with Comcast to fix this issue, as Kate
> pointed out Flash won't work in 2020.

I agree that reaching out to Comcast to fix the problem on their end is the right path forward here.

Comment 18

[Adam Stevenson [:adamopenweb]]

Sorry for the delay. Reaching out to some contacts at Comcast.

Comment 19

[Paul Goldberg]

I tried the latest nightly (January 6 64bit)  and on the Xfinity web site I cannot get and video content to work, I get error messages on live TV, VOD and DVR content. In my email to Adam I submitted the error messages.

Comment 20

[Paul Goldberg]

When I set security.mixed_content.block_object_subrequest to false all Xfinity content (Live, VOD, and DVR) plays properly.

Comment 21

[Christoph Kerschbaumer [:ckerschb]]

Ultimately treating TYPE_OBJECT_SUBREQUEST as either passive or active mixed content depends on our HTTP-Bad plan and also how aggressive we want to push for that change. Putting in the backlog for now because at the moment the change only affects early BETA or earlier versions of Firefox.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

EARLY_BETA_OR_EARLIER has been unset for Fx59 now, so this shouldn't be an issue for that release going forward.

Comment 23

[Julien Cristau [:jcristau]]

This will be disabled by default in 60 once we turn off the EARLY_BETA flag, updating flag by anticipation.

Comment 24

[Chris Peterson [:cpeterson]]

David, does Comcast DVR playback now work in Firefox 61 (Release) and 63 Nightly?

Can we resolve this bug as FIXED or WORKSFORME? Or should we morph this bug into an evangelism bug asking Comcast to fix their mixed content Flash?

Comment 25

[David Parks [:handyman]]

The situation in nightly is bad but things are working in release.  I don't _think_ any of the nightly issues are about content but I'm not sure.  The plugin nightly sandbox is tighter than release ("restricting SIDs") so that could be responsible.  The door hanger in the awesome bar says "No blockable content detected on this page."

In release (in a new profile) I can:

* Go to https://www.xfinity.com/stream .  Click "Sign In" and go through the signin process.
* The page shows a spinner and says "Adding Device", then the main page comes up.
* Click a recording.  It switches to the video HUD.
* The video starts playing.

In today's nightly:

* Go to https://www.xfinity.com/stream .  Click "Sign In" and go through the signin process.
* The page shows a spinner and says "Adding Device", then a "Lets get your Flash player up and running" dialog requesting to allow Flash.  After clicking to allow Flash, it just sits on that same dialog screen.  Clicking reload repeats the process (starting from "Adding Device")
* From there, I closed Nightly, opened release with the same profile, and was able to move past the Flash page.
* Switching back to Nightly, I can now get into XFinity.  So I tried clicking a recording.  It switches to the video HUD.
* The video never starts playing.

Comment 26

[Tom Kloos]

I experience the same problems as David describes.  The FF 63 nightly versions fail to play video and have the same sign-in issue (sign-in is fixed by running 62 beta on the same profile then back to 63 alpha, but not video).  Everything appears to work correctly with FF 61 and the FF 62 release candidate.  63 is broken (or Xfinity is...  :-) )

Comment 27

[Tom Kloos]

The value of "security.mixed_content.block_object_subrequest" no longer has any affect on the problem for me.  firefox-63.0a1.en-US.win64_20180830220110 doesn't work and given that changing this pref previously did allow 63 to work means some change broke something else.

From previous experience, getting Comcast to fix/improve something is next to impossible unless it's some benefit to them.

Opera 55.0.2994.44 and PaleMoon 28.0 both work without issue.

Comment 28

[Tom Kloos]

Changing dom.ipc.plugins.sandbox-level.flash from default 3 to 2 fixes the observed "problems" in the current FF63 nightly.  After this config change, 63 nightly performs the same as 61 and 62, including a functioning sign-on.  The default value in the released FF61 is 2, but in the release candidate for FF62 it's 3.  Is this expected given that FF62 works?

Comment 29

[Chris Peterson [:cpeterson]]

(In reply to Tom Kloos from comment #28)
> Changing dom.ipc.plugins.sandbox-level.flash from default 3 to 2 fixes the
> observed "problems" in the current FF63 nightly.  After this config change,
> 63 nightly performs the same as 61 and 62, including a functioning sign-on. 
> The default value in the released FF61 is 2, but in the release candidate
> for FF62 it's 3.  Is this expected given that FF62 works?

@ David and Jim, it looks like Windows sandbox level 3 (bug 1366256) is breaking Xfinity video in Firefox 62.0 RC! Is this a known issue? 62.0 will be released tomorrow, September 5.

@ Tom, what Flash Player version are you running?

Comment 30

[Julien Cristau [:jcristau]]

Ryan fyi

Comment 31

[Paul Goldberg]

With Firefox 62 using the release version of Flash Player Xfinity Stream runs without any problems.  I tried live TV, VOD and recorded content.  The only minor problem I have is an initial stuttering which last for about 10 to 15 seconds before the content plays smoothly.  This does not happen using Google Chrome.

Comment 32

[Chris Peterson [:cpeterson]]

Jim filed bug 1488439 to investigate the Windows sandbox level 3 issue. This bug is about the "security.mixed_content.block_object_subrequest" issue (which is not currently a problem because the pref is false).

Comment 33

[Tom Kloos]

I'm running flash player 30.0.0.154 which I believe is current.  62 beta and 62 RC have always worked fine for me.

63 alpha (nightly) is where I first saw the problem and as recently as when I made comment 28.  However, today with 63 now beta, all is good with the default values for dom.ipc.plugins.sandbox-level.flash;3 and security.mixed_content.block_object_subrequest;false in firefox-63.0b3_20180904170936.

However, something strange is happening with alpha nightly versus beta builds that I download.  64 alpha, now the nightly, doesn't work unless I set dom.ipc.plugins.sandbox-level.flash to 2.  This is with firefox-64.0a1.en-US.win64_20180904182536.

Comment 34

[Tom Kloos]

Ah ha!   Bug 1488439 explains why I see different results between nightly and beta/RC.  Sorry, I should have read that bug first.   62.0 and 63 beta are "works for me".

Comment 36

[Tom Kloos]

Confirmed now fixed with firefox-65.0b8, both win64 and win32, under Windows 7.  See also bug 1505482.

Comment 37

[Frederik Braun [:freddy]]

I think we can safely close this bug at this point. It was filed 9 years ago and I am willing to guess that the xfinity dvr player does not use flash anymore. It also looks as if comment 36 already confirmed at as working again.