Closed Bug 841357 Opened 11 years ago Closed 6 years ago

Use a different user-agent from mobile when requesting videos

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: henry.fai.hang.chan, Unassigned)

Details

For the Android stock browser (and also Dolphin Browser), the user-agent "stagefright/1.1 (Linux;Android 2.3.6)" is used instead of the default webkit string when fetching media objects.  The stock browser (Safari) on iOS uses AppleCoreMedia user-agent as well.  However, Firefox does not exhibit this behavior.

This causes a problem on Smartone (a ISP in Hong Kong).  Currently the ISP relies on this behavior to serve video content properly.  If stagefright (or AppleCoreMedia) is detected in the user-agent string, the video is served without transcoding.  However, if a browser user-agent string is detected, it will serve up and HTML file containing a meta redirect to a 3gp file (which will transcode the requested video file on demand).

This feature is marketed to save bandwidth by recompressing video streams.  Normally this is not apparent as they replace the flash players by links to the 3gp streams as well, but they do not shim the HTML5 Video tag.  Obviously Firefox is supposed to choke on the HTML file.

Since other mobile browsers send different user-agents when fetching media (Android and iOS), and it will take ages to resolve this on the ISP side (I don't think shimming HTML5 video tags are a great idea), maybe Firefox can follow suit with the other browsers, so ISPs can similarly sniff when not to intercept and recompress video streams.
does Firefox use these frameworks?
Yes firefox uses stagefright too, but not using it to fetch the media file.  Firefox should probably spoof the user-agent as stagefright when requesting media objects so ISPs will not erronously intercept the request and replace it with a meta redirect?

(Same goes with Safari on iPhone and the stock browser)
Hi Henry,

Help me to understand this problem better. The ISP detects these strange user agent strings and so serves... what format, exactly? And is that format a format which Firefox OS can decode and play natively?

You say it serves a 3gp file to desktop user agents - is that right? If I read about 3gp, it tells me it's a mobile standard, and I'm fairly sure desktop Firefox can't play it. So that confuses me. Also, you say that the 3gp file is transcoded on the fly. What is it transcoded from and what is it transcoded to? Are either of those formats things that Firefox OS can play natively? 

How does the ISP save bandwidth? Is it because the file on the server is in high resolution and if it detects a mobile device, it sends a lower resolution version? Do we actually _want_ the lower-resolution version?

Gerv
Sorry for my bad English. 
For Firefox video tag 
1. A video/mp4 file is requested.
2. ISP detects Mozilla/5.0 useragent. 
3. ISP sends text/html, content is meta redirect to 3gp file. 
4. Firefox does not play the HTML file (and not supposed to).
For stock/dolphin/iPhone using video tag
1. A video/mp4 file is requested. 
2. ISP cannot detect Mozilla useragent (stagefright or applecoremedia.is sent)
3. ISP serves video/mp4. 
For stock/dolphin/iPhone/Firefox on direct file avcess
1. A video/mp4 file requested. 
2. Mozilla useragent detected. 
3. Text/HTML served. 
4. Browser launches video player app.

Yes, the 3gp file is lower bitrate. 

Solutions: 
1 ask the ISP stop sending text/html for video tags. 
Problem: requires coordination and a method to detect if the file was requested via video tag or direct. 
2. Ask the ISP to replace the video tag with a direct link to the 3gp file (it does this for flash).
problem: requires coordination and very likely to be fragile code. Missing nodes will break JavaScript code. yuck user experience. 
3. Ask the server send application / octet-stream instead of video/mp4. 
Problems: that doesn't sound a great idea to implement on every server. Firefox loses ability to detect if it can play the file. 
4. Use the stagefright useragent when using stagefright (video tag)
Problems: means that we will have yucky code.

Though my suggestion is use option 4.

Ps: this behavior applies to every video/ mime type, including ogg and mkv. They transcode  everything
I meant that video/ogg, video/mp4 etc are all transcoded to 3gp. The main problem is Firefox receiving an HTML file which Firefox doesn't and shouldn't parse. The details about 3gp and stuff are irrelevant  but just a description of their technology.
Any thoughts on this from the video team?
Henry: thanks for the additional info. So you are saying that if the markup is:

<video width="320" height="240" controls>
  <source src="movie.mp4" type="video/mp4">
</video>  

and Firefox for Android requests "movie.mp4", the site is sending back an HTML page instead of the actual movie? That sounds like the site it broken to me.

Changing our UA for a large class of loads would have an entirely unknown effect on compatibility across the web. We've told lots of people what our UA is; as we've found, changing it randomly does not go down well. 

The fact that we use libstagefright in some cases is an implementation detail which should not be depended on by sites.

Gerv
This is being done by a major mobile ISP, not some website. They aren't relying on some changing UA, they are automatically intercepting all page requests to a video file into HTML files that Meta redirect to lower nitrate 3gp files. This works wonders if I click a link to a mp4 file, e.g. on Facebook. 

Since Firefox sends the same UA for any request, their tech kicks in at the wrong time. 

Firefox for mobile should just mimic what others do for compat.  -or- provide a way to tell the ISP when not to send back text/html when it is really really expecting video/ogg. 

This problem can be worked around by sites to send application / octrt-stream. This is suboptimal and also irrelevant to the bug itself but exists as info for other webmasters who experience the same problem. The problem itself can be reproduced over every single website that uses html5 to load video/ogg or mp4 as long as your 3g data connection is provided by This ISP.
Oh, I understand better now. Sort of.

Why does the ISP want to send a different video file if the video is viewed using the <video> tag compared to if it is viewed by directly accessing the URL?

I think this is fundamentally broken. If a UA requests a video file, sending back an HTML page which meta-redirects to a video file is just wrong. There are ways at the HTTP level to do this, for a start - the Location header. That sort of redirect would work.

Whether the video file is shown by the browser or opened in an app is none of the ISP's business - that should be determined by the user, via their choice of browser (and its capabilities) and via configuring what it does with files it doesn't understand, and by the ISP sending the correct Content-Type.

If we start claiming we are libstagefright (I assume that's what you are suggesting? We aren't going to claim we are AppleCoreMedia) then sites may, understandably, start assuming that Firefox OS supports every video and audio codec standard that Android does. Is that even true?

If this ISP is claiming that this interference with a customer's requests serves the customers, then they are responsible for making it work. And if they can't make it work, they should stop doing it. It's not like it's impossible to tell when Firefox for Android requests a video file - we make an HTTP request, with our user agent, for a file which is served as one of a certain set of content types.

Gerv
> Why does the ISP want to send a different video file if the video is viewed 
> using the <video> tag compared to if it is viewed by directly accessing the URL?
Mobile bandwidth is precious.  And most probably, if I were on the subway, I'd want a smooth lower-res video instead of getting a high quality choppy one.

> I think this is fundamentally broken. If a UA requests a video file, sending 
> back an HTML page which meta-redirects to a video file is just wrong. There are 
> ways at the HTTP level to do this, for a start - the Location header. That sort 
> of redirect would work.

Scenario A.
User A clicks on a link, www.example.org/video.ogg.  Firefox receives a video file and launches the video player app.  During playback, due to platform limitations, there is not enough memory.  Firefox is quit by Android.  User A finishes watching the video.  User A presses the back button and returns to Firefox.  Firefox restores the session and viola the video app is launched.

Scenario B.
User A clicks on a link, www.example.org/video.ogg.  Firefox receives a Location: response to a low bitrate media file and launches the video player app.  During playback, due to platform limitations, there is not enough memory.  Firefox is quit by Android.  User A presses the back button and returns to Firefox.  Firefox restores the session and viola the video app is launched.

Scenario C:
User A clicks on a link, www.example.org/video.ogg.  Firefox receives a text/html response and launches the video player app on a meta redirect. The page also contains some JavaScript to navigate backwards in 1 second after the meta redirect has happened.  Hopefully (and with my experience, always) Firefox navigates backwards to the original page before Firefox is killed by Android due to low memory.  User A finishes watching the video.  User A presses the back button and returns to Firefox.  Firefox restores the session and he is left on the page where he clicked the link.

Obviously Scenario A and B leave the user severely frustrated if he cannot press the back button fast enough.  Scenario C is superior if the ISP only intercepts direct file access / pressing links and not intercepting the request if the browser explicitly expects a video file (via HTML5 video).  Firefox does not give any way to the ISP to distinguish when to intercept and when not.

> Whether the video file is shown by the browser or opened in an app is none of 
> the ISP's business

That's not even the main problem.  On a direct file access, either the file starts a download OR the video app launches anyway.

> - that should be determined by the user, via their choice
> of browser (and its capabilities) and via configuring what it does with files it 
> doesn't understand, and by the ISP sending the correct Content-Type.

I have no idea what you're talking about.  So you mean the ISP should insert a "Low Bandwidth" link after every link to a video file on a page, so the user can choose to play low-bitrate file or high-bitrate file?  That means the ISP would need to prefetch all links to make sure they are video/ files, and generate the correct HTML?  This is not even feasible to implement at the ISP level!

Furthermore, the ISP sends a Content-type:text/html correctly.

And you missed the point completely.  This isn't some kind of shim so ogg and webm plays on the iPhone.  That is only the side effect, not it's main function.  The main function is to transcode the video into 3gp so that there is smoother playback for the user when bandwidth is limited.

> If we start claiming we are libstagefright then sites 
> may, understandably, start assuming that Firefox OS supports every video 
> and audio codec standard that Android does. Is that even true?
This bug isn't even about Firefox OS and I have never mentioned anything about Firefox OS and I do not understand why you're rambling on Firefox OS and sound like this is going to kill the purity on Firefox OS.
Firefox for Android uses libstagefright to play and will play whatever libstagefright will play (every single video and audio codec) and thus should send what other browsers send when they use libstagefright (in-browser videos).

> If this ISP is claiming that this interference with a customer's requests 
> serves the customers, then they are responsible for
> making it work. And if they can't make it work, they should stop doing it.
How can they make it work?  How can they detect when to shim it or not?

> It's not like 
> it's impossible to tell when Firefox for Android requests a video file - 
> we make an HTTP request, with our user agent, for a file which is served 
> as one of a certain set of content types.
Now this tells me exactly you haven't even gotten round to clearly understood what I was saying.  My apologies if I'm still unclear.

==Case 1==
For all browsers on Android / iPhone
(1) Mozilla/5.0 is sent when the file is requested directly (via the URL bar).
(2) The ISP receives the request, and passes the request onto the server.
(3) The server returns a file with a Content-type: video/ogg.
(4) The ISP sends back a html file with THE CORRECT content-type: text/html.
(5) The browser follows the meta redirect and launches the app and navigates backwards, preventing the frustrating behavior mentioned above.

==Case 2==
For other browsers on Android / iPhone
(1) When a HTML5 video is played, the fetching of the video file is done VIA THE PLATFORM's LIBRARY ITSELF.
(2) The ISP sees the request is not made by the browser, and does not change any of its contents, i.e. passes back the high bitrate media file directly.
(3) The browser plays.

For Firefox on Android
(1) When a HTML5 video is played, Firefox takes care of the media file fetching.
(2) The ISP gets a Mozilla/5.0 useragent, incorrectly assumes it can intercept the request and send back a text/html.
(3) Firefox doesn't parse the text/html file (correct behavior).

In my opinion, I am in favor of the approaches of the ISP for the following advantages:
(1) intercepting the request and serving a lower bitrate media file to facilitate smoother playback, and
(2) using the meta redirect + JavaScript prevents that awful restore-relaunch loop due to hardware/platform/browser limitations

== Solution A ==
The ISP needs a way to know when the browser CANNOT accept text/html.  Firefox can spoof it's user-agent, or defer the network request back to the platform media library itself.

== Solution B ==
Evangelize the ISP and persuade them it's not right to intercept requests based on MIME type and User-agent.  Meanwhile, browsers that defer the network requests to the platform library when using in-app players will continue to work smoothly, while Firefox for Android will continue to choke in Hong Kong for quite some users.




I'm a web developer, Hong Kong is my main user base and quite a few number of my classmates are using SmarTone.  I am ok with detecting requests via this ISP and spoofing my video/mp4 files' MIME to application/octet-stream, but not every web developer is willing to take this workaround.

Try telling SmarTone users that HTML5 videos don't work because Firefox for Android refuses to be like other major mobile browsers and tell the ISP that it's currently a media player and not a browser.

Some additional statistics: SmarTone is one of five major ISPs in Hong Kong, there are 7 million citizens and each citizen has an average of more than 1 mobile phone, and nearly every single secondary school graduate will have a mobile phone capable of accessing the internet via 3g.  The only thing is that the market share of Firefox for Android in Hong Kong is relatively small.
Hi Henry,

I hope we are groping our way towards understanding :-)

(In reply to henryfhchan from comment #11)
> > Why does the ISP want to send a different video file if the video is viewed 
> > using the <video> tag compared to if it is viewed by directly accessing the URL?
>
> Mobile bandwidth is precious.  And most probably, if I were on the subway,
> I'd want a smooth lower-res video instead of getting a high quality choppy
> one.

But that's not my question. I'm not asking "why would an ISP want to send a lower-bandwidth video file to a mobile device?" I'm asking: "why would an ISP want to send different video files to the same device, depending on whether the website used the <video> tag or whether the user accessed the video's URL directly from the URL bar?"

> Scenario A.
> User A clicks on a link, www.example.org/video.ogg.  Firefox receives a
> video file and launches the video player app.  During playback, due to
> platform limitations, there is not enough memory.  Firefox is quit by
> Android.  User A finishes watching the video.  User A presses the back
> button and returns to Firefox.  Firefox restores the session and viola the
> video app is launched.

Except that if it does that, it's a bug - if we've already passed a URL off to an external app once, we shouldn't do it again, even on session restore. If you see this happening, file a bug.

Sites should not have to do silly things like your scenario C to work around deficiencies in Firefox; we should fix Firefox.

> Firefox does not give any way to the ISP to distinguish when to intercept
> and when not.

There is no need for the ISP to know. Why not just _always_ transcode the file if the user is on a mobile device?

> > - that should be determined by the user, via their choice
> > of browser (and its capabilities) and via configuring what it does with files it 
> > doesn't understand, and by the ISP sending the correct Content-Type.
> 
> I have no idea what you're talking about.  So you mean the ISP should insert
> a "Low Bandwidth" link after every link to a video file on a page, so the
> user can choose to play low-bitrate file or high-bitrate file?  

No. 

> Furthermore, the ISP sends a Content-type:text/html correctly.

Some ISPs (e.g. T-Mobile here in the UK) manage to implement transparent image compression, without caring whether the image is requested from an <img> tag or directly requested. So I'm sure it's possible to do the same for video.

> > If we start claiming we are libstagefright then sites 
> > may, understandably, start assuming that Firefox OS supports every video 
> > and audio codec standard that Android does. Is that even true?
> This bug isn't even about Firefox OS and I have never mentioned anything
> about Firefox OS and I do not understand why you're rambling on Firefox OS
> and sound like this is going to kill the purity on Firefox OS.

Sorry; I got mixed up there.

However, Firefox OS is relevant. If this system is also going to break on Firefox OS, then that's an additional incentive to get them to fix it rather than changing Firefox for Android.

> > If this ISP is claiming that this interference with a customer's requests 
> > serves the customers, then they are responsible for
> > making it work. And if they can't make it work, they should stop doing it.
>
> How can they make it work?  How can they detect when to shim it or not?

You say this is about bandwidth saving for mobile devices. So why do they not _always_ transcode the video?

> == Solution A ==
> The ISP needs a way to know when the browser CANNOT accept text/html. 
> Firefox can spoof it's user-agent, or defer the network request back to the
> platform media library itself.

_Now_ you are asking the right questions.

Here is an HTTP request that Firefox (for desktop; I assume Android is the same) makes when requesting a video file as part of the <video> element:

GET /html5/videos/big_buck_bunny.ogv HTTP/1.1
Host: www.quirksmode.org
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:22.0) Gecko/20130306 Firefox/22.0
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
DNT: 1
Range: bytes=32768-
Referer: http://www.quirksmode.org/html5/tests/video.html
Connection: keep-alive

Notice the _lack_ of "text/html" in the Accept: header. If you want to know when you can't send text/html, there's your answer.

Gerv
cc Cathy Zhou, who may be able to help with outreach for Smartone.
Hi,

Is this issue still reproducible?

Thank you!
Flags: needinfo?(henry.fai.hang.chan)
I no longer use X-Power with Smartone, so cannot reproduce.
Flags: needinfo?(henry.fai.hang.chan)
Taking into consideration Comment 15, I will close this issue as Incomplete.

If the reporter (or anyone else) can provide more information, please feel free to reopen or comment on the issue, and we'll have a thorough look on it.
Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.