Closed Bug 886249 Opened 11 years ago Closed 11 years ago

[A/V] Some youtube contents cannot be played

Categories

(Firefox OS Graveyard :: Gaia::Video, defect, P1)

Product:
Firefox OS Graveyard
Component:
Gaia::Video
Platform:
ARM
Gonk (Firefox OS)
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX
Milestone:
1.1 QE3 (26jun)

People

(Reporter: leo.bugzilla.gecko, Unassigned)

Details

Attachments

(1 file)

patch for this bug
87 bytes, text/html
daleharvey
: review-
Details 
I cannot figure out what kind of video are denied from the server.
However, there are several videos which is not played in youtube.
They are played well before, so i am trying to find out from when it cannot be played.

Please test with this url

http://www.youtube.com/watch?v=wq7ftOZBy0E

Then, it got forbidden response from the server.

HTTP/1.1 403 Forbidden\r\n
Expert Info (Chat/Sequence): HTTP/1.1 403 Forbidden\r\n
Request Version: HTTP/1.1
Status Code: 403
Response Phrase: Forbidden
blocking-b2g: --- → leo+
Priority: -- → P1
Target Milestone: --- → 1.1 QE3 (24jun)
hmm...It's weird.

I have checked same video file several times.
But I cannot play it by any version which is released in June.
Component: General → Gaia::Video
David - Why are we getting a 403 here? Any ideas?
Flags: needinfo?(dflanagan)
I haven't seen that before, and have no idea why youtube would be sending 403.  To be honest, I don't really understand much of our youtube code. I've done some work on it, but the original code was apparently copied from Firefox for Android.

Leo: it seems silly to assign a deadline of 6/24 to a bug that was filed on 6/24.
Flags: needinfo?(dflanagan)
When I try to reproduce using the URL in comment 0, I get a message in the video app: "Video cannot be played: the format is unsupported or the file is invalid"

Dominic changed the video app recently to give better error messages when invoked with invalid videos.  This was to support videos transferred via MMS, but it is also being triggered here.

In any case, I don't see a 403 error that the reporter sees.

Leo: have you been doing many, many youtube tests with the same IP address? I wonder if youtube limits the number of requests from a single source to prevent copying of its video library. Could that be where the 403 is coming from?
(In reply to David Flanagan [:djf] from comment #5)
> When I try to reproduce using the URL in comment 0, I get a message in the
> video app: "Video cannot be played: the format is unsupported or the file is
> invalid"
> 
> Dominic changed the video app recently to give better error messages when
> invoked with invalid videos.  This was to support videos transferred via
> MMS, but it is also being triggered here.
> 
> In any case, I don't see a 403 error that the reporter sees.
> 
> Leo: have you been doing many, many youtube tests with the same IP address?
> I wonder if youtube limits the number of requests from a single source to
> prevent copying of its video library. Could that be where the 403 is coming
> from?

Did you mean that youtube server blocks all requests from leo devices to specific file because it sends too many requests from same IP address? Have you heard about same symptom?
Hi all,

I also investigate this bug yesterday. I got the 403 error from console through adb, which is printed by video element and looks like this:

E/GeckoConsole( 8749): [JavaScript Warning: "HTTP load failed with status 403. Load of media resource http://r13---sn-nwj7kned.c.youtube.com/videoplayback?expire=1372097039&sver=3&itag=18&id=c2aedfb4e641cb41&gcr=tw&cp=U0hWR1BUVV9IU0NONl9ITllBOmY0dHp5ZmFsS014&upn=7zQt7Rm1b6M&mt=1372072044&sparams=cp%2Cgcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&mv=m&ms=au&source=youtube&fexp=916903%2C921713%2C921047%2C916914%2C928201%2C901208%2C929123%2C929915%2C929906%2C929907%2C929125%2C925714%2C929917%2C929919%2C931202%2C912512%2C912515%2C912521%2C906838%2C904488%2C906840%2C931910%2C931913%2C932227%2C904830%2C919373%2C933701%2C904122%2C926403%2C909421%2C912711%2C935102&newshard=yes&ipbits=8&ratebypass=yes&ip=220.136.8.223&key=yt1&signature= failed." {file: "app://video.gaiamobile.org/view.html" line: 0}]


To find out the source of this bug, I do the procedure of youtube.js hand by hand. I found the signature is not correct[1], so the retrieval of video is failed. If I supply the correct one, the video can be played on my firefox browser. The line 155 uses bestStream.sig to get the signature, but it doesn't have this param in this video. The place I found signature is bestStream.s. I had tested the changed code with bestStream.s, but it did not work.

[1]https://github.com/mozilla-b2g/gaia/blob/a013abc82b6b82c1afddad236c1cee2f2ac72859/apps/video/js/youtube.js#L155
Hi all, 

I dump every stream and try all of it by hand. I found only the webm sources work, like:
 video/webm;+codecs="vp8.0,+vorbis" with 1080p, 720p, large, medium. And they can browse without signature, and others, mp4 or flv, are all failed.

I think it may be related to the change of protocol of youtube.
I had tested two firefox extensions and a youtube analysis website. All of them can't provide mp4, flv url for download on this video.

It is possible to change the priority of formats to make itag '43' // 360p WebM higher than itag '18' // 360p H.264 to make this video play. But the performance may not as good as h.264. And I had tried this small patch with different youtube video. Some of them are still failed to load with itag 43.
Assignee: nobody → johu
Leo, 

In comment 2, you can't play this SNSD video at all builds of june. Have you tried builds of May?? I had tried it, builds of may can't play this video, too. I guess this may be the protocol change of youtube. Can you confirm that??
Flags: needinfo?(leo.bugzilla.gecko)
Yes, right.

I have tested the version in last week on May, because it's the last successful play in my memory. (I have not checked the video from then)
It doesn't work by that version, also.
Flags: needinfo?(leo.bugzilla.gecko)
Thanks Leo, 

I had checked the code of youtube.js. It is not modified since its first version, except one xhr settings change. I don't think it is a IP blocking behavior from youtube as comment 6. It is more possible a protocol change from youtube.
All, I found a bug before about the YouTube API we use: bug 843334. It states we are using unofficial YouTube API to find out the h.264 url, which is the itag 18, for playing.

After some investigation, I found there is a type of youtube video, named Vevo[1], which are licensed music and video. To download these vevo movies need to download the original desktop youtube page and parse the partial JavaScript part of that page. If we look at the code of youtube-dl[2], we may find they use the url_encoded_fmt_stream_map from webpage, instead of get_video_info. If a video is included by vevo group, YouTube will change the policy to deal with that video. This video is the vevo video.

If we really need to play this kind of video, we can implement a vevo parser like youtube-dl. That may be a huge patch. And the kind of behavior may not be legal to YouTube. So, I can't do anything about this bug.

[1] http://en.wikipedia.org/wiki/Vevo
[2] https://github.com/rg3/youtube-dl/blob/b004821fa98a5ac563811f37b744c632dd58d559/youtube_dl/extractor/youtube.py#L486
Assignee: johu → nobody
Attached file patch for this bug — 
David, 

I know you are not familiar with YouTube protocol. But I don't know who can review this bug you.

About this patch:
1. downloads youtube webpage text from desktop version
2. check vevo video and use url_encoded_fmt_stream_map from desktop webpage text
3. add decryptSignature method to decrypt signature with 87 characters.

Although I think it may not legal to parse vevo video and play them in video app, I did the patch for this bug. It is a little huge, but works. If you feel this is not improper, please r- this path.
Attachment #767671 - Flags: review?(dflanagan)
Comment on attachment 767671 [details]
patch for this bug

Change the reviewer to dale because we are in the same timezone now.
Attachment #767671 - Flags: review?(dflanagan) → review?(dale)
Sorry, I found a typo here: If you feel this is improper, please r- this patch.
I had updated the PR to support more types of signature from 82 to 88 which is synced with youtube-dl.
Comment on attachment 767671 [details]
patch for this bug

This is a really good patch, I dont have any issues with the code but I am definitely not comfortable pushing this, we really need to get the partner position clarified and to start using the official API. As you mentioned this may be against TOS, will r- this and set it blocking on the official api bug
Attachment #767671 - Flags: review?(dale) → review-
(In reply to Dale Harvey (:daleharvey) from comment #18)
> Comment on attachment 767671 [details]
> patch for this bug
> 
> This is a really good patch, I dont have any issues with the code but I am
> definitely not comfortable pushing this, we really need to get the partner
> position clarified and to start using the official API. As you mentioned
> this may be against TOS, will r- this and set it blocking on the official
> api bug

YouTube already expressed that they don't like that we're rendering YouTube videos in the Gaia Video app. The current 1.1 plan to resolve this is to get YouTube to serve us h264 content directly on the YouTube mobile site under a Firefox OS user agent. That will plan will go into execution if we pass YouTube certification. If and when this does happens, we'll be able to remove all of this youtube.js hack code we've built. If that plan fails, we'll have to rethink what to do here.

I'm also renoming on the basis of that rationale and comment 18 to minus.
blocking-b2g: leo+ → leo?
Ah ha!  I just saw this on twitter: 

Jamie Zawinski ‏@jwz 26 Jun
Youtube download counter-countermeasures applied. I have worked around Youtube's latest obfuscation. You can again... http://jwz.org/b/yhpm 
Retweeted by BrendanEich 

JWZ says that youtube has changed the way we're supposed to fetch the raw video. He's got code at his site. We really need to get out of the business of doing videos this way, but until we can, maybe jwz's blog post will tell us what we need to change in apps/video/js/youtube.js to make it work again.  

I wonder if we're experiencing similar brokenness in Firefox for Android, from which that youtube.js code origininated (I think).
Heh. I didn't see we already had a patch for this. It looks like John and JWZ both found the same decyphering code.
I agree with Dale's r- here. If youtube is sending 403 Forbidden, we shouldn't go trying to circumvent that.  

We already have code for displaying an error message for other sorts of videos that youtube doesn't want us to download. We should display a useful error message in this case, too.  I wonder if the 403 response page includes any text we can use...
Although I don't like to discard license checking, I already did it at this PR. I used the procedure from youtube-dl, an open source project written in python. So, it is python-dl and JWZ found the decrypting code. If applying this patch (attachment 767671 [details]), fxos plays all youtube videos and doesn't care anything about license staff.

I think Firefox for Android should be also affected by this change from YouTube. I don't know where you cut youtube.js from. Maybe, I can do the check for that.
David,

I also agree with Dale's r-. I send this patch because I don't know the position of us to deal with this kind of staff. And we can't extract any useful text from 403, it just response 403 but nothing.
The status is changed.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
blocking-b2g: leo? → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: