410904 - http_referer request not being sent with embedded flash

Status: VERIFIED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Xavier K]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11

Let me start of by saying this is not a duplicate of bug 337766.  His request was geared towards http_referer being manually inserted into the flash object.  

I have link protection enabled on my website to protect copyrighted images from being downloaded. I also have an embedded flash based slideshow that displays the images inside the browser. The http_referer seems to be handled by Internet Explorer, so the request for the image is processed. This is not the case with Firefox. The request is being denied due to a non-existing/non-valid http_referer. I have had others test it with Camino and the problem seems to exist there as well. They have reported that it also works in Safari.  I have tested this by allowing non-existing referers (Direct URL requests), and it works.  The slideshow users the LoadMovie() object inside of Flash.

Affected Domain Logs:
__________________________________________________
1.2.3.4 - - [05/Jan/2008:02:30:55 -0600] "GET /1.jpg HTTP/1.1" 403 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11"
__________________________________________________

I also have an open forum located here: http://forums.mozillazine.org/viewtopic.php?t=617988.  Additionally, I have contact jmott from Macromedia and requested his involvement, if for nothing else, to confirm my request.



Thank you,

Xavier

Reproducible: Always

Steps to Reproduce:
1.  Go to http://test.dlsgallery.com for a non-working model
2.  Go to http://test2.dlsgallery.com for a working model
3.
Actual Results:  
Http_referer is not being sent to the webserver, causing a deny in request.

Expected Results:  
http_referer being returned to the webserver as an embedded object

Here is a copy of the htaccess file that is on test.dlsgallery.com that is affecting the slideshow

    RewriteEngine on
    RewriteCond %{HTTP_REFERER} !^http://www.test.dlsgallery.com/.*$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://www.test.dlsgallery.com$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://test.dlsgallery.com/.*$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://test.dlsgallery.com$      [NC]
    RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ - [F,NC]

Here is a copy of the htaccess on test2 that allows the slideshow to work

    RewriteEngine on
    RewriteCond %{HTTP_REFERER} !^$
    RewriteCond %{HTTP_REFERER} !^http://www.test2.dlsgallery.com/.*$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://www.test2.dlsgallery.com$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://test2.dlsgallery.com/.*$      [NC]
    RewriteCond %{HTTP_REFERER} !^http://test2.dlsgallery.com$      [NC]
    RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ - [F,NC]

Comment 2

[Xavier K]

(In reply to comment #1)
> 
> *** This bug has been marked as a duplicate of bug 337766 ***
> 

As I stated, this is not a duplicate of bug 337766.

Comment 3

[Dave Townsend [:mossop]]

Yes it is, See comment 5 where it states that they do not set the referer header for flash requests for security reasons.

Comment 4

[Xavier K]

They do not, that doesn't mean Firefox shouldn't.  This is an embedded object that is making the request.  Firefox should process that internally recognizing it is not a unique request, and attach the necessary code.

This is a major bug that interferes with Firefox's ability to support what other browsers do.

Macromedia, may not support the header, but that doesn't mean Firefox shouldn't.  It is required for compatibility.  Otherwise people will be forced to use other browsers such as IE, where it works perfectly.  I am to assume then that Microsoft took the time to modify their code to support the http_referer around an embedded object.



Xavier

Comment 7

[Reed Loden [:reed]]

Can you explain one more time why you don't think this is a dupe of bug 337766?

Comment 8

[Xavier K]

Jeff's request was that they were trying to pass internal data from within flash to Firefox.  My request is the interaction between the server and the browser.  To detect and append the necessary information to complete a valid request from the server side.  Adobe Flash has no way to read the referer information, thus it can't return it.

They don't support it, because what would stop me from opening flash, creating a script that embedded a false http_referer so I can spoof a request from a website.    As you can see from this snipet of log, it is firefox that is making the end request to the webserver.  It isn't Flash making the request.  Everything until this request has the http_referer embedded.

Affected Domain Logs:
__________________________________________________
1.2.3.4 - - [05/Jan/2008:02:30:55 -0600] "GET /1.jpg HTTP/1.1" 403 - "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127
Firefox/2.0.0.11"
__________________________________________________

Here is a copy of the log from a request in an IE browser.
__________________________________________________

1.2.3.4 - - [05/Jan/2008:06:25:35 -0600] "GET /1.jpg HTTP/1.1" 200 62205 "http://test.dlsgallery.com/test.swf" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; Windows-Media-Player/10.00.00.3990; InfoPath.2; .NET CLR 3.0.04506.648)"
1.2.1.4 - - [05/Jan/2008:06:25:42 -0600] "GET /2.jpg HTTP/1.1" 200 69567 "http://test.dlsgallery.com/test.swf" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; Windows-Media-Player/10.00.00.3990; InfoPath.2; .NET CLR 3.0.04506.648)"
__________________________________________________


Xavier

Comment 9

[Matthias Versen [:Matti]]

Does Opera send a referer because they are also using the npapi plugin ?
Are other plugins send a referer like for example quicktime playing a mp3 ?

Comment 10

[Xavier K]

I just downloaded and installed Opera.  It worked out of the box.  I didn't have to do anything.  I've been watching the logs, please see below.


1.2.3.4 - - [05/Jan/2008:06:58:02 -0600] "GET /1.jpg HTTP/1.1" 403 - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; rv:1.9b3pre) Gecko/2008010203 SeaMonkey/2.0a1pre"
1.2.3.4 - - [05/Jan/2008:06:58:40 -0600] "GET /1.jpg HTTP/1.1" 403 - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2) Gecko/2007121120 Firefox/3.0b2"
1.2.3.4 - - [05/Jan/2008:07:00:09 -0600] "GET /1.jpg HTTP/1.1" 200 62205 "http://test.dlsgallery.com/" "Opera/9.25 (Windows NT 5.1; U; en)"

Comment 11

[Vee]

I can confirm it works from Opera.

Comment 12

[Dasa Paddock]

In my testing, only Firefox does not send the Referer, but this is true only for GET requests. Firefox DOES send the Referer for POST requests. I think it should send it for GET requests as well like the rest of the browsers.

Comment 13

[Geoff Stearns]

Here's a test page to help show the problem:

http://dev.deconcept.com/referer_tester/

When clicking the "Push me" button in any browser other than Firefox, the referer is shown, but in Firefox it's always empty.

Comment 15

[Dieter Menne]

Safari 3.1.2 (525.2.1) has the same issue (=problem, bug, or whatever)

Comment 19

[Matthias Versen [:Matti]]

from bug 383708:

>When loading urls from a browser plugin using NPN_Get/PostURLNotify, no Referer
>(referrer) header is added. 
>
>This does work in Safari, which uses the URL of the page embedding the plug-in,
>which in my opinion is correct.

Comment 21

[Matthias Versen [:Matti]]

it seems that bug 157796 regressed

Comment 22

[Tyler]

Anyone looking at fixing this bug?

Comment 23

[Andrew Hagen]

Regression as per comment 21.

Comment 24

[Arnold]

this issue is not sloved in FF 3.5...

Comment 25

[smoothkruger]

My question is, with the installed base of FireFox users spanning so many versions, some still at 1.0 will the bug, when fixed, be a recursive fix, or only for versions going forward. 

Otherwise, the only way to use hotlink protection it to tell people not to use a Gecko-based browser (the same bug is in Camino). That comes off badly for all parties

Comment 26

[Sébastien Deleuze]

Please fix this bug in 3.5.x, it is blocking for a lot of big companies working with Flash.

Comment 31

[pranav]

Is this bug going be to fixed??

Comment 32

[Keli Hlodversson]

Also note that this bug does not only affect the flash plugin. It affects any plugin using NPN_GetURLNotify or NPN_PostURLNotify.

Comment 34

[rui]

Any news from this bug please ??? Still trying to find the solution....

Comment 35

[N0n3]

is it a joke ?? still no fix for that ??
how to setup correctly hotlink if FF don't do like others ??

Comment 36

[alex da re]

there's no other way than to ban completely firefox from your video sharing site. we did it with ours. users cannot use firefox to access the site
I think ff will follow the path of netscape. remember about that hot browser?

Comment 37

[hensyoung]

Three years, this SMALL bug which nobody of FF dev team cares will become NO bug.
Cause as the time goes by, HTML5 will make this bug no more exist.

A really wise way of fixing bug. Hahahaha...

Comment 40

[rgreg]

3.5 years and bug dont solve?

Comment 41

[admin]

wow pretty sad.  all browsers make it work except this one.

Comment 42

[Bozo Clown]

Another vote to get this show stopper fixed.

Comment 43

[Adam Taylor]

from: http://forums.mozillazine.org/viewtopic.php?t=617988
---------
(User grindlay posted):
Just a note with a workaround for cPanel/WHM hosted sites:
In the cPanel Hot Link Protection page,
Check the box that says "Allow direct requests (ie. entering the url to an image in your browser)"
For some reason (?) that will allow Flash movies to load in FF with Hot link protection active.
.htaccess file looks the same so clearly not solved by any wizardry in there.
--------

Comment 44

[Shawn Carnell]

+1 for fix.

Comment 45

[mozilla]

I just want to mark this date. I've been cc'd on this major bug for over 3 years and nothing has happened. I also stopped working on my project where this mattered, and also stopped using FF 2 years ago. So it's time to unsubscribe. Email me directly if it's ever fixed and I'll buy the author a bottle of champagne!

Comment 46

[manu]

+1 for fix

Comment 47

[dgricci]

+1 for fix

Comment 48

[Cellier fabien [:fcellier]]

Attachment: Add referrer with http request

I'll try to ping someone on IRC in order to know if it's not in conflict with NPAPI before adding a reviewer.

Comment 49

[Alice0775 White]

(In reply to Cellier fabien [:tallion] from comment #48)
> Created attachment 582926 [details] [diff] [review]
> Add referrer with http request

FYI. The patch works properly in local win32 build.

Comment 50

[Josh Aas]

Comment on attachment 582926 [details] [diff] [review]
Add referrer with http request

Review of attachment 582926 [details] [diff] [review]:
-----------------------------------------------------------------

It looks like we do set a referrer on embedded streams (browser-initiated, data=) and I don't see why we can't set one on other plugin-initiated streams. Feedback+ on the concept, but I'd like to know Boris Zbarsky's thoughts on security.

::: dom/plugins/base/nsPluginHost.cpp
@@ +3094,5 @@
>    // deal with headers and post data
>    nsCOMPtr<nsIHttpChannel> httpChannel(do_QueryInterface(channel));
>    if (httpChannel) {
> +      
> +    rv = httpChannel->SetReferrer(doc->GetBaseURI().get());  

Why 'GetBaseURI' here? I think 'GetDocumentURI' might be better, it's what we already do for embedded plugin data streams.

@@ +3112,5 @@
>        NS_ASSERTION(uploadChannel, "http must support nsIUploadChannel");
>  
>        uploadChannel->SetUploadStream(aPostStream, EmptyCString(), -1);
>      }
> +    

Unnecessary whitespace change.

@@ +3117,4 @@
>      if (aHeadersData)
>        rv = AddHeadersToChannel(aHeadersData, aHeadersDataLen, httpChannel);
> +    
> +    NS_ENSURE_SUCCESS(rv,rv);

I'd prefer this be in the "if (aHeadersData)" scope, only running after the assignment.

Comment 51

[Josh Aas]

Comment on attachment 582926 [details] [diff] [review]
Add referrer with http request

Adding r- for the issues I outlined in my comments (leaving feedback+ on the concept).

Comment 52

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 582926 [details] [diff] [review]
Add referrer with http request

> I think 'GetDocumentURI' might be better

Yes.  It would also have the virtue of not leaking, which the GetBaseURI() call there does (having to do the explicitly .get() to try to leak is supposed to protect you from doing that accidentally!).

I don't see any security problems with this; that part is fine.

Comment 53

[Cellier fabien [:fcellier]]

Attachment: Add referrer with http request

Comment 54

[Boris Zbarsky [:bzbarsky]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/65da0d7b408d

Comment 55

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/65da0d7b408d

Comment 57

[dindog]

Well, this bug is meant to fixed some "deny" because of not sending referer in some flash.

But after it landed, many flash video also deny the request because it sending referer. (aside from the above depend on, i also have this two:

    Some youtube videos posted on fb not playing on Nightly
         http://forums.mozillazine.org/viewtopic.php?f=23&t=2424113
    the second video of this page:
         http://tieba.baidu.com/p/1405989753?pid=17126722962&cid=0#17126722962


And the I have a quick look to other browsers, none of IE and Chrome send referer in Flash request. It's kind of standard by custom. 

This is only landding on Nightly, and just about a month since it's landed, we got more than a half negative report than this bug 4 years history, affecting big site like Facebook. I'm quite sure there will be more *bug* report when Fx 12 ship with this patch.

So are this bug should be re-considered?

Comment 58

[Boris Zbarsky [:bzbarsky]]

Requesting tracking.  We need to figure out whether to back this out on branches.  

As far as other browsers go, do they never send Referer?  Or just not in POST requests (but do send with GET requests)?

Comment 59

[dindog]

(In reply to Boris Zbarsky (:bz) from comment #58)
> Requesting tracking.  We need to figure out whether to back this out on
> branches.  
> 
> As far as other browsers go, do they never send Referer?  Or just not in
> POST requests (but do send with GET requests)?

I'm sorry, I am wrong, Chrome does send referer, but not the referer of the page's URI, but seem like the the embed SRC... that would solved both of this bug and also the patch breaks site issue..

Comment 60

[Boris Zbarsky [:bzbarsky]]

Hmm.  Sending the plug-in itself as the referrer would make a lot of sense to me.

Comment 61

[dindog]

In the testcase, other browser sending is:
[http://dev.deconcept.com/referer_tester/referrer_test.swf]

Current Nightly is:
[http://dev.deconcept.com/referer_tester/]

Thing goes bad when embed plugin in other domain. It should reopen and let someone fix it.

Comment 62

[Cellier fabien [:fcellier]]

Yes, sorry for the delay :S. I will work on it next week, I'll try to use the dom element in order to find the url:

(My idea use https://hg.mozilla.org/projects/webcl/file/74c6ef4291c7/dom/plugins/base/nsPluginHost.cpp#l3086 ? could be a good solution).

Comment 63

[Boris Zbarsky [:bzbarsky]]

We should either open a new bug for that or use one of the bugs blocking this one, instead of doing the work here in a closed bug.

Comment 64

[Benjamin Smedberg]

I do not think that we should send the plugin as the referrer, because there are going to be lots of weird inconsistencies, but I'd love to be cc'ed on the newly-filed bug.

Comment 65

[Ache]

For the main .swf included by embed or object recent browsers (IE, Opera, Chrome) sends page referer, but for all materials which this .swf gets they send URL of this .swf.

Comment 66

[Matthias Versen [:Matti]]

We could use bug 724465. I will sort the mess with the depending bugs later and dupe them to bug 724465
Opera doesn't send a referer on post requests initiated by Plugins.i have a wireshark snippet in bug 724465 that confirms this.

This seems to be only a problem with IIS (bug 721311#c2) and not Apache

Comment 67

[Cellier fabien [:fcellier]]

>For the main .swf included by embed or object recent browsers (IE, Opera, Chrome) sends >page referer, but for all materials which this .swf gets they send URL of this .swf.
 

The first request is OK for all browser, the problem is after initialisation 

 Flash is one plugin but the change is directly in NPAPI and we don't  know if it's the material or the pluginInstance/runtime which want the  request, so referer could be irrevelant...

 So we can :
 - do nothing : In my knowledge, there is no standard for that, and having dom ref is more revelant (from my point of view)
 - accept change proposed by adobe
 (https://wiki.mozilla.org/NPAPI:GetRequestHeaders) AND the plugin update
 - create specific code for flash/silverlight...

 Furthermore, I don't understand the value of the material referer :
 It's not a security because we can easily use iframe or create a proxy
 (5 min with nodejs or python!). In fact you can also do the same with simple referer .

Comment 68

[Boris Zbarsky [:bzbarsky]]

This bug as filed is fixed.  Please do not reopen it unless you back the patch out.  Followup work should be in followup bugs...

Comment 69

[Cellier fabien [:fcellier]]

Ha ok, sorry for that.

Comment 70

[Ache]

>  Furthermore, I don't understand the value of the material referer :
>  It's not a security because we can easily use iframe or create a proxy
>  (5 min with nodejs or python!). In fact you can also do the same with
> simple referer .

Site_#1, flashplayer #1, #1.mp4 protected by referer in .htaccess from outside linking

Site_#2, flashplayer #2 attempts to play site_#1/#1.mp4, all browsers can't because of protection, excepting FF.

Comment 71

[Benjamin Smedberg]

Tests at https://hg.mozilla.org/integration/mozilla-inbound/rev/af615c25e502

Note that in bug 724465 this has now been disabled for POST requests.

Comment 72

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/af615c25e502

Comment 73

[dindog]

I see all related bug are mark as dupe or resolved, just in case, the video playing issue remain, for Firefox sending wrong referer.

Comment 74

[dindog]

Attachment: this video won't play with wrong referer.

Comment 75

[dindog]

Sorry for the spam, the testcase is invalid, for some reason, attachment here don't send referer... you can test it here:

http://www.cnblogs.com/dindog/articles/2360745.html

Comment 76

[Benjamin Smedberg]

The difference in referer for GET posts is now filed as bug 729009.

Comment 77

[Benjamin Smedberg]

This was backed out of Firefox 12 because of changes that Josh has decided on in bug 729009 that we don't want to rush into late-aurora. https://hg.mozilla.org/releases/mozilla-aurora/rev/7e5412504995

Comment 79

[Ioana (away)]

Verified the issue as fixed on Firefox 13.0 beta 3 (BuildID: 20120509070325).