Last Comment Bug 170477 - XMLHttpRequest()s have no referer
: XMLHttpRequest()s have no referer
Status: RESOLVED FIXED
: fixed1.8.1
Product: Core
Classification: Components
Component: XML (show other bugs)
: Trunk
: All All
: -- normal with 1 vote (vote)
: mozilla1.8.1beta2
Assigned To: Johnny Stenback (:jst, jst@mozilla.com)
: Ashish Bhatt
Mentors:
: 311189 (view as bug list)
Depends on:
Blocks: referer
  Show dependency treegraph
 
Reported: 2002-09-24 01:45 PDT by Sven Neuhaus
Modified: 2006-07-13 16:22 PDT (History)
14 users (show)
darin.moz: blocking1.8.1+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Send the caller source location as referrer. (806 bytes, patch)
2006-07-05 17:30 PDT, Johnny Stenback (:jst, jst@mozilla.com)
jonas: review+
darin.moz: superreview+
mtschrep: approval1.8.1+
Details | Diff | Splinter Review

Description Sven Neuhaus 2002-09-24 01:45:19 PDT
The HTTP request sent by a XMLHttpRequest has no referer.

Tested with Mozilla 1.1

Example code:

<script>
 var x = new XMLHttpRequest();
 x.open("GET", "example.cgi?" + Math.random(), false);
 x.send(null);
</script>

The webserver's access log will reveal that the referer header is missing.
Comment 1 Sven Neuhaus 2002-09-26 01:37:57 PDT
Are you able to reproduce this bug? Comments?
Comment 2 Sven Neuhaus 2004-02-09 01:47:15 PST
This bug is still present in v1.6. Any idea how to fix this?
Comment 3 Wladimir Palant 2004-02-21 02:46:30 PST
Should it really have a referer by default?

x.setRequestHeader("Referer", location.href);
Comment 4 Wladimir Palant 2004-03-27 12:16:39 PST
IE 6.0 doesn't set the Referer header for XMLHttpRequest by default, suggesting
won't fix for this.
Comment 5 Heikki Toivonen (remove -bugzilla when emailing directly) 2004-03-27 21:57:46 PST
Sounds good to me.
Comment 6 Sven Neuhaus 2004-10-11 01:45:34 PDT
I think it would be very useful if requests had a referrer by default. It helps
tracking down problems if your XML-RPC server suddenly gets a lot of hits. 
IE 6's behaviour shouldn't matter, should it?
Comment 7 Jesse Ruderman 2005-10-05 14:33:31 PDT
*** Bug 311189 has been marked as a duplicate of this bug. ***
Comment 8 Jesse Ruderman 2005-10-05 14:34:51 PDT
The workaround in comment 3 no longer works because it was a security hole.  I
think this bug should be fixed.
Comment 9 Christian :Biesinger (don't email me, ping me on IRC) 2005-10-05 14:58:20 PDT
according to bug Bug 311189 IE does send the referrer, contrary to comment 4...
Comment 10 Sven Neuhaus 2005-10-06 05:46:16 PDT
I rechecked this and
IE 6.0.2800.1106 (Windows 2000) and
Safari 1.3.1 on MacOS now do send a referer.

Konqueror 3.4.2 (Linux) does not yet send a referer (I just opened bug 113962 at
bugs.kde.org)

Looks like Microsoft and Apple saw the light, when will Mozilla? ;-)
Comment 11 Mark Nottingham 2006-03-25 12:49:30 PST
Another use case is access control using Referer headers; several large Web sites use Referers to control content theft; Akamai, for example, provides this as a service. 

This extends to Ajax sites; If Mozilla doesn't send Referer from XmlHttpRequest, many sites will be more reluctant to use it.
Comment 12 Dão Gottwald [:dao] 2006-03-25 13:15:35 PST
IE7 with its native XMLHttpRequest object sends the referrer, too. Tested here: http://design-noir.de/bugzilla/XMLHttpRequest-referrer.php
Comment 13 Mike Shaver (:shaver -- probably not reading bugmail closely) 2006-04-26 12:46:32 PDT
This sucks for many web apps, I'd like to see it fixed in 1.8.1/Fx2.

Is it still correctly assigned?
Comment 14 Sven Neuhaus 2006-04-27 02:42:46 PDT
FYI, Konqueror 3.5.2 now sends a referrer, too.
Comment 15 Mike Schroepfer 2006-06-13 16:35:08 PDT
JST - can you take a whirl at this one?
Comment 16 Darin Fisher 2006-06-26 11:43:39 PDT
-> jst

Looks like this isn't going to happen for FF2 beta1.
Comment 17 Mike Shaver (:shaver -- probably not reading bugmail closely) 2006-06-26 13:41:54 PDT
Does not making b1 necessarily mean 1.8.1-?  The status whiteboard indicating ff2b2 makes me hope (!) that we can see a fix before Fx2, so I'm not sure why it would get a - here.  (Am I just out of date on the meaning of the flags?  Boldly re-requesting, at least for clarification.)
Comment 18 Darin Fisher 2006-06-28 18:17:45 PDT
Yeah, [ff2b2] was a placeholder to keep this on the radar for 1.8.1b2.  The new system is to use the target milestone to record that info, so this gets 1.8.1+ again.  Sorry for the confusion.
Comment 19 Mark Baker 2006-06-29 16:59:42 PDT
(In reply to comment #0)
> The HTTP request sent by a XMLHttpRequest has no referer.

Is that such a big deal?  Referer doesn't have the same value when you (the author) know you'll be getting back one of your own URIs.  And it just adds to the size of the message even when it's not needed, which is most of the time.  I don't see what the big deal is to require that authors set it explicitly.
Comment 20 Dão Gottwald [:dao] 2006-06-29 17:14:40 PDT
(In reply to comment #19)
> And it just adds to
> the size of the message even when it's not needed, which is most of the time.

That's not an argument. There are several HTTP headers that are not needed "most of the time".
Comment 21 Mark Baker 2006-06-30 06:48:53 PDT
(In reply to comment #20)
> That's not an argument. There are several HTTP headers that are not needed
> "most of the time".

Of course, the decision needs to be made on a header-by-header basis.  But it's clear to me that the value of Referer is dramatically lower for XHR than for typical browser requests.  I doubt it's the only one.
Comment 22 Dão Gottwald [:dao] 2006-06-30 07:08:14 PDT
(In reply to comment #21)
> But it's
> clear to me that the value of Referer is dramatically lower for XHR than for
> typical browser requests.  I doubt it's the only one.

There are sites that are not fully powered by XHR, but where it still saves some page loads. In such cases it can make sense to offer the request URL as an hyperlink for users with JS disabled. Then again this URL can be followed by robots and it can make sense to ignore them -- e.g. by checking the referer.
Of course that's not a secure solution, but checking the referer never was. Yet it meight be enough in several scenarios.
Comment 23 Mark Baker 2006-06-30 07:44:53 PDT
(In reply to comment #22)
> There are sites that are not fully powered by XHR, but where it still saves
> some page loads. In such cases it can make sense to offer the request URL as an
> hyperlink for users with JS disabled. Then again this URL can be followed by
> robots and it can make sense to ignore them -- e.g. by checking the referer.
> Of course that's not a secure solution, but checking the referer never was. Yet
> it meight be enough in several scenarios.

I can't quite grok your scenario, but is it not possible for the Referer header to be set via setRequestHeader() for those interactions that would benefit from it?
Comment 24 Hixie (not reading bugmail) 2006-06-30 11:24:52 PDT
It's important that we do this so that once XXX is enabled it'll already be supported. (For XXX this will be critical.) It's true that for a single site it's not a big deal. (XXX = cross-site extensions to XMLHttpRequest)
Comment 25 Dão Gottwald [:dao] 2006-06-30 12:36:52 PDT
(In reply to comment #23)
> I can't quite grok your scenario

My English is somewhat labile, I don't know if I can describe it any better :)
But I'll give it one more try. In my case, it's about very simple feedback links, looking like this:

    Select a download mirror:
    [mirror 1]  [report broken link]
    [mirror 2]  [report broken link]
    [mirror 3]  [report broken link]

The user can click [report broken link] in order to notify me. I'll get an email and look into it.
With XMLHttpRequest, the URL will be requested and the user will get alert("Blah blah, Thanks for your feedback"). Without JS/XMLHttpRequest, it works as an ordinary hyperlink (thus it is visible to robots). The script on the server-side should check the referer in order to disregard robot requests (they usually don't send a referer) so that I don't get false-alarm mails. Evaluating the UA would be an alternative, but that's much more complicated and fault-prone.

> but is it not possible for the Referer header
> to be set via setRequestHeader() for those interactions that would benefit from
> it?

See comment 8.
Comment 26 Mark Baker 2006-07-02 08:21:09 PDT
(In reply to comment #25)
> (In reply to comment #23)
> > I can't quite grok your scenario
> 
> My English is somewhat labile, I don't know if I can describe it any better :)
> But I'll give it one more try. In my case, it's about very simple feedback
> links, looking like this:
> 
>     Select a download mirror:
>     [mirror 1]  [report broken link]
>     [mirror 2]  [report broken link]
>     [mirror 3]  [report broken link]
> 
> The user can click [report broken link] in order to notify me. I'll get an
> email and look into it.
> With XMLHttpRequest, the URL will be requested and the user will get
> alert("Blah blah, Thanks for your feedback"). Without JS/XMLHttpRequest, it
> works as an ordinary hyperlink (thus it is visible to robots). The script on
> the server-side should check the referer in order to disregard robot requests
> (they usually don't send a referer) so that I don't get false-alarm mails.
> Evaluating the UA would be an alternative, but that's much more complicated and
> fault-prone.

Thanks for the description.  But if "report broken link" sends email, then you should be using POST instead of GET.
 
> > but is it not possible for the Referer header
> > to be set via setRequestHeader() for those interactions that would benefit from
> > it?
> 
> See comment 8.

Ick.  I did some looking around though, and couldn't find either a description of the security problem with setting Referer, nor any bug for it, nor any code that implements that.  All I found in that respect was code which blocked Via, Host, Upgrade, and Transfer-Encoding from being set.

I can't see the benefit of blocking it from being set.

Mark.
Comment 27 Johnny Stenback (:jst, jst@mozilla.com) 2006-07-05 17:30:01 PDT
Created attachment 228231 [details] [diff] [review]
Send the caller source location as referrer.
Comment 28 Darin Fisher 2006-07-05 21:02:04 PDT
Comment on attachment 228231 [details] [diff] [review]
Send the caller source location as referrer.

Why do we want the principal's URI and not the document's URI?  Couldn't they differ in some cases?
Comment 29 Mike Shaver (:shaver -- probably not reading bugmail closely) 2006-07-06 07:56:45 PDT
Because the XHR is triggered by script, my gut says that we want to use the principal and not the document on whose window object the XHR constructor happened to be found.  I've been wrong a lot already this week, though...
Comment 30 Darin Fisher 2006-07-06 11:51:41 PDT
Hmm... maybe the principal does make sense since you would want the Referer header to identify the source of the JS invoking XHR?
Comment 31 Mike Shaver (:shaver -- probably not reading bugmail closely) 2006-07-06 12:00:32 PDT
That is my belief, yeah.  That would seem to make it much more useful and reliable.

What does IE do? :)
Comment 32 Dão Gottwald [:dao] 2006-07-06 13:36:16 PDT
If I got you right, IE does the opposite (i.e. use the document on whose window object the XHR constructor happened to be found). So does Opera. That's also what I've expected, though I didn't really think about that before.

Tested here:
http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php

IE and Opera get this response:
'HTTP_REFERER: http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php'
Comment 33 Johnny Stenback (:jst, jst@mozilla.com) 2006-07-06 15:13:10 PDT
Fix landed on the trunk.
Comment 34 Dão Gottwald [:dao] 2006-07-06 17:15:23 PDT
Now it's 'HTTP_REFERER:
http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php' with Firefox, too. So I guess I got this principal vs. document thing completely wrong. :)
Comment 35 Sven Neuhaus 2006-07-07 00:50:04 PDT
Thanks all, especially -jst- for the patch!
Comment 36 David Baron :dbaron: ⌚️UTC-7 (review requests must explain patch) 2006-07-13 10:46:12 PDT
A few questions:
 * does this patch change the behavior when the script sets the Referer header explicitly?
 * was the document vs. principal issue sorted out?  Are we compatible with other browsers there?  (and compatible with any spec?)
Comment 37 Johnny Stenback (:jst, jst@mozilla.com) 2006-07-13 16:22:20 PDT
Fix landed on the 1.8.1 branch.

Note You need to log in before you can comment on or make changes to this bug.