170477 - XMLHttpRequest()s have no referer

Status: RESOLVED FIXED

(Core :: XML, defect)

Description

[Sven Neuhaus]

The HTTP request sent by a XMLHttpRequest has no referer.

Tested with Mozilla 1.1

Example code:

<script>
 var x = new XMLHttpRequest();
 x.open("GET", "example.cgi?" + Math.random(), false);
 x.send(null);
</script>

The webserver's access log will reveal that the referer header is missing.

Comment 1

[Sven Neuhaus]

Are you able to reproduce this bug? Comments?

Comment 2

[Sven Neuhaus]

This bug is still present in v1.6. Any idea how to fix this?

Comment 3

[Wladimir Palant]

Should it really have a referer by default?

x.setRequestHeader("Referer", location.href);

Comment 4

[Wladimir Palant]

IE 6.0 doesn't set the Referer header for XMLHttpRequest by default, suggesting
won't fix for this.

Comment 5

[Heikki Toivonen (remove -bugzilla when emailing directly)]

Sounds good to me.

Comment 6

[Sven Neuhaus]

I think it would be very useful if requests had a referrer by default. It helps
tracking down problems if your XML-RPC server suddenly gets a lot of hits. 
IE 6's behaviour shouldn't matter, should it?

Comment 7

[Jesse Ruderman]

*** Bug 311189 has been marked as a duplicate of this bug. ***

Comment 8

[Jesse Ruderman]

The workaround in comment 3 no longer works because it was a security hole.  I
think this bug should be fixed.

Comment 9

[Christian :Biesinger (don't email me, ping me on IRC)]

according to bug Bug 311189 IE does send the referrer, contrary to comment 4...

Comment 10

[Sven Neuhaus]

I rechecked this and
IE 6.0.2800.1106 (Windows 2000) and
Safari 1.3.1 on MacOS now do send a referer.

Konqueror 3.4.2 (Linux) does not yet send a referer (I just opened bug 113962 at
bugs.kde.org)

Looks like Microsoft and Apple saw the light, when will Mozilla? ;-)

Comment 11

[u7736]

Another use case is access control using Referer headers; several large Web sites use Referers to control content theft; Akamai, for example, provides this as a service. 

This extends to Ajax sites; If Mozilla doesn't send Referer from XmlHttpRequest, many sites will be more reluctant to use it.

Comment 12

[Dão Gottwald [:dao]]

IE7 with its native XMLHttpRequest object sends the referrer, too. Tested here: http://design-noir.de/bugzilla/XMLHttpRequest-referrer.php

Comment 13

[Mike Shaver (:shaver emeritus)]

This sucks for many web apps, I'd like to see it fixed in 1.8.1/Fx2.

Is it still correctly assigned?

Comment 14

[Sven Neuhaus]

FYI, Konqueror 3.5.2 now sends a referrer, too.

Comment 15

[Mike Schroepfer]

JST - can you take a whirl at this one?

Comment 16

[Darin Fisher]

-> jst

Looks like this isn't going to happen for FF2 beta1.

Comment 17

[Mike Shaver (:shaver emeritus)]

Does not making b1 necessarily mean 1.8.1-?  The status whiteboard indicating ff2b2 makes me hope (!) that we can see a fix before Fx2, so I'm not sure why it would get a - here.  (Am I just out of date on the meaning of the flags?  Boldly re-requesting, at least for clarification.)

Comment 18

[Darin Fisher]

Yeah, [ff2b2] was a placeholder to keep this on the radar for 1.8.1b2.  The new system is to use the target milestone to record that info, so this gets 1.8.1+ again.  Sorry for the confusion.

Comment 19

[Mark Baker]

(In reply to comment #0)
> The HTTP request sent by a XMLHttpRequest has no referer.

Is that such a big deal?  Referer doesn't have the same value when you (the author) know you'll be getting back one of your own URIs.  And it just adds to the size of the message even when it's not needed, which is most of the time.  I don't see what the big deal is to require that authors set it explicitly.

Comment 20

[Dão Gottwald [:dao]]

(In reply to comment #19)
> And it just adds to
> the size of the message even when it's not needed, which is most of the time.

That's not an argument. There are several HTTP headers that are not needed "most of the time".

Comment 21

[Mark Baker]

(In reply to comment #20)
> That's not an argument. There are several HTTP headers that are not needed
> "most of the time".

Of course, the decision needs to be made on a header-by-header basis.  But it's clear to me that the value of Referer is dramatically lower for XHR than for typical browser requests.  I doubt it's the only one.

Comment 22

[Dão Gottwald [:dao]]

(In reply to comment #21)
> But it's
> clear to me that the value of Referer is dramatically lower for XHR than for
> typical browser requests.  I doubt it's the only one.

There are sites that are not fully powered by XHR, but where it still saves some page loads. In such cases it can make sense to offer the request URL as an hyperlink for users with JS disabled. Then again this URL can be followed by robots and it can make sense to ignore them -- e.g. by checking the referer.
Of course that's not a secure solution, but checking the referer never was. Yet it meight be enough in several scenarios.

Comment 23

[Mark Baker]

(In reply to comment #22)
> There are sites that are not fully powered by XHR, but where it still saves
> some page loads. In such cases it can make sense to offer the request URL as an
> hyperlink for users with JS disabled. Then again this URL can be followed by
> robots and it can make sense to ignore them -- e.g. by checking the referer.
> Of course that's not a secure solution, but checking the referer never was. Yet
> it meight be enough in several scenarios.

I can't quite grok your scenario, but is it not possible for the Referer header to be set via setRequestHeader() for those interactions that would benefit from it?

Comment 24

[Hixie (not reading bugmail)]

It's important that we do this so that once XXX is enabled it'll already be supported. (For XXX this will be critical.) It's true that for a single site it's not a big deal. (XXX = cross-site extensions to XMLHttpRequest)

Comment 25

[Dão Gottwald [:dao]]

(In reply to comment #23)
> I can't quite grok your scenario

My English is somewhat labile, I don't know if I can describe it any better :)
But I'll give it one more try. In my case, it's about very simple feedback links, looking like this:

    Select a download mirror:
    [mirror 1]  [report broken link]
    [mirror 2]  [report broken link]
    [mirror 3]  [report broken link]

The user can click [report broken link] in order to notify me. I'll get an email and look into it.
With XMLHttpRequest, the URL will be requested and the user will get alert("Blah blah, Thanks for your feedback"). Without JS/XMLHttpRequest, it works as an ordinary hyperlink (thus it is visible to robots). The script on the server-side should check the referer in order to disregard robot requests (they usually don't send a referer) so that I don't get false-alarm mails. Evaluating the UA would be an alternative, but that's much more complicated and fault-prone.

> but is it not possible for the Referer header
> to be set via setRequestHeader() for those interactions that would benefit from
> it?

See comment 8.

Comment 26

[Mark Baker]

(In reply to comment #25)
> (In reply to comment #23)
> > I can't quite grok your scenario
> 
> My English is somewhat labile, I don't know if I can describe it any better :)
> But I'll give it one more try. In my case, it's about very simple feedback
> links, looking like this:
> 
>     Select a download mirror:
>     [mirror 1]  [report broken link]
>     [mirror 2]  [report broken link]
>     [mirror 3]  [report broken link]
> 
> The user can click [report broken link] in order to notify me. I'll get an
> email and look into it.
> With XMLHttpRequest, the URL will be requested and the user will get
> alert("Blah blah, Thanks for your feedback"). Without JS/XMLHttpRequest, it
> works as an ordinary hyperlink (thus it is visible to robots). The script on
> the server-side should check the referer in order to disregard robot requests
> (they usually don't send a referer) so that I don't get false-alarm mails.
> Evaluating the UA would be an alternative, but that's much more complicated and
> fault-prone.

Thanks for the description.  But if "report broken link" sends email, then you should be using POST instead of GET.
 
> > but is it not possible for the Referer header
> > to be set via setRequestHeader() for those interactions that would benefit from
> > it?
> 
> See comment 8.

Ick.  I did some looking around though, and couldn't find either a description of the security problem with setting Referer, nor any bug for it, nor any code that implements that.  All I found in that respect was code which blocked Via, Host, Upgrade, and Transfer-Encoding from being set.

I can't see the benefit of blocking it from being set.

Mark.

Comment 27

[Johnny Stenback (:jst)]

Attachment: Send the caller source location as referrer.

Comment 28

[Darin Fisher]

Comment on attachment 228231 [details] [diff] [review]
Send the caller source location as referrer.

Why do we want the principal's URI and not the document's URI?  Couldn't they differ in some cases?

Comment 29

[Mike Shaver (:shaver emeritus)]

Because the XHR is triggered by script, my gut says that we want to use the principal and not the document on whose window object the XHR constructor happened to be found.  I've been wrong a lot already this week, though...

Comment 30

[Darin Fisher]

Hmm... maybe the principal does make sense since you would want the Referer header to identify the source of the JS invoking XHR?

Comment 31

[Mike Shaver (:shaver emeritus)]

That is my belief, yeah.  That would seem to make it much more useful and reliable.

What does IE do? :)

Comment 32

[Dão Gottwald [:dao]]

If I got you right, IE does the opposite (i.e. use the document on whose window object the XHR constructor happened to be found). So does Opera. That's also what I've expected, though I didn't really think about that before.

Tested here:
http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php

IE and Opera get this response:
'HTTP_REFERER: http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php'

Comment 33

[Johnny Stenback (:jst)]

Fix landed on the trunk.

Comment 34

[Dão Gottwald [:dao]]

Now it's 'HTTP_REFERER:
http://design-noir.de/bugzilla/XMLHttpRequest-referrer-external.php' with Firefox, too. So I guess I got this principal vs. document thing completely wrong. :)

Comment 35

[Sven Neuhaus]

Thanks all, especially -jst- for the patch!

Comment 36

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

A few questions:
 * does this patch change the behavior when the script sets the Referer header explicitly?
 * was the document vs. principal issue sorted out?  Are we compatible with other browsers there?  (and compatible with any spec?)

Comment 37

[Johnny Stenback (:jst)]

Fix landed on the 1.8.1 branch.