1582 - [NECKO][webshell] Send HTTP Referer field to server, "network.sendRefererHeader"

Status: VERIFIED FIXED

(SeaMonkey :: UI Design, defect, P1)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The HTTP_REFERER variable is not sent to the server... it should have the URL
of the referring page (and should not be sent if the URL was typed in or if it
is the home page, etc.).

Comment 1

[leger]

Setting all current Open Critical and Major to M3

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Added test page to URL that shows what HTTP fields are being sent:

http://www.fas.harvard.edu/~dbaron/html/httpreq

Comment 3

[Paul MacQuiddy]

setting paulmac as QA contact for all gagan's bugs (sorry for the spam)

Comment 4

[Suresh Duddi (gone)]

Moving to M4.

Comment 5

[Gagan]

Marking till Necko lands.

Comment 6

[Paul MacQuiddy]

*** Bug 4882 has been marked as a duplicate of this bug. ***

Comment 7

[Gagan]

Per DP's suggestion marking these till M8. Though Necko lands with M7, we will
be able to verify it for M8.

Comment 8

[James Everingham]

I'm moving this to target M9, Necko will be enabled somewhere during late M8 or
early M9.  We will need to get on this and it cannot be postponed past the M9
milestone.

Comment 9

[Eli Goldberg]

*** Bug 9178 has been marked as a duplicate of this bug. ***

Comment 10

[leger]

Changing all Networking Library/Browser bugs to Networking-Core component for
Browser.

Occasionally, Bugzilla will burp and cause Verified bugs to reopen when I do
this in a bulk change.  If this happens, I will fix. ;-)

Comment 11

[Gagan]

Neither it is sent with Necko. Jud what was the decision on this regard? Did we
want to set a preference for the same?

Comment 12

[Judson Valeski]

This isn't a clear cut decision. However, we've hashed out a few absolutes wrt
how the referrer should be set.

1. It is _never_ sent when a user types in a url.
2. It is _always_ sent when a link is clicked on (there is no pref to turn this
off).
3. In the 4.5+ (somewhere around there) codebase we did not send the referrer
header when the url was an image (i.e. the image lib wouldn't set referrer
anymore), _unless_ it had been clicked on (i.e. in an <a href=x><img src=></a>

For the time being, it's ok to send it everytime, except when the user types in
a url.

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think you should send the referer for inline images.  It allows people to see
where bad links come from.  I was under the impression most browsers did so.  A
quick look at referer logs shows this is true.

Section 14.36 of HTTP 1.1 draft,
http://www.w3.org/Protocols/HTTP/1.1/draft-ietf-http-v11-spec-rev-06.txt , says:

   The Referer[sic] request-header field allows the client to specify,
   for the server's benefit, the address (URI) of the resource from
   which the Request-URI was obtained (the "referrer", although the
   header field is misspelled.) The Referer request-header allows a
   server to generate lists of back-links to resources for interest,
   logging, optimized caching, etc. It also allows obsolete or mistyped
   links to be traced for maintenance. The Referer field MUST NOT be
   sent if the Request-URI was obtained from a source that does not have
   its own URI, such as input from the user keyboard.

       Referer        = "Referer" ":" ( absoluteURI | relativeURI )

   Example:

       Referer: http://www.w3.org/hypertext/DataSources/Overview.html

   If the field value is a relative URI, it SHOULD be interpreted
   relative to the Request-URI. The URI MUST NOT include a fragment. See
   section 15.1.3 for security considerations.

Comment 14

[Judson Valeski]

I'm having trouble understanding how sending a referrer w/ every inline image
could help bad links get resolved. Can you please provide an explaination?

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

If people move inline images, they may not know all the files that refer to
those images - inline images are often used by multiple pages.

Sometimes inline images are even used by people on other sites (and are
sometimes intended to be).  If someone else starts using my inline images, I
don't want to see my error logs fill up and not know who is still using those
images (possibly even me!).

This is what the large majority of web browsers do, I think, judging from the
frequency of inline images appearing in the referrer logs I just looked at.  I
just went through the last few hundred (?) lines of a referer log for a server
where the root page "/" links to one inline image.  There were 28 refers to the
root page of the server.  26 of those 28 were immediately followed by a refer
from the home page to the inline image in it.  Many of these hits were probably
from Netscape 4.x.  The two times there was no referer for the inline image
could easily be people not loading inline images (or using lynx) or who clicked
on a link in the page before the image started loading.

If you break this behavior, you will make life much harder for web server
admins.

Comment 16

[Judson Valeski]

This is good info. Our motivation for halting referrer fields on inline images
was bandwidth based. we wanted to reduce the size of the HTTP request. However,
it appears we didn't put enough weight on the point you're now making. I guess
we just assumed the log admin receiving the referrer of the broken inline
wouldn't be using this info. Looks like we assumed incorrectly and we should be
sending the referrer header along w/ inline images.

Thanks for clarifying.

Comment 17

[Gagan]

In light of the discussion we have had here, it looks like the referrer field
decision really will have a tighter association with the UI. And so for more
details on who is going to add this where I am assigning this to webshell. (Does
 webshell know the difference between a link clicked or one typed in Location
bar?) Since I dont see webshell in the components list I am assigning this to
XPApps... Feel free to correct me.

Comment 18

[don]

Radha, do we need to fix this in M9?

Comment 19

[Radha on family leave (not reading bugmail)]

Answering and asking few questions....

1) I read thro' this and I am not sure why this is a UI thing.
2) webshell does know the difference between a link click and a url typed in the
urlbar. But only to some extent. After a couple of calls, they both go thro' the
same codepath down to Necko
3) If necko *really* needs to know whether the url load request occured because
of link click or otherwise, then  webshell needs to pass some info to DocLoader.
4) Jevering says that this *has* to be fixed in M9.
5) cc'ing warren to help clarify things.

Comment 20

[Warren Harris]

I agree that the webshell needs to pass the referrer to the doc loader, but I'm
not sure how this worked before. The doc loader should define a callback that
the protocol (http) can call to get the referrer. Gagan should handle that
part.

Judson was going to look at the web shell code to fix up the default protocol,
www. and .com parts -- I would suspect that the distinction between the
location bar and clicking on a link would be coming into play in that same
general vacinity, so maybe he should handle that part.

Comment 21

[Radha on family leave (not reading bugmail)]

Based on warren's comments, I'm reassigning this  to valeski. Let scc know if
you want some API modified in webshell since he is the current owner of it.

Note:
When the url is typed in to the urlbar, BrowserAppcore calls in to webshell
thro' LoadURL(). When a link is clicked, layout calls in to
nsWebShell::HandleLinkClick() which eventually calls LoadUrl().

cc'ing scc

Comment 22

[Gagan]

I recently fixed post requests to send a referer field since most post servers
require referer field. Pls. use the same logic in whereever the inline URLs for
the imageas are being constructed. For more details look at
mozilla/layout/html/forms/nsFormFrame.cpp and grep for "Referer"

Comment 23

[Nicholas Cull]

*** Bug 10099 has been marked as a duplicate of this bug. ***

Comment 24

[Radha on family leave (not reading bugmail)]

OK. I spoke to Rick.  Here is what we came up with,

0) Since load groups get passed around to all types of necko customers,
it looked like a good centralized place to manage referer.
1) add a baseuri member to load group
2) Based on privacy preference settings(which webshell will know about), and
other deciding factors, webshell/docloader will set the base uri in the load
group.
3) And down in necko, this base uri will be used to set the referer field. The
point is, if the referer has to be sent, baseuri will be set, if not, baseuri
will not be set.
3) Other components like libimage has to do similar things for inline images.
when a image load request comes thro' does libimg know about the base url? maybe
pnunn can answer this.
4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?

adding rpotts,pnunn to cc list. warren, can you comment on this.

I don't believe this is a M9 blocker. So, moving to M10.

Comment 25

[Warren Harris]

Sounds like a good plan. One question: Should the baseuri be the same thing as
the first URI inserted into the group?

Comment 26

[Hixie (not reading bugmail)]

*** Bug 5408 has been marked as a duplicate of this bug. ***

Comment 27

[Radha on family leave (not reading bugmail)]

Rick, I'm reassigning this to you, so that it gets counted in your plate. Once
you have the loadgroup part working, you can re-assign back to me so that I
hook-up webshell to it.

Comment 28

[Tim Taylor]

"4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?"

Referer need never be sent.  Any servers relying on referrer being sent are
flawed.

<opinion>I would consider it a bug, and a security flaw, were Mozilla to
transmit referer despite my preference indicating that it should not, assuming
such a preference setting existed.</opinion>

Comment 29

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Example of server relying on referrer: http://www.intellicast.com/

Most of the weather images within the site can only be viewed if the referrer is
the page that contains the image.  It also has an ad.  Otherwise people would
link straight to the images.

Comment 30

[Matthew Tuck [:CodeMachine]]

Sounds like another candidate for bug #7380 use.

Comment 31

[Masatoshi Kimura [:emk]]

*** Bug 16404 has been marked as a duplicate of this bug. ***

Comment 32

[Warren Harris]

*** Bug 14129 has been marked as a duplicate of this bug. ***

Comment 33

[Warren Harris]

*** Bug 14128 has been marked as a duplicate of this bug. ***

Comment 34

[chris hofmann]

m13

Comment 35

[Oleg Rekutin]

For the paranoid, one may consider including a preference option to disable the
sending of the referrer, but no matter what this preference should either be
enabled or if there isn't one, the referrer should be sent.

Currently, referrer is the *only* way for webmasters to ensure that images or
other copyrighted files are viewed/downloaded exclusively from their own pages,
and not pirated by someone else.

Secondly, there's the issue of backwards compatibility: all previous Netscape
versions sent HTTP_REFERER. Not doing so in Mozilla will break many sites that
depend on this behavior (also exhibited by IE, Opera, Lynx, and others).

Finally, Mozilla's going to be rejected by webmasters if it doesn't send
HTTP_REFERER not only because their CGI scripts or other referrer-dependent
protections will break, but for the lack of alternative solution (that is at
least remotely reliaible).

Comment 36

[Mats Palmgren (inactive)]

*** Bug 19054 has been marked as a duplicate of this bug. ***

Comment 37

[Judson Valeski]

*** Bug 18331 has been marked as a duplicate of this bug. ***

Comment 38

[Eric Pollmann]

*** Bug 21794 has been marked as a duplicate of this bug. ***

Comment 39

[Eric Pollmann]

*** Bug 22131 has been marked as a duplicate of this bug. ***

Comment 40

[Eric Pollmann]

*** Bug 20819 has been marked as a duplicate of this bug. ***

Comment 41

[Eric Pollmann]

*** Bug 23319 has been marked as a duplicate of this bug. ***

Comment 42

[Eli Goldberg]

*** Bug 19037 has been marked as a duplicate of this bug. ***

Comment 43

[rpotts (gone)]

We definately need to deal with this all the way from the URI loader down
through the nsHTTPChannel...  I'll look at this when I get back.

Comment 44

[[e-mail]]

Not for alpha, but eventually, you MUST make sure you never let non-HTTP

referers into the real world. This is a security hole, because non-HTTP

protocols can have all sorts of secret stuff in them, like my FS layout (in

file://) or which news articles I read (in news:// or nntp://) or my passwords

(ftp:// in some cases).

Unlike HTTP URLs, these other protocols are NOT expecting to be exposed to

random web logs owned by potentially hostile people. So, Mozilla must prevent

them from escaping into REFERER at all. No useful web pages will be broken by

this if it's implemented correctly.

Comment 45

[Judson Valeski]

*** Bug 16587 has been marked as a duplicate of this bug. ***

Comment 46

[Jonathan Abbey]

I disagree with [e-mail].  It can be very useful to see via the
referer when someone mentions my web site in a USENET posting.  I would hate
to see Mozilla unilaterally decide to block this in all circumstances.

Comment 47

[Eric Pollmann]

*** Bug 25257 has been marked as a duplicate of this bug. ***

Comment 48

[Damian Christey]

*** Bug 25877 has been marked as a duplicate of this bug. ***

Comment 49

[Judson Valeski]

*** Bug 25977 has been marked as a duplicate of this bug. ***

Comment 50

[Judson Valeski]

*** Bug 25063 has been marked as a duplicate of this bug. ***

Comment 51

[Steven de Hoog]

*** Bug 26510 has been marked as a duplicate of this bug. ***

Comment 52

[leger]

Due to Beta indication in Summary, putting beta1 into keyword field.

Comment 53

[Sitsofe Wheeler]

*** Bug 26663 has been marked as a duplicate of this bug. ***

Comment 54

[rickg]

RickP: can we get your input on this. Is it essential for beta?

Comment 55

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This prevents use of some major sites, for example, http://www.intellicast.com/
(which is probably the best source of US weather radar on the Net).

Not fixing this will also annoy webmasters because they won't get the referrer
field in their logs.

Comment 56

[[e-mail]]

I don't disagree that it would be USEFUL to a web master to see which news
articles I read; he'd probably also find my PASSWORDS and CREDIT CARD NUMBER
useful, but that's no reason to create potential security problems. Bugtraq has
seen several "interesting" things sent as referer URLs, and restricting them to
HTTP-only seems like the most conservative solution.

Comment 57

[Sitsofe Wheeler]

*** Bug 16003 has been marked as a duplicate of this bug. ***

Comment 58

[daver]

still need input from Rick Potts to determine PDT status.

Comment 59

[Gagan]

taking over from rick... 

Comment 60

[leger]

Putting on PDT+ radar for beta1.

Comment 61

[Hervé Renault]

this bug is really annoying. all sites on altern.org are viewable as text only

with mozilla...

example: http://www.altern.org/bmxzine/



an interesting test is to load this page with Nav4.x (it should display

correctly), then load one of the images in a new window and then reload the

image : due to the fact no referrer is transmitted during the reload, you get a

permission denied, as in mozilla.



i guess that the server puts this restriction in order not to be used as a mere

"images server"...

Comment 62

[Hervé Renault]

for the accuracy, i must add that this server blocks only .jpg files with the
referrer trick, not .gif (don't ask me why)

Comment 63

[Hixie (not reading bugmail)]

Herve: No need to complain, it's on PDT+ radar which means it should be fixed
very very soon. :-)

PDT: Product Development Team. 
PDT+: A bug which MUST be fixed by beta1.

Comment 64

[Hervé Renault]

well, that's not really a complaint. just a way of submitting another "real life
example" of a server that won't give the images if no appropriate referrer is
transmitted.

Comment 65

[Hixie (not reading bugmail)]

Oh, right. In that case thanks!

Comment 66

[Gagan]

fixed. waiting for review. 

Here is what the "network.sendRefererHeader" pref means-
0- Won't send the "Referer" header in any case. 
1- Will send it from links clicked on a document. 
2- Will send it for image requests from a page as well as (1). 

The default of this pref is 2. 

Comment 67

[Gagan]

fix checked in.

Comment 68

[Tom Everingham]

verified referer header w/Trace32
NT 2000022408

Comment 69

[Richard Zach]

*** Bug 29264 has been marked as a duplicate of this bug. ***

Comment 70

[rxsherm]

*** Bug 34735 has been marked as a duplicate of this bug. ***

Comment 71

[rxsherm]

It looks like this bug may have reopened? See bug 34735 on M14 build 033112

Comment 72

[Lars]

Hey,
please tell me, if I am correct in this Bug.
Mozilla doesn't send a HTTP_REFERER when it loads a document
in an ILAYER.
Netscape and Microsoft send one.
I made a little testpage where I explain all details.
You can find it under: http://212.227.152.190/mozilla/

I hope I am correct in this bug. Else please tell me
the correct one.

Thanx,
Lars

Comment 73

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

[e-mail] - Please file that as a separate bug report.

Comment 74

[ckritzer (gone)]

Reopening for MacOS platform.

Nominating for nsbeta2 since it crashes on MacOS9.

Comment 75

[ckritzer (gone)]

Crashes on:
- MacOS 2000-05-16-09-M16 Commercial
Works fine on:
- Win98, Linux 2000-05-16-09-M16 Commercial

Comment 76

[ckritzer (gone)]

Upgrading severity

Comment 77

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Hmmm... what crashes?  My test page?  If so, does it have anything to do with
this bug?

(Also retitling to remove my silly Apache-ism.)

Comment 78

[Gagan]

please file separate bugs for the new problem(s). thx. closing this again. 

Comment 79

[ckritzer (gone)]

Hmmnn....lemme try it again.  I believe (and will confirm or deny this later) 
that the page crashes when loading.n -kritzer

Comment 80

[ckritzer (gone)]

Okay, my bad.  This bug does *not* cause a crash.

The bug still exists on the Mac.  The section of the page that deals with 
referer looks like this:

Referer: (none)

Which is incorrect according to this bug report.  Lastest mac bits have been 
flaky, so I'll check it one last time with today's (2000-05-18-xx-M16) bits.

Comment 81

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

When testing, make sure to click on the URL to get to the page, rather than type
it in (if you type in the url, (none) is correct).

Comment 82

[ckritzer (gone)]

Ahhhh...yes, well that's the trick, isn't it?  <grin>  Thanks for the clue, 
dbaron.
Marking VERIFED FIXED on:
 - MacOS9 2000-05-18-09-M16 Commercial Build
 - Linux6 2000-05-18-09-M16 Commercial Build
 - Win98 2000-05-18-09-M16 Commercial Build

Comment 83

[Michel Carroll]

from: Michel Carroll <[e-mail]>

still not fixed in NN 6.1 AND Mozilla Build ID: 2001080110.

Note test bed: browser running on Win 2K sp2 client with Win 2K sp2 server
running ASP 3 on IIS5.

ASP O/P:

Request.ServerVariables("all_http"):
HTTP_ACCEPT:text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
HTTP_ACCEPT_LANGUAGE:en-us
HTTP_CONNECTION:keep-alive
HTTP_HOST:www.swipe.plc.uk
HTTP_USER_AGENT:Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.3)
Gecko/20010801
HTTP_CONTENT_LENGTH:33
HTTP_CONTENT_TYPE:application/x-www-form-urlencoded
HTTP_ACCEPT_ENCODING:gzip, deflate, compress;q=0.9
HTTP_ACCEPT_CHARSET:ISO-8859-1, utf-8;q=0.66, *;q=0.66
HTTP_KEEP_ALIVE:300

Comment 84

[Steve Freeland]

Any particular conditions needed to reproduce?  I just put together a simple
test case and it seems to work fine; got the following request (Linux build
2001090408):

GET /test.html HTTP/1.0
Host: lintra:1355
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3+) Gecko/20010904
Accept: text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
Accept-Language: en-us
Accept-Encoding: gzip, deflate, compress;q=0.9
Accept-Charset: ISO-8859-1, utf-8;q=0.66, *;q=0.66
Connection: close
Referer: http://www/~nephtes/test.html

Comment 85

[benc]

I'm resolving and verifying this bug.

Look for existing Referer bugs and file new ones. Please file off 0.9.4 (coming
soon, NS 6.1 or the daily trunk build.

Comment 86

[Athman Boukhaoua]

This bug still partially exists in build 2001090908.

When following a link through a simple click, the Referer-Header gets sent
(expected behavior). However when right-clicking and selecting "Open Link in New
Window", _no_ Referer gets sent (Which is definitely wrong). You can test this
on the Link posted by [e-mail]
But I don't think this bug really is a blocker. Major or even normal would be
more appropriate.

And as for the privacy issues: RFC 2068 chapter 14.37 has it's own collection of
SHOULD's and MUST NOT's on this topic. However, I feel that the discussion of
these issues are out of scope for this bug, and should rather be posted as an
enhancement query. Implementing a preference to specify whether Mozilla sends
cross-site Referers might be a good idea though.

Comment 87

[rpotts (gone)]

hey gagan,

i don't know if i should mention this or not - but i'm actually looking at the
issue of passing the referer header along when new windows are targeted :-)

If that is *really* what this bug has digressed to, then maybe i should take it
over...

But don't stick me with *all* the 'referer' bugs :-)

-- rick


P.S.
It might be better to open a new bug specific to the window targeting situation!  

Comment 88

[benc]

Bug 48902. We stuck paul with that one.

I'm calling this one: RESOLVED/FIXED.

Please see my previous comment.

Comment 89

[Bradley Baetz (:bbaetz)]

re-VERIFIED, cause this has been working for ages (modulo the specific bugs
listed in the comments)