Closed Bug 1582 Opened 26 years ago Closed 23 years ago

[NECKO][webshell] Send HTTP Referer field to server, "network.sendRefererHeader"

Categories

(SeaMonkey :: UI Design, defect, P1)

Product:
SeaMonkey
Component:
UI Design
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: dbaron, Assigned: gagan)

References

(
URL
)

Details

(Keywords: crash, helpwanted, testcase, Whiteboard: [PDT+] ETA 02/22/00) 

The HTTP_REFERER variable is not sent to the server... it should have the URL
of the referring page (and should not be sent if the URL was typed in or if it
is the home page, etc.).
Status: NEW → ASSIGNED
Setting all current Open Critical and Major to M3
Added test page to URL that shows what HTTP fields are being sent:

http://www.fas.harvard.edu/~dbaron/html/httpreq
setting paulmac as QA contact for all gagan's bugs (sorry for the spam)
Target Milestone: M3 → M4
Moving to M4.
Target Milestone: M4 → M6
Marking till Necko lands.
*** Bug 4882 has been marked as a duplicate of this bug. ***
Per DP's suggestion marking these till M8. Though Necko lands with M7, we will
be able to verify it for M8.
I'm moving this to target M9, Necko will be enabled somewhere during late M8 or
early M9.  We will need to get on this and it cannot be postponed past the M9
milestone.
*** Bug 9178 has been marked as a duplicate of this bug. ***
Changing all Networking Library/Browser bugs to Networking-Core component for
Browser.

Occasionally, Bugzilla will burp and cause Verified bugs to reopen when I do
this in a bulk change.  If this happens, I will fix. ;-)
Summary: referer not sent → NECKO: referer not sent
Neither it is sent with Necko. Jud what was the decision on this regard? Did we
want to set a preference for the same?
This isn't a clear cut decision. However, we've hashed out a few absolutes wrt
how the referrer should be set.

1. It is _never_ sent when a user types in a url.
2. It is _always_ sent when a link is clicked on (there is no pref to turn this
off).
3. In the 4.5+ (somewhere around there) codebase we did not send the referrer
header when the url was an image (i.e. the image lib wouldn't set referrer
anymore), _unless_ it had been clicked on (i.e. in an <a href=x><img src=></a>

For the time being, it's ok to send it everytime, except when the user types in
a url.
I think you should send the referer for inline images.  It allows people to see
where bad links come from.  I was under the impression most browsers did so.  A
quick look at referer logs shows this is true.

Section 14.36 of HTTP 1.1 draft,
http://www.w3.org/Protocols/HTTP/1.1/draft-ietf-http-v11-spec-rev-06.txt , says:

   The Referer[sic] request-header field allows the client to specify,
   for the server's benefit, the address (URI) of the resource from
   which the Request-URI was obtained (the "referrer", although the
   header field is misspelled.) The Referer request-header allows a
   server to generate lists of back-links to resources for interest,
   logging, optimized caching, etc. It also allows obsolete or mistyped
   links to be traced for maintenance. The Referer field MUST NOT be
   sent if the Request-URI was obtained from a source that does not have
   its own URI, such as input from the user keyboard.

       Referer        = "Referer" ":" ( absoluteURI | relativeURI )

   Example:

       Referer: http://www.w3.org/hypertext/DataSources/Overview.html

   If the field value is a relative URI, it SHOULD be interpreted
   relative to the Request-URI. The URI MUST NOT include a fragment. See
   section 15.1.3 for security considerations.
I'm having trouble understanding how sending a referrer w/ every inline image
could help bad links get resolved. Can you please provide an explaination?
If people move inline images, they may not know all the files that refer to
those images - inline images are often used by multiple pages.

Sometimes inline images are even used by people on other sites (and are
sometimes intended to be).  If someone else starts using my inline images, I
don't want to see my error logs fill up and not know who is still using those
images (possibly even me!).

This is what the large majority of web browsers do, I think, judging from the
frequency of inline images appearing in the referrer logs I just looked at.  I
just went through the last few hundred (?) lines of a referer log for a server
where the root page "/" links to one inline image.  There were 28 refers to the
root page of the server.  26 of those 28 were immediately followed by a refer
from the home page to the inline image in it.  Many of these hits were probably
from Netscape 4.x.  The two times there was no referer for the inline image
could easily be people not loading inline images (or using lynx) or who clicked
on a link in the page before the image started loading.

If you break this behavior, you will make life much harder for web server
admins.
This is good info. Our motivation for halting referrer fields on inline images
was bandwidth based. we wanted to reduce the size of the HTTP request. However,
it appears we didn't put enough weight on the point you're now making. I guess
we just assumed the log admin receiving the referrer of the broken inline
wouldn't be using this info. Looks like we assumed incorrectly and we should be
sending the referrer header along w/ inline images.

Thanks for clarifying.
Assignee: gagan → don
Status: ASSIGNED → NEW
Component: Networking-Core → XPApps
In light of the discussion we have had here, it looks like the referrer field
decision really will have a tighter association with the UI. And so for more
details on who is going to add this where I am assigning this to webshell. (Does
 webshell know the difference between a link clicked or one typed in Location
bar?) Since I dont see webshell in the components list I am assigning this to
XPApps... Feel free to correct me.
Assignee: don → radha
Summary: NECKO: referer not sent → [NECKO] referer not sent
Radha, do we need to fix this in M9?
Answering and asking few questions....

1) I read thro' this and I am not sure why this is a UI thing.
2) webshell does know the difference between a link click and a url typed in the
urlbar. But only to some extent. After a couple of calls, they both go thro' the
same codepath down to Necko
3) If necko *really* needs to know whether the url load request occured because
of link click or otherwise, then  webshell needs to pass some info to DocLoader.
4) Jevering says that this *has* to be fixed in M9.
5) cc'ing warren to help clarify things.
I agree that the webshell needs to pass the referrer to the doc loader, but I'm
not sure how this worked before. The doc loader should define a callback that
the protocol (http) can call to get the referrer. Gagan should handle that
part.

Judson was going to look at the web shell code to fix up the default protocol,
www. and .com parts -- I would suspect that the distinction between the
location bar and clicking on a link would be coming into play in that same
general vacinity, so maybe he should handle that part.
Assignee: radha → valeski
Based on warren's comments, I'm reassigning this  to valeski. Let scc know if
you want some API modified in webshell since he is the current owner of it.

Note:
When the url is typed in to the urlbar, BrowserAppcore calls in to webshell
thro' LoadURL(). When a link is clicked, layout calls in to
nsWebShell::HandleLinkClick() which eventually calls LoadUrl().

cc'ing scc
Assignee: valeski → gagan
Assignee: gagan → don
I recently fixed post requests to send a referer field since most post servers
require referer field. Pls. use the same logic in whereever the inline URLs for
the imageas are being constructed. For more details look at
mozilla/layout/html/forms/nsFormFrame.cpp and grep for "Referer"
Whiteboard: does confusion still reign?
*** Bug 10099 has been marked as a duplicate of this bug. ***
Assignee: don → radha
Whiteboard: does confusion still reign?
Whiteboard: It's b-a-c-k! and Radha is investigating it ...
Status: NEW → ASSIGNED
Target Milestone: M9 → M10
OK. I spoke to Rick.  Here is what we came up with,

0) Since load groups get passed around to all types of necko customers,
it looked like a good centralized place to manage referer.
1) add a baseuri member to load group
2) Based on privacy preference settings(which webshell will know about), and
other deciding factors, webshell/docloader will set the base uri in the load
group.
3) And down in necko, this base uri will be used to set the referer field. The
point is, if the referer has to be sent, baseuri will be set, if not, baseuri
will not be set.
3) Other components like libimage has to do similar things for inline images.
when a image load request comes thro' does libimg know about the base url? maybe
pnunn can answer this.
4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?

adding rpotts,pnunn to cc list. warren, can you comment on this.

I don't believe this is a M9 blocker. So, moving to M10.
Sounds like a good plan. One question: Should the baseuri be the same thing as
the first URI inserted into the group?
Priority: P2 → P1
Summary: [NECKO] referer not sent → [FEATURE][NECKO] Send HTTP_REFERER value to server
Whiteboard: It's b-a-c-k! and Radha is investigating it ... → Radha is investigating
Summary: [FEATURE][NECKO] Send HTTP_REFERER value to server → [NECKO] Send HTTP_REFERER value to server
*** Bug 5408 has been marked as a duplicate of this bug. ***
Whiteboard: Radha is investigating → Waiting for input from rpotts
Whiteboard: Waiting for input from rpotts → HELP WANTED Waiting for input from rpotts
Whiteboard: HELP WANTED Waiting for input from rpotts → [NEED HELP] Waiting for input from rpotts
Whiteboard: [NEED HELP] Waiting for input from rpotts → [HELP WANTED] Waiting for input from rpotts
Assignee: radha → rpotts
Status: ASSIGNED → NEW
Rick, I'm reassigning this to you, so that it gets counted in your plate. Once
you have the loadgroup part working, you can re-assign back to me so that I
hook-up webshell to it.
Blocks: 13465
"4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?"

Referer need never be sent.  Any servers relying on referrer being sent are
flawed.

<opinion>I would consider it a bug, and a security flaw, were Mozilla to
transmit referer despite my preference indicating that it should not, assuming
such a preference setting existed.</opinion>
Example of server relying on referrer: http://www.intellicast.com/

Most of the weather images within the site can only be viewed if the referrer is
the page that contains the image.  It also has an ad.  Otherwise people would
link straight to the images.
Sounds like another candidate for bug #7380 use.
*** Bug 16404 has been marked as a duplicate of this bug. ***
*** Bug 14129 has been marked as a duplicate of this bug. ***
*** Bug 14128 has been marked as a duplicate of this bug. ***
Summary: [NECKO] Send HTTP_REFERER value to server → [NECKO][webshell] [beta] Send HTTP_REFERER value to server
Target Milestone: M11 → M13
m13
For the paranoid, one may consider including a preference option to disable the
sending of the referrer, but no matter what this preference should either be
enabled or if there isn't one, the referrer should be sent.

Currently, referrer is the *only* way for webmasters to ensure that images or
other copyrighted files are viewed/downloaded exclusively from their own pages,
and not pirated by someone else.

Secondly, there's the issue of backwards compatibility: all previous Netscape
versions sent HTTP_REFERER. Not doing so in Mozilla will break many sites that
depend on this behavior (also exhibited by IE, Opera, Lynx, and others).

Finally, Mozilla's going to be rejected by webmasters if it doesn't send
HTTP_REFERER not only because their CGI scripts or other referrer-dependent
protections will break, but for the lack of alternative solution (that is at
least remotely reliaible).
*** Bug 19054 has been marked as a duplicate of this bug. ***
*** Bug 18331 has been marked as a duplicate of this bug. ***
*** Bug 21794 has been marked as a duplicate of this bug. ***
*** Bug 22131 has been marked as a duplicate of this bug. ***
No longer blocks: 13465
*** Bug 20819 has been marked as a duplicate of this bug. ***
*** Bug 23319 has been marked as a duplicate of this bug. ***
*** Bug 19037 has been marked as a duplicate of this bug. ***
QA Contact: paulmac → tever
Target Milestone: M13 → M15
We definately need to deal with this all the way from the URI loader down
through the nsHTTPChannel...  I'll look at this when I get back.
Not for alpha, but eventually, you MUST make sure you never let non-HTTP

referers into the real world. This is a security hole, because non-HTTP

protocols can have all sorts of secret stuff in them, like my FS layout (in

file://) or which news articles I read (in news:// or nntp://) or my passwords

(ftp:// in some cases).

Unlike HTTP URLs, these other protocols are NOT expecting to be exposed to

random web logs owned by potentially hostile people. So, Mozilla must prevent

them from escaping into REFERER at all. No useful web pages will be broken by

this if it's implemented correctly.
*** Bug 16587 has been marked as a duplicate of this bug. ***
I disagree with ruth@innocent.com.  It can be very useful to see via the
referer when someone mentions my web site in a USENET posting.  I would hate
to see Mozilla unilaterally decide to block this in all circumstances.
Keywords: helpwanted
*** Bug 25257 has been marked as a duplicate of this bug. ***
*** Bug 25877 has been marked as a duplicate of this bug. ***
*** Bug 25977 has been marked as a duplicate of this bug. ***
*** Bug 25063 has been marked as a duplicate of this bug. ***
*** Bug 26510 has been marked as a duplicate of this bug. ***
Due to Beta indication in Summary, putting beta1 into keyword field.
Keywords: beta1
*** Bug 26663 has been marked as a duplicate of this bug. ***
Summary: [NECKO][webshell] [beta] Send HTTP_REFERER value to server → [NECKO][webshell] Send HTTP_REFERER value to server
Whiteboard: [HELP WANTED] Waiting for input from rpotts → Waiting for input from rpotts
RickP: can we get your input on this. Is it essential for beta?
This prevents use of some major sites, for example, http://www.intellicast.com/
(which is probably the best source of US weather radar on the Net).

Not fixing this will also annoy webmasters because they won't get the referrer
field in their logs.
I don't disagree that it would be USEFUL to a web master to see which news
articles I read; he'd probably also find my PASSWORDS and CREDIT CARD NUMBER
useful, but that's no reason to create potential security problems. Bugtraq has
seen several "interesting" things sent as referer URLs, and restricting them to
HTTP-only seems like the most conservative solution.

*** Bug 16003 has been marked as a duplicate of this bug. ***
still need input from Rick Potts to determine PDT status.
taking over from rick...
Assignee: rpotts → gagan
Putting on PDT+ radar for beta1.
Whiteboard: Waiting for input from rpotts → [PDT+]Waiting for input from rpotts
this bug is really annoying. all sites on altern.org are viewable as text only

with mozilla...

example: http://www.altern.org/bmxzine/



an interesting test is to load this page with Nav4.x (it should display

correctly), then load one of the images in a new window and then reload the

image : due to the fact no referrer is transmitted during the reload, you get a

permission denied, as in mozilla.



i guess that the server puts this restriction in order not to be used as a mere

"images server"...


for the accuracy, i must add that this server blocks only .jpg files with the
referrer trick, not .gif (don't ask me why)

Herve: No need to complain, it's on PDT+ radar which means it should be fixed
very very soon. :-)

PDT: Product Development Team. 
PDT+: A bug which MUST be fixed by beta1.

well, that's not really a complaint. just a way of submitting another "real life
example" of a server that won't give the images if no appropriate referrer is
transmitted.

Oh, right. In that case thanks!
fixed. waiting for review. 

Here is what the "network.sendRefererHeader" pref means-
0- Won't send the "Referer" header in any case. 
1- Will send it from links clicked on a document. 
2- Will send it for image requests from a page as well as (1). 

The default of this pref is 2.
Status: NEW → ASSIGNED
Whiteboard: [PDT+]Waiting for input from rpotts → [PDT+] ETA 02/22/00
fix checked in.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
verified referer header w/Trace32
NT 2000022408
Status: RESOLVED → VERIFIED
*** Bug 29264 has been marked as a duplicate of this bug. ***
*** Bug 34735 has been marked as a duplicate of this bug. ***
It looks like this bug may have reopened? See bug 34735 on M14 build 033112

Hey,
please tell me, if I am correct in this Bug.
Mozilla doesn't send a HTTP_REFERER when it loads a document
in an ILAYER.
Netscape and Microsoft send one.
I made a little testpage where I explain all details.
You can find it under: http://212.227.152.190/mozilla/

I hope I am correct in this bug. Else please tell me
the correct one.

Thanx,
Lars
lars@topsitez.de - Please file that as a separate bug report.
Reopening for MacOS platform.

Nominating for nsbeta2 since it crashes on MacOS9.
Status: VERIFIED → REOPENED
Keywords: crash, nsbeta2
OS: other → All
Resolution: FIXED → ---
Crashes on:
- MacOS 2000-05-16-09-M16 Commercial
Works fine on:
- Win98, Linux 2000-05-16-09-M16 Commercial
Upgrading severity
Severity: major → blocker
Hmmm... what crashes?  My test page?  If so, does it have anything to do with
this bug?

(Also retitling to remove my silly Apache-ism.)
Summary: [NECKO][webshell] Send HTTP_REFERER value to server → [NECKO][webshell] Send HTTP Referer field to server
please file separate bugs for the new problem(s). thx. closing this again.
Status: REOPENED → RESOLVED
Closed: 25 years ago → 24 years ago
Resolution: --- → FIXED
Hmmnn....lemme try it again.  I believe (and will confirm or deny this later) 
that the page crashes when loading.n -kritzer
Okay, my bad.  This bug does *not* cause a crash.

The bug still exists on the Mac.  The section of the page that deals with 
referer looks like this:

Referer: (none)

Which is incorrect according to this bug report.  Lastest mac bits have been 
flaky, so I'll check it one last time with today's (2000-05-18-xx-M16) bits.
When testing, make sure to click on the URL to get to the page, rather than type
it in (if you type in the url, (none) is correct).
Ahhhh...yes, well that's the trick, isn't it?  <grin>  Thanks for the clue, 
dbaron.
Marking VERIFED FIXED on:
 - MacOS9 2000-05-18-09-M16 Commercial Build
 - Linux6 2000-05-18-09-M16 Commercial Build
 - Win98 2000-05-18-09-M16 Commercial Build
Status: RESOLVED → VERIFIED
from: Michel Carroll <michel@swipe.plc.uk>

still not fixed in NN 6.1 AND Mozilla Build ID: 2001080110.

Note test bed: browser running on Win 2K sp2 client with Win 2K sp2 server
running ASP 3 on IIS5.

ASP O/P:

Request.ServerVariables("all_http"):
HTTP_ACCEPT:text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
HTTP_ACCEPT_LANGUAGE:en-us
HTTP_CONNECTION:keep-alive
HTTP_HOST:www.swipe.plc.uk
HTTP_USER_AGENT:Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.3)
Gecko/20010801
HTTP_CONTENT_LENGTH:33
HTTP_CONTENT_TYPE:application/x-www-form-urlencoded
HTTP_ACCEPT_ENCODING:gzip, deflate, compress;q=0.9
HTTP_ACCEPT_CHARSET:ISO-8859-1, utf-8;q=0.66, *;q=0.66
HTTP_KEEP_ALIVE:300
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Any particular conditions needed to reproduce?  I just put together a simple
test case and it seems to work fine; got the following request (Linux build
2001090408):

GET /test.html HTTP/1.0
Host: lintra:1355
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3+) Gecko/20010904
Accept: text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
Accept-Language: en-us
Accept-Encoding: gzip, deflate, compress;q=0.9
Accept-Charset: ISO-8859-1, utf-8;q=0.66, *;q=0.66
Connection: close
Referer: http://www/~nephtes/test.html
I'm resolving and verifying this bug.

Look for existing Referer bugs and file new ones. Please file off 0.9.4 (coming
soon, NS 6.1 or the daily trunk build.
Keywords: testcase
Summary: [NECKO][webshell] Send HTTP Referer field to server → [NECKO][webshell] Send HTTP Referer field to server, "network.sendRefererHeader"
This bug still partially exists in build 2001090908.

When following a link through a simple click, the Referer-Header gets sent
(expected behavior). However when right-clicking and selecting "Open Link in New
Window", _no_ Referer gets sent (Which is definitely wrong). You can test this
on the Link posted by dbaron@fas.harvard.edu
But I don't think this bug really is a blocker. Major or even normal would be
more appropriate.

And as for the privacy issues: RFC 2068 chapter 14.37 has it's own collection of
SHOULD's and MUST NOT's on this topic. However, I feel that the discussion of
these issues are out of scope for this bug, and should rather be posted as an
enhancement query. Implementing a preference to specify whether Mozilla sends
cross-site Referers might be a good idea though.
hey gagan,

i don't know if i should mention this or not - but i'm actually looking at the
issue of passing the referer header along when new windows are targeted :-)

If that is *really* what this bug has digressed to, then maybe i should take it
over...

But don't stick me with *all* the 'referer' bugs :-)

-- rick


P.S.
It might be better to open a new bug specific to the window targeting situation!  
Bug 48902. We stuck paul with that one.

I'm calling this one: RESOLVED/FIXED.

Please see my previous comment.
Status: REOPENED → RESOLVED
Closed: 24 years ago → 23 years ago
Resolution: --- → FIXED
re-VERIFIED, cause this has been working for ages (modulo the specific bugs
listed in the comments)
Status: RESOLVED → VERIFIED
Product: Core → Mozilla Application Suite
You need to log in before you can comment on or make changes to this bug.