55477 - UI-Pref to send HTTP "Referer" (referrer) always/never/only at same server

Status: RESOLVED FIXED

(Core :: Networking: HTTP, enhancement)

Description

[Matthew T (active 1999-2002)]

Mozilla should offer a user pref to send the HTTP referrer header (a) always, (b) 
never, or (c) only if the requested resource is at the same server as the 
originating resource.

Such an option is provided by iCab, and is useful for those who are more 
concerned with their own privacy than with the ability of Web site administrators 
to tell where their visitors are coming from.

Comment 1

[Ben Bucksch (:BenB)]

The pref is already there:

pref("network.sendRefererHeader",           2); // 0=don't send any, 1=send only
on clicks, 2=send on image requests as well

Are you talking about a UI? If so, please adjust the summary.

Comment 2

[Matthew T (active 1999-2002)]

If `// 0=don't send any, 1=send only on clicks, 2=send on image requests as well' 
accurately describes the pref, then it's not what I'm asking for. This RFE is not 
asking for the Referrer header to be sent based on the method of requesting the 
resource, but based on the relation of the resource to the place it was requested 
from.

Comment 3

[Gagan]

http bugs to "Networking::HTTP"

Comment 4

[Gagan]

not critical for 0.8

Comment 5

[Boris Zbarsky [:bzbarsky]]

*** Bug 69046 has been marked as a duplicate of this bug. ***

Comment 6

[Christian :Biesinger (don't email me, ping me on IRC)]

An UI for this preference would be very nice, I didn't know about it until my
bug was marked a duplicate (ok, I couldn't spell referrer when I searched if my
bug was already submitted...).

Comment 7

[Ben Bucksch (:BenB)]

> I couldn't spell referrer

You're not the only one - see the HTTP spec :).

Comment 8

[Darin Fisher]

This enhancement cannot be solved from within necko.  It needs an owner outside
of necko... perhaps in the uriloader?  -> mscott?

taking this off the radar for 0.9 since there is no owner yet.

Comment 9

[HJ]

I have an UI for this up and running. Take a look at my screenshot. I will dump
that after this comment. I can check-in, a.s.a.y.w.i

Comment 10

[HJ]

Attachment: Screenshot

Comment 11

[Andreas M. "Clarence" Schneider]

RFC 2616 strongly recommends us to fix this bug (section 15.1.3):

   Because the source of a link might be private information or might
   reveal an otherwise private information source, it is strongly
   recommended that the user be able to select whether or not the
   Referer field is sent. For example, a browser client could have a
   toggle switch for browsing openly/anonymously, which would
   respectively enable/disable the sending of Referer and From
   information.

Comment 12

[timeless]

can CAPS help?

Comment 13

[Ben Bucksch (:BenB)]

timeless, the Center for Analysis and Prediction of Storms
<http://www.caps.ou.edu>? Please provide a URI.

Comment 14

[timeless]

http://bugzilla.mozilla.org/describecomponents.cgi?product=Browser
Security: CAPS

Comment 15

[Ben Bucksch (:BenB)]

> Security: CAPS

OPS!
/me hides.

Comment 16

[HJ]

I think that this RFE is more about extending the current pref settings because 

(a) always = pref("network.sendRefererHeader", 2);
(b) never  = pref("network.sendRefererHeader", 0);

in the current implementation. So only (c) is missing at this moment, right? 
What about adding pref("network.sendRefererHeader", 3); for this??

Comment 17

[epu]

mozilla sends file:/ in Referer

One solution would be sending for the URL http://www.server.com/path/file.ext
Referer: http://www.server.com/path/

This doesn't causes problems with broken servers and maintains user anonymity.

Comment 18

[Jesse Ruderman]

[e-mail]: the problem that we currently send referers from file://
URLs seems to be covered by bug 83038.

Comment 19

[Boris Zbarsky [:bzbarsky]]

*** Bug 86949 has been marked as a duplicate of this bug. ***

Comment 20

[Jeremy M. Dolan]

> 1=send only on clicks
> 2=send on image requests as well

I don't see much point in differenciating between 1 and 2 there. Idealy, the
options would be:

 Always send
 Don't send cross-site (needs better wording for UI)
 Never send

This might want to go next to the User-Agent disable, UI wise, since they're
both privacy affecting headers. (bug 92716)

Comment 21

[slice1900]

A couple more options for this would be nice:

1) Have it "lie" and always show the top level site as the referrer (i.e., if
you request http://www.mozilla.org/foo/bar.html it'll send
http://www.mozilla.org as the referrer)  This would help fix some (but certainly
not all) sites that refuse to work properly if they don't like the referrer you
send.

2) Be able to set these options (always/never/lie) on a per site basis, similar
to the other security stuff.  Ideally it'd let you set things based on either
the site you are going to, or coming from.  I'd certainly love to disable
referrer when I'm coming _from_ google, I don't want sites knowing what my
search terms were that caused me to visit their site.  That kind of information
is just (ab)used for them to figure out how to make their site rank higher on
google.

#2 would be much more work to implement, but perhaps #1 would be pretty easy and
could be slipped in sooner (even without a UI to control it would be a wonderful
option to have, and much more practical than trying to browse without referrer
totally disabled)

Comment 22

[Jeff Thieleke]

Attachment: Patch to add additional referrer settings (no UI)

Comment 23

[Jeff Thieleke]

Here is some back-end code (no GUI) that I wrote to augment the current
network.http.sendRefererHeader preference settings, and also addresses some of
the above suggestions (along with a few of my own ideas).  Note that the
preference item isn't 'network.sendRefererHeader', as mentioned above...I think
that is a Netscape 4.x-ism.


My patch adds three new levels to sendRefererHeader, and the abilities to send
referrers only to the same domain and/or fake the referrer:

* "Send no referrer to 3rd party servers" (network.http.sendRefererHeader = 4). 

* "Send a fake referrer to 3rd party servers" (network.http.sendRefererHeader = 5). 

* "Always send a fake referrer" (network.http.sendRefererHeader = 6).

(the descriptions need rewording, but hopefully you get the idea)


An example of sending a fake referrer is say you are on
http://www.example.com/home.html, which loads an image from
http://www.foo.com/img.jpg.  Normally, foo.com would be sent the referrer
'http://www.example.com/home.html'.  Using sendRefererHeader levels 5 or 6 (send
fake referrer to 3rd parties), foo.com would instead receive the referrer
'http://www.foo.com', or nothing at all with level 4 (send nothing to 3rd
parties).  www.example.com and images.example.com would get the correct referrer
with levels 4 or 5, or just 'http://www.example.com' at level 6. 


Hopefully these three settings are flexible enough to maintain privacy, without
blocking access to servers that enforce referrer rules to block off-site access.
 While I typically block referrer altogether, I find level 5 (send fake referrer
to 3rd parties) particularly useful, since it sends the valid referrer to
servers in the domain you are visiting, and a valid looking referrer - without
providing any useful information - to 3rd party servers.

Comment 24

[Jeff Thieleke]

Here is some back-end code (no GUI) that I wrote to augment the current
network.http.sendRefererHeader preference settings, and also addresses some of
the above suggestions (along with a few of my own ideas).  Note that the
preference item isn't 'network.sendRefererHeader', as mentioned above...I think
that is a Netscape 4.x-ism.


My patch adds three new levels to sendRefererHeader, and the abilities to send
referrers only to the same domain and/or fake the referrer:

* "Send no referrer to 3rd party servers" (network.http.sendRefererHeader = 4). 

* "Send a fake referrer to 3rd party servers" (network.http.sendRefererHeader = 5). 

* "Always send a fake referrer" (network.http.sendRefererHeader = 6).

(the descriptions need rewording, but hopefully you get the idea)


An example of sending a fake referrer is say you are on
http://www.example.com/home.html, which loads an image from
http://www.foo.com/img.jpg.  Normally, foo.com would be sent the referrer
'http://www.example.com/home.html'.  Using sendRefererHeader levels 5 or 6 (send
fake referrer to 3rd parties), foo.com would instead receive the referrer
'http://www.foo.com', or nothing at all with level 4 (send nothing to 3rd
parties).  www.example.com and images.example.com would get the correct referrer
with levels 4 or 5, or just 'http://www.example.com' at level 6. 


Hopefully these three settings are flexible enough to maintain privacy, without
blocking access to servers that enforce referrer rules to block off-site access.
 While I typically block referrer altogether, I find level 5 (send fake referrer
to 3rd parties) particularly useful, since it sends the valid referrer to
servers in the domain you are visiting, and a valid looking referrer - without
providing any useful information - to 3rd party servers.

Comment 25

[Ben Bucksch (:BenB)]

> An example of sending a fake referrer is say you are on
> http://www.example.com/home.html, which loads an image from
> http://www.foo.com/img.jpg.  Normally, foo.com would be sent the referrer
> 'http://www.example.com/home.html'.

What, if I am on <http://www.example.com/home.html> and click on a link to 
<http://www.foo.com/home.html>? Normally, this would count as 3rd party. Also in
your new code?

Generally, thanks for the code! (Didn't have a chance to look at it yet.)

Comment 26

[Jeff Thieleke]

Yes, that would also be considered a 3rd party request.  So when you clicked on
the link at example.com,  foo.com would receive the referrer
'http://www.foo.com' (assuming you were using one of the modes that sends a fake
referrer to 3rd parties), or no referrer at all.

A more concrete example of why you might want to block or alter 3rd party
referrers (as given above by slice1900) is doing a Goggle search.  Since Google
uses GET to handle the search form, the search keywords are part of the URL and
thus part of the referrer.  Maybe you don't want that information being sent out
every time you click on a search result.  

Another possible scenario is let's say you log into a website which puts some
sort of personally identifiable information in the URL (username, account
number, etc.).  At this same site, doubleclick.net as a banner ad (or a web
bug...) displayed.  Unless you are blocking 3rd party/all referrers, or sending
a fake referrer to 3rd parties, doubleclick.net now has your personally
identifiable information.

Comment 27

[Kelson]

>Hopefully these three settings are flexible enough to maintain privacy,
>without blocking access to servers that enforce referrer rules to block
>off-site access.

Unfortunately, the "fake referers" prevent servers from limiting off-site access.

Take this example: earlier this month someone embedded one of my scenic photos
in his signature on a web forum.  It only displayed at 75x75, but he linked to
an 800x550 image, so every time someone looked at a thread he posted in, they
were downloading a 112 KB file from my server.

So I adjusted my server to block access to the full images if the referrer
didn't show my own site.  My bandwidth use then returned to normal.  (I could
have just renamed the file, but then I'd have to do it again the next time.)

The fake referer that sends the target site's own address would completely
bypass this by making it appear that the embedded image in the forum was
actually a request from my own server.  If Mozilla were a more popular browser,
and if this were a popular setting, I'd have the same problem all over again.  

Admittedly there are more foolproof (and complicated) ways to handle the
problem, and the people most likely to use this setting would probably also be
blocking third-party images, but I wanted to bring up an issue from a web
developer's perspective.

Just something to consider.

Comment 28

[Ben Bucksch (:BenB)]

kelson, from a user's perspective:

> So I adjusted my server to block access to the full images if the referrer
> didn't show my own site.

I do have referers off al the time, because they are an invasion to my privacy.
So, since nothing != your site, you would block me, even though I visit your
site normally.
It is exactly this short-sighted behaviour of websites why the fake referer is
useful.

Be glad that some users to have referers on so that you at least know which page
linked to your resources. Now, you can maybe do something against the linking
resource. *That* is a legitimate use of referers. And the fake referer option
doesn't diminish it.

Compare bug 86194.

Comment 29

[Jeff Thieleke]

[e-mail] makes a good case for the legitimate uses of referrer. 
Unfortunately, since Referer is user generated header, it is and has always been
open to manipulation.  So relying on it to protect your content is really just
'security through obscurity', since most users don't have the means to modify it.

It is hard to disagree with your particular situation, but Mozilla has a strong
history of favoring the user over the content provider, especially when it comes
 to the user's privacy.  

If it is any consolation, it is very unlikely that 100% of users would be
blocking or faking their referrer headers, so you would still be able to track
down the offending site and deal with it in some other manner.

Comment 30

[Daniel Veditz [:dveditz]]

Even with faked referrers the blocking described in comment #27 would still
accomplish its purpose: the luser referencing your image might set the pref and
view his own .sig, but the thousands of people reading his posts and
involuntarily flooding your server are not likely to have done so.

Comment 31

[Jeff Thieleke]

Attachment: Updated patch - now matches on same host instead of same domain

I've updated the patch to now match on the same host, rather the same domain.  


I originally wanted to be able to match on the same domain, based on the
mistaken belief that any host on the same domain would be controlled by the
same organization.  But this is clearly not a valid assumption, especially for
countries that use a common second level domain, such as .co.uk.  

So now the check for a 3rd party server is based solely on comparing hostnames,
similar to the handling of 3rd party cookies and images.  Other than this
change, and some minor code cleanup, the patch’s functionality is unchanged.

Comment 32

[Jeff Thieleke]

Attachment: Added a pref item to allow a fake referrer to be sent for disallowed URI schemes

I've added a preference boolean, network.http.referrerSchemeOverride, to enable
the user allow the sending of a fake referrer for URI schemes (file, resource,
etc.) that normally prohibit sending any referrer.

This allows items such as bookmarks to send a valid looking referrer, so you
may be able to bookmark URLs that would otherwise be inaccessible due to a
page's requiring a referrer header.  Since the referrer sent is based solely on
the target URL, there is no affect on user privacy.

Comment 33

[Jeremy M. Dolan]

Jeff, your patch has nothing to do with this bug, nor do I see a valid purpose
in lieing to a server like that.

Comment 34

[Jeff Thieleke]

Jeremy - 

Since you haven't read the above comments, please do so now.  Then let me know
if you have any questions.

Comment 35

[Jeff Thieleke]

Attachment: Referrer patch, with preferences GUI

This patch should be sufficient to resolve this bug, since it includes both the
referrer enhancements of my previous patches as well as the Preference user
interface.

Comment 36

[Jesse Ruderman]

>Relying on [the referer header] to protect your content is really just
>'security through obscurity', since most users don't have the means to modify it.

Referer-checking is not about protecting your content from your users.  It's
about protecting your bandwidth from other web sites that are too lazy to copy
your images onto their site.

>If it is any consolation, it is very unlikely that 100% of users would be
>blocking or faking their referrer headers, so you would still be able to track
>down the offending site and deal with it in some other manner.

If Mozilla gains a pref to lie about referers, the following will happen:
1. Shady porn sites will advise Mozilla users to configure Mozilla to lie about
referers, and then use copies of an image hosted elsewhere.
2. The victim porn hosts will block Mozilla just like they block requests not
containing referers.  (Why do porn sites block requests without referers?  IE
contains a bug that allows sites to create links that do not send referers when
followed.)

Besides, lying about referers doesn't do anything more for users than "Send
referers only within the same host".

Comment 37

[Ben Bucksch (:BenB)]

> Shady porn sites will advise Mozilla users to configure Mozilla to lie about
> referers

Does anyone read text at porn sites? ;-P

> IE contains a bug that allows sites to create links that do not send referers when
> followed.

Even in <img src> or just <a href>?

> Besides, lying about referers doesn't do anything more for users than "Send
> referers only within the same host".

It allows to follow deep linking to HTML pages, even if the target site doesn't
want to.

As for images, what about this situation: A site hosts images on a different
host than the HTML page (like images.aol.com). That image host blocks requests
with no referer (bad site!) or with a referer that is not from known-good sites.
Since the browser can't know the relationship between the 2 hosts, that is a
potential privacy leak and it blocks the referer. No images.

As a sidenote: While I usually don't like to care about dirty people like porn
hosters, Beonex Communicator ships with a privacy-protecting default setting.
But porn seems to be a much-wanted content type, so users will be upset, if half
the porn sites don't work (I wonder why I didn't hear complaints yet). That's
why a setting is needed that 1) protects the privacy to the maximum extend *and*
*at the same time* 2) works with almost all sites. And I think the faked referer
is a very good idea to achieve that, even though it might not be the nice way.

Thanks again, Jeff, for contributing that code.

If people disagree with the option given, you could just remove them from the
UI, but leave the backend in place. I will gladly take the risk of Beonex
Communicator being blocked.

Anyways, can anyone review the patch? I'd do it, but I don't know that code part
and are getting short on time, too. I'll do a quick review before inclusion into
B.C., but that can't count as r=.

mscott, any objections to reassigning the bug to Jeff?

Comment 38

[Jeremy M. Dolan]

> It allows to follow deep linking to HTML pages

Whatever that is...

Here's the deal. We should eaither send a correct referer, or none at all. If
unscrupulous third-parties want to make a gecko browser that lies, fine. The
only thing of concern in a legitimate browser is the privacy issue of whether or
not to send a referer header across hosts/domains, which brings up the same
problem from third party image cookie detection, how do you tell if a.b.c and
d.b.c are third-parties. Depends how c and possibly b is administered. So if the
pref is only to send referer to the same host (only garuntee), many sites may
break image loading, if their images are on static.domain.com or some such.

Comment 39

[Jeff Thieleke]

Jesse Ruderman writes:

> Besides, lying about referers doesn't do anything more for users than "Send
> referers only within the same host".

Sure it does.  Say you are browsing www.example.com, which serves images from
images.example.com and requires an valid example.com referrer.  If you have the
referrer set to 'same host only', images.example.com won't receive a referrer
and would deny access.  

There are plenty of other "legitimate" reasons why one would want to send the
modified referrer.  There is simply no reliable way to send a referrer to only
valid recipients, and as you can see, 'Same Host Only' is actually not much
better than never sending a referrer at all.  Whether you think it is ethically
appropriate or not, sending a modified referrer is really the only way to
maintain both the user's privacy as well as access to many websites.

Comment 40

[Jeff Thieleke]

Ben Bucksch writes:

First of all, thanks for your support.


> If people disagree with the option given, you could just 
> remove them from the UI, but leave the backend in place. 
> I will gladly take the risk of Beonex Communicator being 
> blocked.

I have been thinking about this, and I'm sure that Netscape/AOL wouldn't include
this patch in the next version of Navigator, for the valid reasons that some
people have already given.  I concur with your suggestion to remove the new
referrer settings from the UI for some browsers, and leave it in for browsers
(Beonex and hopefully Mozilla...) that favor the user's privacy over the
convienience of content providers.  It will be very easy to simply hide the new
referrer setting radio buttons, and I'll probably address this in the next patch.


> Anyways, can anyone review the patch? I'd do it, but I 
> don't know that code part and are getting short on time,
> too. I'll do a quick review before inclusion into
> B.C., but that can't count as r=.

Actually, as of the major check-in for bug #124042 (UTF-8 URIs), this patch is
likely completely broken.  I have been waiting for the smoke to clear from the
check-in before I update my code and go about reimplementing my changes. 
Nothing major, just switching string objects to those that support i18n characters.

Comment 41

[Jeff Thieleke]

Attachment: Updated for UTF-8 URI changes

Ok, I updated the patch so that it patches cleanly and works with the UTF-8 URI
changes.  

I also added a variable to pref-referrer.xul to hide the options that send the
modified referrer.  So it is possible to ship a browser that only displays the
standard referrer options (never, only on click, on clicks and images), as well
as the "Same Host Only" option.

Comment 42

[Jeff Thieleke]

Attachment: Updated to patch cleanly

Comment 43

[Christian :Biesinger (don't email me, ping me on IRC)]

comments about the patch:
Firstly, isn't the referer spelt referer?
Secondly, <outliner> was renamed to <tree>.

Instead of |nsCRT::strcasecmp(referrerHost.get(), host.get()) != 0| you should
probably use
|!referrerHost.Equals(host, nsCaseInsensitiveCStringComparator)|

+    nsIURI *refPtr = referrer;
Personally, I don't usually like such pointers...

Comment 44

[Vadim Berezniker]

"referrer" is the correct spelling.

The RFC incorrectly(?) used "referer".

Comment 45

[Christian :Biesinger (don't email me, ping me on IRC)]

Well, the question is, do we want to use the name from the RFC (the
authoritative source about HTTP headers), or from the dictionary?

Comment 46

[Jeff Thieleke]

> Secondly, <outliner> was renamed to <tree>.

Yeah, but only after I had already submitted my patch.  Thanks for the reminder,
though.



> +    nsIURI *refPtr = referrer;
> Personally, I don't usually like such pointers...

Any particular reason, in the context of how it is used in this patch?




> Well, the question is, do we want to use the name from the RFC (the
> authoritative source about HTTP headers), or from the dictionary?

I see no need to perpetuate the spelling mistake.  Referrer is the English
descriptive name, and Referer happens to be the HTTP header name.  If one were
localizing the UI to a non-English language, they would translate "referrer" to
the equivalent native word.  Leaving it as "Referer" would confuse everyone,
except for readers of the RFC, and I think they can figure it out...

Comment 47

[Christian :Biesinger (don't email me, ping me on IRC)]

>> +    nsIURI *refPtr = referrer;
>> Personally, I don't usually like such pointers...
>Any particular reason, in the context of how it is used in this patch?

well, no, not really, this line is probably ok

Comment 48

[Christian :Biesinger (don't email me, ping me on IRC)]

oh yeah, and why not move the frontend to
xpfe/components/prefwindow/resources/.../? The backend code isn't part of the
cookie extension, so the frontend shouldn't be either.

Comment 49

[Ben Bucksch (:BenB)]

I am currently merging the patch into the branch, of course it bitrotted again.

Comment 50

[Ben Bucksch (:BenB)]

Attachment: Basically same patch, as textdiff

Comment 51

[Ben Bucksch (:BenB)]

Attachment: UI

- I moved the UI to xpfe/components/prefwindow, because (as biesi said) the
  backend lives in netwerk/protocol/http/ and the whole thing has nothing
  to do with cookies or wallet.
- Added Groups to match the rest of the prefs UI
- Devided the explanation text, partly so that we can hide the
  fake header text, if that UI is disabled. Also looks better IMO.
- I think that "modified referrer" (previous wording) is by far to general. I
  used "fake referrer" now, which IMO describes it better. Also made clear
  that this violates the HTTP standard.
- Added explanation what referrers are at all. Not sure, if it's clear enough
  or how it could be made clearer.
- Cutted wording of some of the options, so they are all one-line now.
- Changed the "schemeOverride" (other schemes) pref UI from checkbox to radios,

  so I can make clear, that no referrer at all is sent, if the option is
  disabled. (I was not sure, if it sends nothing or the normal referrer, if
  disabled; in fact, I had to read the source to find out.)

BTW: I did not review the backend part, at least not completely.

Requesting review.

Comment 52

[Ben Bucksch (:BenB)]

-> default HTTP owner

Comment 53

[Ben Bucksch (:BenB)]

This time for real.

Comment 54

[Ben Bucksch (:BenB)]

Attachment: Screenshot of prefs panel

Comment 55

[Jeff Thieleke]

Thanks for the UI help, Ben.  

One concern - I think the word "fake" is a little too strong of a word and has
too many bad connotations.  I'm not really happy with "modified" either...so
maybe something like "privacy referrer", or "anonymous referrer"?

Comment 56

[Ben Bucksch (:BenB)]

It's deliberately wrong information designed to look correct. From the POV of
the site, we're making the referrer header completely useless to them, if that
feature is enabled. I don't think that "fake" is too strong for a standard
violation.
"privacy" - too vage
"anonymous" - It's so much not about anonymity. Only in very rare occasions do
you reveal identity through the referrer (which would not already be known).

Comment 57

[Ben Bucksch (:BenB)]

> It's so much not about anonymity.

eh, It's not so much about anonymity.

Comment 58

[Mitchell Stoltz (not reading bugmail)]

I'm not sure I like this "fake referrer" feature. We should definitley allow
fine-grained control over when referrers are sent, but I believe they should
either be sent as correct values or not at all. I'm going to get some more
opinions on that.

Comment 59

[Ben Bucksch (:BenB)]

Attachment: Screenshot of pref panel with fake options disabled

Mitch, I don't *like* it either, in fact I have a bad feeling, but I see no
better solution, see comment 37.
Does it help, if the UI for these options is disabled in Mozilla? Changing one
constant gives this UI.

Comment 60

[James Paige]

Privacy is very important, and a pref+gui to disable the referer is definately a
good thing, which I would use myself... but lying about the referrer is going
too far. Ommitting the referrer is privacy. falsifying the referrer is ... well,
just lying.

I think the sites that would block a blank referrer represent a *very small*
subset of sites, and they ought to be dealt with as evangelism issues, not with
a fake referrer header.

Of course I guess I would also be happy if you make the text of the last two
prefs in the gui BLINKING RED and add the words "Very Bad Karma" somewhere in
the description :)

Comment 61

[Jeremy M. Dolan]

"3rd party" should read "third-party". What did P3P call it?

The whole paragraph explaining what the pref does needs to be redone. I *know*
what HTTP Referer: is, and I can't even make sense of that wording.

Re: the bookmark section of the first screenshot. Do any browsers send a referer
for a bookmark? This might be nice for sites to be able to know, if we could
default such a thing to being on by default.

Referer: X-Mozilla Personal Toolbar Bookmark
Referer: X-Mozilla Bookmark

if that doesn't violate the spec. I don't think that's much of a privacy issue,
eaither. A normal cross-site referer is much worse, since it could show you as
coming from pornos-r-us.com, etc.

Also, I second/third/whatever the feeling that we not include a "lie" option.

Comment 62

[Ben Bucksch (:BenB)]

> The whole paragraph explaining what the pref does needs to be redone.

Feel free to do it. I couldn't think of a better wording.

> Referer: X-Mozilla Bookmark
> if that doesn't violate the spec

It does:
"The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained"
"The Referer field MUST NOT be sent if the Request-URI was obtained from a
source that does not have its own URI, such as input from the user keyboard."
(HTTP 1.1 spec, <http://www.w3.org/Protocols/rfc2068/rfc2068>,
section 14.37 on page 130)

Comment 63

[Jeff Thieleke]

(this is a lot of deja vu from the prior comments, but...)

1. Providing the modified/fake/privacy/anonymous/whatever referrer is the only
way to maintain both privacy and access to many websites.  If you don't like it,
there are the other options available.  In general, Mozilla would only be
"lying" to 3rd party sites, who shouldn't be receiving a referrer in the first
place (see RFC 2616).

2. (Optionally) sending the referrer for bookmarks is again necessary for access
to many websites.  Incidently, since the bookmarks do have an URI, sending a
referrer doesn't violate RFC 2068.  Mozilla just never used to send a referrer
because, since the URI's scheme was internal to Mozilla, it wasn't useful for
the website.  Also, sending the actual referrer or something like "X-Mozilla
Bookmark" betrays the fact that the user has bookmarked the website - a fact
that the user may not want to reveal.

Comment 64

[Jesse Ruderman]

I don't think it's useful to differentiate between links and images, because
links can point to images, and because javascript+frames can imitate a link to
an image.  Also, images are "protected" using a referrer check far more often
than links are.

Proposed pref UI:

When loading links and images, tell the server the referring [or linking] address:
( ) Never
( ) Only when the linking page is on the same server
(o) Always

That would be simple enough to understand quickly, and it would also be
consistent with the radio sets for image and cookies prefs.

Comment 65

[Jesse Ruderman]

In response to BenB's comment 37: you don't need the ability to fake referrers.
 There's already a setting you can use as the default for Beonex that would
respect Beonex users' privacy without breaking porn sites and without prompting
sites to block requests with "Beonex" in the user-agent string.  It's "Only
[send the linking address] when the linking page is on the same server".

Comment 66

[Jesse Ruderman]

I think we should consider removing the "Never" option.  Sending the referrer 
within a site does not hurt privacy because sites can keep track of what the 
user loads without using referrers.  Not sending the referrer within a site 
breaks a lot of sites (not just porn sites) and many users would not figure out 
why the sites are broken.

Comment 67

[Matthew T (active 1999-2002)]

In my opinion, the current patch is not of check-in-able quality. Briefly, the 
problems are as follows.
(1) It uses an entire prefs panel, which is far too much interface for such an
    obscure feature.
(2) It has three paragraphs of text at the start. This is a dialog, not a help
    file. Even one paragraph of text would be far too much.
(3) It has no controls outside the group boxes, which is an accurate predictor
    of the group box widget being misused. In this case `Referrer' is separated
    from `From bookmarks and other non-web sources', as if `From bookmarks and
    other non-web sources' is not to do with referrers, when actually it is.
I'm not picking on Ben in particular here; many contributors -- including 
Netscape employees -- make the same mistakes, but I point them out whenever I 
get the chance, and 35 wrongs don't make a right. (Filing bugs on other 
perpetrations of (1), (2), and (3) is left as an exercise for the reader.)

In line with comment 0 and comment 64, I suggest the following:

| | Scripts&Windows | | Tell Web pages _which page you came from    |
| |:Privacy:::::::::| | (send "Referer" information):               |
| | History & Cache | | (*) always ( ) if the same server ( ) never |

I've been fine-tuning this design for the past two years (yes, the `Referer' 
spelling is deliberate, and no, I don't think I'm dumbing it down too much), so 
I'm unlikely to change it in response to Bugzilla comments. But if you think 
Darin and Tom aren't getting enough bugspam, you're welcome to try. :-)

Comment 68

[timeless]

'Web pages'... I don't usually think of a picture as a webpage, would you 
consider 'web server'.

Comment 69

[Ben Bucksch (:BenB)]

> I don't think it's useful to differentiate between links and images, because
> links can point to images

I don't know the backend implementation, but the description of the pref is
probably inaccurate in an attempt to be understandable (it is the same
description in the default prefs file). I guess what really is being meant are
"embedded objects". (I tried to make that clear in the tooltip.) Jeff, can you
confirm that?
It might be worth to be added, because I can control link clicks, but I cannot
control loading of embedded objects (possibly from another server).

> I think we should consider removing the "Never" option.

No! It is "strongly recommended" by the HTTP spec.

> "Only [send the linking address] when the linking page is on the same server".

That doesn't work, if the linked images are on another server. I think, that's
not unusual in that scene (Not sure, though; and I'm not in the mood to check it
right now ;-P .)

> (1) It uses an entire prefs panel, which is far too much interface for such an
>     obscure feature.

Feel free to make a better design with the same options.

> (2) It has three paragraphs of text at the start. This is a dialog, not a help
>     file. Even one paragraph of text would be far too much.

First, the one to be used in Mozilla has one subheader and one paragraph of text.

I agree that it's too much text and not understandable enough. I said:
"Not sure, if it's clear enough or how it could be made clearer." and "Feel free
to [re]do it. I couldn't think of a better wording."

We can remove the subheader altogether.

I like your wording. Suggestion:

Mozilla (fakes disabled):
----
Tell Web servers which page you came from. This can harm your privacy by
allowing third parties to see which sites and pages you visited just before them.
+- Web -----
| o Never
| o Always
| o For all pages, but not for [embedded objects | images]
| o Only if the same server
+---
----
Beonex Communicator (fakes enabled):
----
Tell Web servers which page you came from. This can harm your privacy by
allowing third parties to see which sites and pages you visited just before them.
Using one of the faked referrer options violates the HTTP standard, but can help
you to protect your privacy, while still allowing access to sites which wrongly
require the referrer header to be present.
+- Web -----
| o Never
| o Always
| o For all pages, but not for [embedded objects | images]
| o Only if the same server
| o Send faked referrer, if not the same server
| o Always send faked referrer
+---
+- Bookmarks, emails etc.--- 
| o Never
| o Send faked referrer
+---
---

Legend: [either | or]

> (3) It has no controls outside the group boxes, which is an accurate predictor
>     of the group box widget being misused.

No, not in all cases. Option groups are grouped by group boxes (on some systems
even technically), so if the option group is the only content of a dialog, just
one group box is reasonable.

Also, it looks much better (compare the screenshots).

> In this case `Referrer' is separated
> from `From bookmarks and other non-web sources', as if `From bookmarks and
> other non-web sources' is not to do with referrers, when actually it is.

Correct, and that annoyed me, too. I couldn't come up with a better name for the
first group box back then.

> I'm not picking on Ben in particular here; many contributors -- including 
> Netscape employees -- make the same mistakes, but I point them out whenever I 
> get the chance, and 35 wrongs don't make a right. (Filing bugs on other 
> perpetrations of (1), (2), and (3) is left as an exercise for the reader.)

Yes, I am aware of that, that's a major part of the reason why I did it. I agree
that our use of group boxes in the pref panel generally sucks, but consistency
is also a value to consider and I think that the group box use is acceptable in
this case.

Comment 70

[Jeff Thieleke]

> I don't think it's useful to differentiate between links and images, because
> links can point to images

The difference is that links are directly initiated by the user (e.g. clicking
on an anchor), and images/embeds are usually automatically loaded when the page
loads, so the user doesn't have the same amount of control over when the
referrer is sent.

I didn't create this distinction - this is a holdover from Netscape
Communicator.  With the new option of only sending the referrer to the same
host, this distinction isn't as useful as it once might have been...  

I can't really think of a good reason to have both options from an end user's
perspective.  From a developer's perspective, some reasons might be that the
code is already in place, it is nice to have a GUI that fully represents the
available prefs options, and to maintain compatability with Netscape 4's
referrer options.

Comment 71

[Ben Bucksch (:BenB)]

Beonex Communicator 0.8-stable has been released with this patch, default 3rd
party fakes. I saw no problem myself, and I got no complains.

Comment 72

[Beau Bennett]

This issue has significant ramifications regarding e-commerce. From the
AuthorizeNet web site:
"Effective May 28, 2002, Authorize.Net will require all Merchants who connect to
the payment gateway via WebLink or ADC Relay Response to implement a Referrer
URL on their account. ANY MERCHANT WHO FAILS TO MEET THIS REQUIREMENT WILL BE
UNABLE TO PROCESS TRANSACTIONS AFTER MAY 28, 2002, UNTIL THEY ARE IN COMPLIANCE
WITH THIS NEW SECURITY MEASURE. IF YOUR WEBSITE CONNECTS TO THE AUTHORIZE.NET
PAYMENT GATEWAY VIA WEBLINK OR ADC RELAY RESPONSE AND YOU HAVE NOT SET A
REFERRER URL ON YOUR ACCOUNT, ALL YOUR TRANSACTIONS WILL BE DECLINED AS OF MAY
28, 2002."

Comment 73

[Beau Bennett]

This issue has a significant impact on e-commerce. AuthorizeNet is one of the
world's largest online credit card processors. From the AuthorizeNet web site: 

"Effective May 28, 2002, Authorize.Net will require all Merchants who connect to
the payment gateway via WebLink or ADC Relay Response to implement a Referrer
URL on their account. ANY MERCHANT WHO FAILS TO MEET THIS REQUIREMENT WILL BE
UNABLE TO PROCESS TRANSACTIONS AFTER MAY 28, 2002, UNTIL THEY ARE IN COMPLIANCE
WITH THIS NEW SECURITY MEASURE."

As a web developer specializing in e-commerce sites, this issue is huge.
Thanks.

Comment 74

[Ben Bucksch (:BenB)]

Beau Bennett, bug 141164 is specifically about that site. There, you can also
read that his site violates the HTTP standard by this measure.

Comment 75

[Ben Bucksch (:BenB)]

Rather, bug 141641.

Comment 76

[Daniel de Wildt]

Someone implemented a toolbar to spoof the referer.
You can find it at
http://refspoof.mozdev.org/

Comment 77

[Matt]

Instead of using the option of "same server", how about "same domain"? That
would keep user privacy intact while not breaking sites which require
appropriate referrers to load images but host said images on other servers. It
also resolves the "pages versus embedded objects" debate, because if the
automatically-loaded item was hosted by someone else the referrer would be withheld.

Comment 78

[Jeff Thieleke]

> Instead of using the option of "same server", how about "same domain"? That
> would keep user privacy intact while not breaking sites which require
> appropriate referrers to load images but host said images on other servers. It
> also resolves the "pages versus embedded objects" debate, because if the
> automatically-loaded item was hosted by someone else the referrer would be 
> withheld.

It is a great idea, but I don't think "same domain" can be made to work.  See
<http://bugzilla.mozilla.org/show_bug.cgi?id=55477#c31> for more details.

Comment 79

[Jeremy M. Dolan]

While comment 34 is true, unfortunately this means this referer setting may be
quite unusable. Consider all the www.foo.com sites that have their images on
images.foo.com. Many likely block no referers, so other sites don't use their
images and steal bandwidth.

Comment 34 also points out an even worse problem. Domain cookies can't tell
where they are valid for. I have a cookie set for ".co.uk", which would let
every .uk site share my data. Yikes.

Do the RFCs say anything about this, at least for cookies? Would it be out of
the question to include a listing of TLDs and how many levels of generality they
have?

com   1
us    1
uk    2
au    2
ca    1
# I think some are 3 levels too

Probably impossible to manage, since any TLD could change without notice. Just
an idea.

Comment 80

[Jeff Thieleke]

> While comment 34 is true, unfortunately this means this referer setting may be
> quite unusable. Consider all the www.foo.com sites that have their images on
> images.foo.com. Many likely block no referers, so other sites don't use their
> images and steal bandwidth.

Just to be clear, although at one point, there was a patch that had the "same
domain" feature, the latest code does not have this feature for the reasons
previously mentioned.  

This again points out why the fake referrer is so useful.  If example.com loads
images from images.example.com, and images.example.com needs a *.example.com
referrer, the fake referrer will send "images.example.com" and the page will
load normally.  "Same Host" will result in blocked images in this case.

That isn't to say that "Same Host" is a useless option - for most sites, it
probably would work fine - at least as well as the same host only options for
Images and Cookies.

Comment 81

[Jeremy M. Dolan]

> the fake referrer will send "images.example.com" and the page will load normally

No, in fact that's less likely to load then no referrer. It would obviously have
been faked, bacause "images.example.com" is unlikely to have links to those
images. A direct request (no referrer) would much more likely be allowed.

I imagine a lot of image sites are set to allow if no referrer, or if referrer =
one specific site that should be linking in.

Comment 82

[Ben Bucksch (:BenB)]

> It would obviously have been faked, bacause "images.example.com" is
> unlikely to have links to those images.

Indeed, that's what just came to me. That's a way to block people with fake
referrers enabled :-(. Let's just hope that webmasters are not that hostile and
engaged.

> That isn't to say that "Same Host" is a useless option

A slightly offtopic anecdote: I have once been hurt by the "same host" policy of
the image blocker. I had the same host policy enabled and used ebay, where a lot
of sellers link images on their own servers from the auktion pages on ebay. Of
course, that image was blocked. I ended up buying a GameBoy game when I wanted
to buy an N64 one. There was a tiny category name, which I overlooked. It would
have been obvious, had I seen the image.
Anybody needs WaveRace for GameBoy? ;-P

Comment 83

[Matt]

Something to consider here is that no scheme is going to be perfect. A given
non-conforming or restrictive site will always choke on one anti-referrer policy
or another. Therefore, schemes should be included that strike different balances
of privacy enhancement versus obstacles to browsing. Let the user decide. (This
is already what's happened, but "same host" is the only thing between all and
nothing. You don't have to get rid of "same host", just add a couple more choices.)

Could a "99%" version of a "same domain" scheme be offered? Drop the first
segment of the two server addresses, and if what remains matches you go ahead
and send the referrer. This is more inclusive than "same host", which means less
obstacles to browsing. In most cases it will prevent referrers being sent when
you switch sites, which enhances privacy. The exceptions for this scheme are:

- If you have loaded a page at a URL that contains only the domain instead of a
real server name (http://ebay.com), and the link you click also only contains a
domain (http://macintouch.com), the referrer will send.
- Sites that use sub-domains extensively and require referrers to load content
may choke. If a page at abc.fu.bar.com embeds an object from xyz.zoo.bar.com, no
referrer is sent.
- When a site is expecting a specific referrer from a site in a different domain
(see comment 72), a failure would occur with any referrer-blocking scheme.

Anyway, I believe the exceptions I listed will amount to a very small proportion
of browsing situations. Because the policy is less restrictive than "same host",
it means less user compensation is required. Any coders care to actually build
it? (I'm a non-coding freeloader bum. :-)

Comment 84

[slice1900]

There's a lot of possible scenarios being listed here.  I think we need to get
back to basics here and consider what it is we really need.  Thinking in terms
of privacy only, here are some situations I definitely do not want a valid
referrer sent:

* Coming from a file: or any other non http/https address
* Any time I type a URL in the address bar
* Any time I select a bookmarked site
* Any time I click a link from google or other search engines
* Any time I select something from history (including the back button)

Handling the first three is trivial, just send no referrer at all.  The fourth
is more difficult, and may require a blacklist of domains for which to never
send a valid third party referrer (just send nothing instead)  To handle the
fifth properly, I'd want the referrer that was sent when that page was
originally accessed resent (is it currently kept in history?  could it be?)

None of these should result in any problems at all.  Where we run into problems
is providing a global ban against third party referrers.  Perhaps we ought to
see about getting the items above (especially the first four) taken care of,
then move on from there.  If we wait to get a perfect and mutually agreeable
scenario (assuming one was possible) before anything is done, we'll be leaking a
lot of data about our browsing habits that web sites have no business having. 
I'm most particularly concerned about websites using google search terms (which
are encoded in the referrer link) to gather information I'd rather they not
have.  Personally I'm not so worried if a site widgetinfo.com has a link to
widgetsales.com, and they know that's how I found out about widgetsales.com. 
Privacy is good when the information can be used against you or in a way you
don't like, but privacy for no reason is pointless (anyway, there can always be
a paranoid 'no referrer ever' option for the tinfoil hat crowd)

Comment 85

[Kelson]

Regarding #4:

If someone wants to block search engine results from showing up, but doesn't
mind letting the site know where they came from, wouldn't it be easiest to just
strip off the query string?  That way the site knows someone found it on Google,
but doesn't know what the person was searching for.  No need to make a
blacklist, try to guess whether a site is a search engine, or anything else. 
And if they don't want the site to know they came from Google, well, thats what
the "block all third-party referrers" option is for.

Comment 86

[Chris Rebert]

I think we should let the user choose preferences like this:

------------------------------------------------------------
 |Referer Prefs
 |
 |            Case1  Case2 Case3
 |Action1      (o)    (o)    ()
 |Action2      ()     ()     ()
 |Action3      ()     ()     (o)


where the user can choose an action(i.e. fake referer) for each case (i.e. same
server page), and allow the user to assign an action to multiple cases.

Comment 87

[Stephen P. Morse]

Re: comment 79.  That cookie problem (domain = .co.nz) has been know for a long
time and it is unsolvable -- at least not without breaking major sites.  See bug
8743 for a lengthy treatise on this.

Comment 88

[Jeremy M. Dolan]

Thanks for the pointer, Morse. The pref discussed there--
network.cookie.strictDomains--seems to have been removed though. It seems rather
hopeless eaither way, though. Same would apply to this bug.

Comment 89

[Stephen P. Morse]

That pref has not been removed.  See nsCookies.cpp line 66 and line 1244.

Comment 90

[Susie Wyshak]

This is a comment from a top site regarding this feature:

Our website security systems verify that access is allowed by checking the
referrer. If this is left in the user’s hands, the user will probably not choose
to send it, and will then be unable to be authorized for access by a number of
systems. This feature would only be useful if it was either always on, or always
on by default and required a little extra sophistication on the user’s end to
find and change that setting.

Comment 91

[Jonas Jørgensen]

> Our website security systems verify that access is allowed by checking the
> referrer.

According to the HTTP spec, such behavior is broken.

Comment 92

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

According to common sense, that's broken.  Any malicious user can send a false
referrer.  (wget has a command-line option for it.)

Comment 93

[Darin Fisher]

mike, jonas: yup of course it's broken for websites to rely on the HTTP referrer
for anything meaningful, but _many_ sites do depend on it.  more importantly,
they depend on browsers having a consistent referrer policy.  that means, if
Mozilla doesn't send a referrer when IE sends a referrer, then Mozilla is going
to appear broken in the eyes of many users (they won't blame the websites), and
we really can't afford that IMO.

Comment 94

[Jeremy M. Dolan]

There was never any question that a pref that never sends a referer would break
many many (broken) sites. Even the pref setting to send referer to same site
will break some sites, as I described in comment 38. Netscape will very likely
not want to expose UI for this. Mozilla may not as well. But I, an apperantly
some others here would at least like a 'same host only' referer pref, and will
be able to deal with the sites that break.

ANYTIME we do ANYTHING different from the standard (ie, what IE does), there's
something somewhere that will break. Just look at all the evangelism bugs.

Limiting sending referer is a balance between breaking things and privacy.
Unfortunately it seems too many people here want that balance at too many
different places. Maybe Mozilla isn't the place for all these options.
Personally I'd just like 1 setting of not sending it on cross-host PAGE loads
(but still on images, since there's not much privacy lost there, but a high page
break probability). But I'm willing to concede maybe this is not something
Mozilla needs to be bothered with.

Similarly, the current setting of '1' for network.sendRefererHeader should also
likely be removed and treated as a 2. It's quite useless. 0 is also not very
practical but recommended by the spec. It might be useful it if was on a prefbar
type thing.

Comment 95

[Jeff Thieleke]

Attachment: Updated code to track nsIHttpChannel changes

Same great taste, less confict with current code.

Comment 96

[Curtis Jewell]

For those sites that are broken, would the possibility of sending only the host
part of the referring URI (either for 3rd-party changes or all the time) work?

Say Mozilla went to a page on www.mozilla.com from a Google JP search. With this
option, Instead of either "lying" and sending http://www.mozilla.com/, or
revealing everything or nothing, send http://www.google.co.jp/ WITHOUT the rest
of the URI (therefore not revealing search terms or location within the site.)

Comment 97

[Jeff Thieleke]

> Say Mozilla went to a page on www.mozilla.com from a Google JP 
> search. With this option, Instead of either "lying" and sending
> http://www.mozilla.com/, or revealing everything or nothing, 
> send http://www.google.co.jp/ WITHOUT the rest of the URI 
> (therefore not revealing search terms or location within the site.)


Sounds like a good option to me...I'll whip up a new patch forthwith.

Comment 98

[Jeff Thieleke]

Attachment: Add option to strip the referrer's path from 3rd party requests

I've added the option to strip the referrer's path for 3rd party requests:

//  6 - Send the actual referrer to all hosts, but strip off the path for 3rd
party requests.

Comment 99

[Jeff Thieleke]

Attachment: Updated to patch cleanly with nsHttpChannel.cpp v1.143

Excellent...SetReferrer() is getting closer and closer to my changes! :)

Comment 100

[Brant Gurganus]

For example of what can be found in a referer header, see:
<http://extremetracking.com/open;ref1?login=bob1647>.
There are many local links/mail links that give paths which include user names.
 This opens up the layout of a person's profile on a computer so that a cracker
knows exactly where to damage information.  You can even using the same site,
associate (with luck), the referrer with an IP address which increases the
chances of a successful crack.  I don't see any reason why referer headers even
need sent from local files/mail.

Comment 101

[Darin Fisher]

brant:

mozilla will only send http://, ftp://, gopher://, and in some special cases
https:// URLs as HTTP referrers.  there is an explicit whitelist.

Comment 102

[Paul Rubin]

Hi, a few comments:

1) Whatever bad karma comes from sending a deliberately false referer header
probably also falls on anyone who sends a deliberately false User-Agent header
(i.e. configures Mozilla to claim that it's MSIE).  And yet it's often to the
user's benefit to do this, so Mozilla provides a UI feature for the precise
purpose of falsifying the User-Agent header.  So those whose sensibilities are
offended by falsifying Referer should also be arguing to remove the User-Agent
preference UI.  After all, look at all those browser popularity statistic
gathering that gets messed up.  Yet I think most people think being able to
configure User-Agent is a good thing.

2. I don't understand comment #72.  Merchants have no control over whether
customer browsers send referer headers or not.  Whatever agreement the payment
processor has with the merchant is not really the customer's concern.  If the
customer's payment is declined, the customer will probably just buy the same
thing from a different merchant.  It's up to the merchant to slap some sense
into the payment processor.  Any payment processor who thinks referer headers
are a "security" feature needs a smack of a clue-by-four anyway.

3. Mozilla 0.9.2 was obnoxiously aggressive about sending referer headers from
non-web resources, including from bookmarks, from javascript, etc.  I'm using
1.2.1 now but haven't tested if it's better.

4. One type of resource not mentioned in this thread is javascript.  If I have a
link on a page with href="javascript:window.open(some_url)", some_url should NOT
receive a referer header and MSIE doesn't send one.  Moz 0.9.2 sent a referer
and I think that's a bug.  

5. I agree with those who want the behavior of: send a referer if the referer is
a web url on the same site as the target url.  Otherwise, don't send one.  I'm
indifferent to whether there's a button to send fake referers, i.e. I don't
especially want it but I don't oppose it and I disagree with the "bad karma"
argument opposing it.

6. The really winning (but maybe complicated) fix for this is to allow running
user-provided Javascript on every browser navigation, that can modify all the
outgoing http headers as it chooses, sort of like MSIE Browser Helper Objects
(BHO).  That would allow combining all the UI issues of cookies, referer
headers, user-agent strings, etc. into a single interface that XUL applications
(like the  xulplanet Preferences Toolbar) could operate.  See my comment in bug
# 79515 for a little more about this.

Comment 103

[Paul Rubin]

Oops, previous comment was supposed to refer to bug #75915 regarding a BHO-like
inerface, not 79515 (typo).  Sorry.

Comment 104

[David Aznar Reguero]

This is a security bug and there's a patch that fixes it.

What is blocking this patch from being applied?

Comment 105

[Ben Bucksch (:BenB)]

Attachment: Older patch, but applying against mozilla 1.4

Duh. If I only remembered the new patch here in this bug. I recently merged my
1.0 patches with 1.4 and this bug was one of them (and the way hardest to
merge, as always :-( , so could we please get the non-UI part in the tree?).
for this reason, it bases on an older version (IIRC attachment from
2002-04-18), not the current one (attachment from 2002-11-20). However, I think
the "current" one bitrotted again, so we now have 2 branches/forks of the patch
*sigh*. Could we please get the fix for this bug (I don't care about which
version, take the better one) into the tree, so save us from the mess? (oh, I
said that already.) I leave it up to Jeff Thielke to sort out the
differences/merges, sorry Jeff.

Comment 106

[Ben Bucksch (:BenB)]

Attachment: Older patch, but applying against mozilla 1.4

ops, left in some irrelevant Beonex changes.
I only tested the "fake referrer to 3rd parties" pref, and that seems to work.

Comment 107

[Jeff Thieleke]

Attachment: Patch applies cleanly as of 07/07/03

Same old patch, applies to the latest Mozilla.

I obsoleted Ben's patch (id=127034), since it is based off an older version. 
This is the latest code, applies cleanly against Mozilla as of 07/07/03, and is
fully tested. If someone were going to commit the referrer privacy feature,
this is the patch you'd want...

Comment 108

[amano]

what about a GUI for firebird?

Comment 109

[Matthias Versen [:Matti]]

Comment on attachment 127143 [details] [diff] [review]
Patch applies cleanly as of 07/07/03

ask the Firebird people about a UI for Firebird.

Comment 110

[amano]

I filed bug #  216596 asking for adding a GUI privacy option to change the
referrer settings in firebird. The bug was confirmed and classified a future
enhancement.

Comment 111

[amano]

Is there any development with this bug or are all folks waiting for Darin
Fisher's review/superreview?

Comment 112

[Ben Bucksch (:BenB)]

The patch is done and working fine, tested by (tens of?) thousands of Beonex
users and me since over a year. It's waiting for review, so that it can be
checked in. Which is what the status whiteboard says, BTW.

Comment 113

[amano]

Mike Tigas (aka MadmanNova) applied this patch into his MNG enabled nightly
firebird builds. So from now on firebird users can use and test the patch.
Testing by doing, rather than reviewing (wow, that rhymes ;) ).

See the thread: http://forums.mozillazine.org/viewtopic.php?t=36797
Download here: http://www.missouri.edu/~msa0288/MozillaBuild/
These builds are built automatically on a daily basis.

Works fine for me BTW. Thanks to Jeff and Ben.

Comment 114

[amano]

@Darin Fisher:

Is there any chance that there will be some comment from you regarding this patch?

I think that the work that Ben Bucksch and Jeff Thieleke put into creating this
patch deserve at least a r/sr- or a wontfix. In my eyes the current pure
ignorance is rather disrespectful.

Comment 115

[Darin Fisher]

My appologies for not reviewing or providing much of any feedback on the patch.
 This bug has been in my future queue (low on my priority list), no one has
pressed me to do anything with this patch, and I've been terribly swamped with
other work :-/

That said, I will try to look at this patch during the 1.8 alpha development cycle.

Comment 116

[amano]

Thanks Darin for the update.

If you want a Win32 build to test the practical use of this patch, I can send
you a 2004/01/22 firefox build that includes this patch (the last one that was
built by Mike Tigas). Just send a mail to me.

Comment 117

[mozillaspam.sender.bluetack]

(In reply to comment #116)
> Thanks Darin for the update.
> 
> If you want a Win32 build to test the practical use of this patch, I can send
> you a 2004/01/22 firefox build that includes this patch (the last one that was
> built by Mike Tigas). Just send a mail to me.
> 
Do you happen to by chance have a 0.9 Firefox version? I'm interested in trying
it out.

Comment 118

[amano]

A 0.9 Firefox version with the patch manually applied can be found here:
http://www.pryan.org/mozilla/firefox/amano/Fx-0_9-O1-Gopher-MNG-Exp.exe

Go to this google site
http://www.google.com/search?num=20&hl=de&ie=UTF-8&q=browser+referrer+test&btnG=Suche&lr=
and choose the second link (asp.flaaten.dk/test/atest.htm). This is a test site
that reports the referrer. It always reports should report google, unless you
change the "referer" setting.

Comment 119

[mozillaspam.sender.bluetack]

Works great! Thanks!

Had to change network.http.sendRefererHeader to 6 before it worked like you
said. Before that it was at 0. Are the settings as follows?

* "Send no referrer to 3rd party servers" (network.http.sendRefererHeader = 4). 

* "Send a fake referrer to 3rd party servers" (network.http.sendRefererHeader = 5). 

* "Always send a fake referrer" (network.http.sendRefererHeader = 6).

The last was later replaced with 

//  6 - Send the actual referrer to all hosts, but strip off the path for 3rd
party requests.

For what's it's worth, I'll be using 5 most of the time

In the past I've always used proxomitron's default referrer blocking which I
believe is closest to the old setting 6 (always fake) . It fakes it slightly
different from your patch though

I believe it sends back exactly the same referrer as the page being requested.
So if you are visiting  http://www.foo.com/example.html from
http://www.test.com/index.html, it fakes the referrer as
http://www.foo.com/example.html

Not sure if it's any better or worse than sending http://www.foo.com as it is now.

I don't think a UI is realistic to expect in Firefox for this, it seem  to me
faking referrers looks like one of those advanced options. I'm just happy for it
to be in the core.

Comment 120

[derbyshire2]

network.http.sendRefererHeader 5 does not work in Firefox 0.9.

I set this using about:config, exited and restarted Firefox, went to
http://www.google.com/search?num=20&hl=de&ie=UTF-8&q=browser+referrer+test&btnG=Suche&lr,
and followed the 2nd hit and the full Google URI was sent, incorrectly, despite
the setting. about:config shows it still set to 5, so I didn't somehow lose
instead of saving the prefs change either.

This, coming so soon after hearing that config.trimOnMinimize was added to the
core and finding it also not working in the publicly available Firefox 0.9, is
very disappointing. Are these fixes ever propagated to end-users that simply
download the browser from the Web site and expect it to be reasonably
up-to-date, or are they only ever available to people who use CVS repositories
and other arcane stuff?

Comment 121

[amano]

Neither this referrer patch nor the config.trimOnMinimize patch are included in
the public 0.9 release. For the first, you have to stick to my manually patched
builds, for the latter you have to use current trunk nightly builds.

Comment 122

[derbyshire2]

This is ridiculous. Both of these bugs have been known for bloody years; both
have patches that either fix the problem, or provide a workaround, and have had
for months. Why is the publicly available browser not up to date with fixes
developed months ago for bugs reported years ago? I thought it was Microsoft
that was bad about taking ages to release fixes for bugs, but this takes the cake.

I don't want to have to sign up for some sort of developer level access just to
be able to browse with all of the fixes already made for bugs years old, and
install the hundreds of megs of development environment needed to compile the
sources.

There's no point in fixing bugs if the fixes are not made available to the
general public, but instead kept within some "inner circle" of developers for
which you have to at least pay a steep price in disk space and download times,
if not money or something.

I wouldn't be so annoyed, but both these bugs have been in the database for
years, and the patches for nearly as long. There's simply no excuse for them to
be so slow in becoming available to end-users.

Once a patch has been supplied that's tested and works it should be available as
a bugfix update to the publicly available executables *immediately*. I want you
to enact such a policy retroactively, which means I want to see an updated
firefox 0.91 executable on your main download site within seven (7) days that is
current as regards bugfixes for which patches have been submitted and tested and
proven to work. Particularly for patches that have no effect if the user doesn't
fiddle about in about:config there is no excuse for not bundling them into the
very next public release.

Right now, my feeling is that the publicly-available browser and, probably,
other apps like thuderbird are lagging a full year, if not more, behind what's
actually *tested and working*.

If you're really worried about some obscure problem making a bugfix backfire
then at least offer public betas that are well ahead of the current public
releases! A year is an eternity in Internet time! Stop keeping all the yummiest
bugfixes to yourselves and let the general public have them for crying out loud
-- especially for showstoppers like the browser hanging when unminimized, and
serious privacy holes like this bug attempts to address!

Comment 123

[Matthew T (active 1999-2002)]

If you want a patch to be checked in, find someone to review it. Bugzilla is not an 
appropriate place to find a reviewer, because such people have often turned off 
their Bugzilla notifications. They do that so they won't have to wade through 
unhelpful and uninformed rants such as the comment immediately above. If you're not 
going to be part of the solution, please at least avoid being part of the problem.

Comment 124

[derbyshire2]

The developers' internal affairs are not my concern -- I'm an end user. Having a
product in front of me that works is my concern. Making that product, and making
fixes available to the general public in a timely manner -- that is your
concern. Also, four hours of your seven days are now up.

Comment 125

[mozillaspam.sender.bluetack]

> Neither this referrer patch nor the config.trimOnMinimize patch are included in
> the public 0.9 release. For the first, you have to stick to my manually patched
> builds, for the latter you have to use current trunk nightly builds.

Thanks for your excellent work. I was looking for 0.9.1. Until this bug is
fixed, those of us who like this feature will have to depend on your generousity
in helping.

Thanks again,

Comment 126

[Jeff Thieleke]

As the author of this patch, I share many of [e-mail]'s
frustrations, although he/she is way off base with the rants, quasi-threats, and
time limits.

The facts are that this patch is substantially unchanged in over two years, it
has been included in, and well tested by, several browser forks and private
builds, and it has been reviewed and is simply awaiting Darin Fisher's
superreview (see his April 5th post).  Yet in two years, no one has seen fit to
spend the 15 minutes it would take to review and commit this change, or
otherwise make suggestions for improvement. 

At one point, I was very enthusiastic about Mozilla/Phoenix/Firebird/Firefox
development.  But basically due to situations like this, I have given up
contributing patches to Bugzilla, and Mozilla development in general except for
keeping my small set of patches up-to-date.  I'm sure everyone is terribly
heartbroken, but that is my story...

Comment 127

[Sven Krohlas]

@Jeff write Darin a polite reminding mail, I'm sure he'll find the time then...
Or ask him for someone else to review the patch, if he hasn't.

Comment 128

[David Aznar Reguero]

This is very interesting, because it's true.
Mozilla development seems to have stopped, at least from my point of view

Maybe Mozilla needs an automatic and democratic patch approval system
and not just some dictators elected by nobody knows who, that just block
user accepted patches for unknown reasons.

Do you think this development model will scale in the future?
Or will this development model kill mozilla?

Comment 129

[mozillaspam.sender.bluetack]

(In reply to comment #128)
> This is very interesting, because it's true.
> Mozilla development seems to have stopped, at least from my point of view
> 
> Maybe Mozilla needs an automatic and democratic patch approval system
> and not just some dictators elected by nobody knows who, that just block
> user accepted patches for unknown reasons.
> 
> Do you think this development model will scale in the future?
> Or will this development model kill mozilla?


Wrong place to discuss this don't you think? If I didn't know better, I would
have thought, people are trying to mess up this bug with irrelevant comments
just to decrease it's chances of being approved.

Comment 130

[Darin Fisher]

Comment on attachment 127143 [details] [diff] [review]
Patch applies cleanly as of 07/07/03

>Index: mozilla/netwerk/protocol/http/src/nsHttpChannel.cpp

>+#define REFERRER_NONE                0   /* Never send the referrer */
>+#define REFERRER_USER_ACTION         1   /* Actions directly initiated by the user (e.g. clicking on a link) */
>+#define REFERRER_INLINE_CONTENT      2   /* Images or other inline content */
>+#define REFERRER_NON_HTTP            3   /* NOT USED - remains for backwards compatability */
>+#define REFERRER_SAME_HOST_ONLY      4   /* Send the referrer only for requests from the same host, otherwise send no referrer. */
>+#define REFERRER_3RDPARTY_PREPATH    5   /* Send the referrer only for requests from the same host, otherwise send target URI's pre-path as the referrer. */
>+#define REFERRER_3RDPARTY_NO_PREPATH 6   /* Strip off the path from the referrer for 3rd party requests, otherwise leave it alone. */
>+#define REFERRER_PREPATH_URI_ALWAYS  7   /* Always send the target URI's pre-path as the referrer. */

these shouldn't be lumped into one pref.  what if i wanted both USER_ACTION and
SAME_HOST_ONLY?  this makes those mutually exclusive and that doesn't make any
sense to me.

perhaps these could be expressed as flags instead that can be bit-wise or'd
together to select a referrer mode.

Comment 131

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Should we make the pref handling deal with a comma-separated list of actual,
meaningful strings, instead of bitflags?

"user_action,same_host_only"

Though we'd need to figure out if that sent the referrer in the union of those
sets of events, or the intersection.

(Paul Derbyshire, is that you?)

Comment 132

[Darin Fisher]

the pref value doesn't need to be user-friendly.  i think we should minimize the
footprint of this patch as much as possible... that is if it is to be included
in necko...

which, come to think of it... this feature could probably be implemented as an
extension.  we have nsIHttpChannel::referrer, which can be read and then set. 
why not listen for "http-on-modify-request" and then perform the referrer
modifications there?  (see the livehttpheaders extension to learn how to
intercept this event.)  then development of this feature would not be
constrained by waiting for feedback from core mozilla developers such as myself.

i personally don't see our default referrer policy changing if this patch were
to land, so this patch would likely impact only a very limited number of mozilla
users.  i doubt firefox devs would accept UI for this feature... unless...
sometimes it is easier to start with an extension, prove the concept, and then
on the heals of that momentum, roll the feature into the default mozilla builds.
 or at least that's how it is supposed to work.

Comment 133

[Ben Bucksch (:BenB)]

darin, I am using the patch in my browsers and have to merge it with new
versions all the time, so I would appreciate it going into Mozilla directly.

Comment 134

[Darin Fisher]

> darin, I am using the patch in my browsers and have to merge it with new
> versions all the time, so I would appreciate it going into Mozilla directly.

But what about the browsers Mozilla.org ships?  Those are my primary concern. 
Firefox has one answer with its extension system.

If someone develops an improved patch for Necko, I will definitely consider it.
 I'm not saying this has to be done as an extension.  I'm just saying that it is
an option that would allow the developer(s) of this feature to move at their own
pace.

Moreover, doing this as an extension doesn't preclude it being added to Mozilla
later on.  In fact, if it is successful as an extension, then that would only
help it find its way into the default Mozilla builds.

Comment 135

[Jeff Thieleke]

(In reply to comment #130)
> (From update of attachment 127143 [details] [diff] [review])
> >Index: mozilla/netwerk/protocol/http/src/nsHttpChannel.cpp
> 
> >+#define REFERRER_NONE                0   /* Never send the referrer */
> >+#define REFERRER_USER_ACTION         1   /* Actions directly initiated by
the user (e.g. clicking on a link) */
> >+#define REFERRER_INLINE_CONTENT      2   /* Images or other inline content */
> >+#define REFERRER_NON_HTTP            3   /* NOT USED - remains for backwards
compatability */
> >+#define REFERRER_SAME_HOST_ONLY      4   /* Send the referrer only for
requests from the same host, otherwise send no referrer. */
> >+#define REFERRER_3RDPARTY_PREPATH    5   /* Send the referrer only for
requests from the same host, otherwise send target URI's pre-path as the
referrer. */
> >+#define REFERRER_3RDPARTY_NO_PREPATH 6   /* Strip off the path from the
referrer for 3rd party requests, otherwise leave it alone. */
> >+#define REFERRER_PREPATH_URI_ALWAYS  7   /* Always send the target URI's
pre-path as the referrer. */
> 
> these shouldn't be lumped into one pref.  what if i wanted both USER_ACTION and
> SAME_HOST_ONLY?  this makes those mutually exclusive and that doesn't make any
> sense to me.

Values <= REFERRER_NON_HTTP are for backwards compatability only.  And the only
reason to differentiate between a user action and not (i.e. images) is because
Mozilla is currently unable to do anything special with the referrer other than
not send it.  

When you are able to control what referrer is sent, and more importantly, to
which server/domain, there is no need to differentiate between a user and page
initiated event.


> i personally don't see our default referrer policy changing if this patch 
> were to land, so this patch would likely impact only a very limited number
> of mozilla users.

Why not?  As was noted early in this bug, RFC 2616 (Section 15.1.2) notes that
the current Mozilla referrer behavior can be dangerous.  There are plenty of
examples in this bug report as well.  REFERRER_SAME_HOST_ONLY or
REFERRER_3RDPARTY_NO_PREPATH, as the default referrer level, would both be
appropriate for general use and then *all* Mozilla users would benefit from an
increase in privacy. 


> sometimes it is easier to start with an extension, prove the concept, and 
> then on the heals of that momentum, roll the feature into the default 
> mozilla builds. or at least that's how it is supposed to work.

There is no need to prove this concept - it has already been included in many
Mozilla forks and private builds.  It has been proven for going on two years now...

Since the referrer header can be both a privacy and security issue, it needs to
be part of the Mozilla core, just as surely as remote server image and cookie
blocking should be part of Mozilla, and not some optional extension.

Comment 136

[Hixie (not reading bugmail)]

So long as this doesn't (by default) stop people who are browsing blogs from
sending the full referrer URI when they click from one blog to another. It is
imperative that we don't screw over the blogging community by breaking the
extensive referrer tracking systems that have been set up to track discussions.
Bloggers are amongst our most vocal advocates.

Also note that most people don't care about this supposed "privacy leak", and
some would go even further -- I, for example, actively _want_ to be sending
referrer data. Don't try to "save" people by breaking their community!

Comment 137

[Matthew T (active 1999-2002)]

(Weblogs generally gave up on referers years ago, because of pornographic referer 
spam, in favor of services like Technorati. Nevertheless, Mozilla probably shouldn't 
*default* to a more private setting until we learn how much of the Web it breaks.)

Comment 138

[Aaron Kaluszka]

That would certainly break TONS of sites.  Many, many servers (especially free
hosting) use the REFERER to check for offsite linking of images (such a function
is even described in the Apache documentation).  A lot of online magazines also
use it to tell if you've come from a search engine or their own website and
render diferent content (perhaps based on their search keywords), for example. 
While the defaults might not break this directly, they most likely would for
servers that have a list of "okay" REFERERs (or perhaps caches/multiple server
front-ends) rather than using only the current domain name.  These are just a
few examples of how "most users" are going to be affected.  If you want a broken
Internet by default, that's fine, but don't make it broken for everyone by default.

Comment 139

[Jeff Thieleke]

> That would certainly break TONS of sites.  Many, many servers (especially free
> hosting) use the REFERER to check for offsite linking of images (such a function
> is even described in the Apache documentation).  A lot of online magazines also
> use it to tell if you've come from a search engine or their own website and
> render diferent content (perhaps based on their search keywords), for example. 
> While the defaults might not break this directly, they most likely would for
> servers that have a list of "okay" REFERERs (or perhaps caches/multiple server
> front-ends) rather than using only the current domain name.  These are just a
> few examples of how "most users" are going to be affected.  If you want a broken
> Internet by default, that's fine, but don't make it broken for everyone by
default.

The merits of referrer privacy have been discussed ad nauseam in this bug
report, but let me just assert that I have been using this feature for over 2
years now, and over the course of that period, I have had almost no problems
using REFERRER_3RDPARTY_PREPATH (*far* fewer problems than blocking 3rd party
cookies).  

That said, I wouldn't have a problem with leaving the default value the way it
is - just don't use the "we aren't going to change the default value so this
patch isn't necessary" excuse.

Comment 140

[Christian :Biesinger (don't email me, ping me on IRC)]

this is not fixed. reopening bug.

Comment 141

[Ben Bucksch (:BenB)]

The question of default should not be covered by this bug, but I think the HTTP
spec alone (in addition to actual need) gives enough reason to say that this
pref should be in Mozilla anyways, *at least* as backend pref.

I need to merge this patch with Mozilla 1.7.x. Maybe I can find the time to
adjust the patch to darin's demands, but the only point for me would be to get
it into Mozilla, so I need to understand what exactly darin wants (I currently
don't) and that he is generally OK with this.

Comment 142

[Darin Fisher]

What is needed is for someone to own the patch.
 
I explained above that I do not think it is correct to overload the existing
preference.  I would recommend that someone interested in owning this patch
address that issue first.  I can't promise that I won't find other issues with
this patch.  Code development is an iterative process.  If you per se want this
patch to go into Mozilla, then step up to the plate and drive the patch.

Comment 143

[Anish Goenka]

Well, I think the default should simply stay the way it currently is(sending the
correct referer).  We should let the end-user decide if they want to change
anything and if so, in what way.  As this will be a presumably hidden preference
for the more 'advanced' users, I dont see this as a problem.  As a result, all
the preferences mentioned above should be available as a precautionary measure.

Comment 144

[Ernesto Bricca]

i read through the comments and ordered my impressions by listing the made
suggestions in a flag list as proposed by darin
(about the comment #130 .. the missed option would read like ar:always;er:same_host)
didnt know where to place the cut_away_cgi_arguments ..

about the handling of the feature i think it should be only visible in the prefs
aka about:config
.. as ex with the trivial cases 0-7 and an 8 for the flaglist or a complete
flaglist switch 
.. or as said a recoding as a .js component though wondering myself how to
elegantly distinguish ar from er  ..

------------

action referer .. (ar) // if user clicks on a link
never .. same_host .. always

inline/embedded content referer .. (er)
never .. same_host .. always

// fake and blacklist referer behaviour are if enabled the counterparts of ar
and er like:
// always .. not same site .. never   // else there would imho occur logical
collisions

fake referer .. (fr) 
never .. ar .. er .. ar&er

blacklist referer .. (br) // overrides fr .. element impl as comma sep list ? as
ex for search sites
disable .. ar .. er .. ar&er

blacklist referer send behaviour .. (brsb)
send_no_referer .. cut_away_cgi_arguments .. send_fake_referer

Comment 146

[Ben Bucksch (:BenB)]

More than 10 years later, part of this were implemented in bug 822869. Thanks!

Comment 147

[Patrick McManus [:mcmanus]]

I think as close as we're going to get was implemented in comment 146 and meta referrer (other bug)