284868 - HTTP Referer not sent at all when "saving link as..."

Status: RESOLVED FIXED

(Toolkit :: Downloads API, defect)

Description

[Deomid Ryabkov]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b2) Gecko/20050304 Firefox/1.0+
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b2) Gecko/20050304 Firefox/1.0+

just like that: no "Referer" header at all.
it is, however, sent correctly when just clicking on the same link.
this breaks some sites that check referer when downloading files.

this is a firefox-specific bug: i checked mozilla 1.7.3 and is works fine.

Reproducible: Always

Steps to Reproduce:
1. point on a link
2. right-click
3. "save link as..."
4. inspect the request sent to the server (e.g. with tcpdump).

Actual Results:  
no referer header is present among those sent to server.

Expected Results:  
firefox should have sent the location of the referring page

Comment 1

[Deomid Ryabkov]

here, i've coded up a small script to demonstrate the behaviour.

http://rojer.pp.ru/cgi-bin/refcheck.cgi

when called with "check" parameter it returns 403 unless referer is present.

Comment 2

[Deomid Ryabkov]

http://rojer.pp.ru/cgi-bin/refcheck.cgi?check

hm...

Comment 3

[Deomid Ryabkov]

firefox seems not to send referer even when the link is just clicked from this page.

anyway, you get the idea: something is wrong with Referer header handling.

(okay, now i'm about to stop posting additional comments :))

Comment 4

[Jesse Ruderman]

Firefox doesn't send a referrer if you click a link from this page because this
page is https (and because the other page is http?).  That behavior makes little
sense to me, but most browsers have the same behavior.

Comment 5

[Jesse Ruderman]

Steps to reproduce:
1. Go to http://rojer.pp.ru/cgi-bin/refcheck.cgi.
2. Right-click the link on that page and select "Save Link As...".

Result: Dialog saying "The link could not be saved. The web page might have been
removed or had its name changed."

Expected: Save the link.

Confirmed using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b2)
Gecko/20050303 Firefox/1.0+.

This bug occurs with the Alt+click shortcut to save links, too.

I suspect that this bug is the reason I often get "The link could not be saved"
dialogs and tiny HTML files when I try to download porn videos.

This appears to have been fixed before in bug 236194.  I wonder what happened to
that fix.

Internet Explorer seems to have the same behavior.

Comment 6

[Deomid Ryabkov]

okay, i think i've tracked the problem to its source.
before link is saved, it is "sniffed", to determine its content type.
it turns out that this first "sniff" (as in nsHeaderSniffer,
browser/base/content/contentAreaUtils.js) is performed in such a way, that only
basic headers + cookies are sent. no referrer information from the current
context is inherited.

the exact component involved is nsURIChecker
(netwerk/base/src/nsURIChecker.cpp), that only goes as far as changing request
method to "HEAD", not setting any other request headers.
this is no wonder, because it only accepts uri parameter for its initialization
and. cookies seem to be added later by nsHttpChannel initialization (by an
independent lookup based on uri).
it is clear that referer should be initialized by nsURIChecker and it seems that
its current interface has no provision to do that.

so it looks like it has to be changed to operate properly, by adding a second
parameter to Init() method, for example.
however, i might be completely wrong, because this is my first peek at mozilla
source and i might have missed something.

can anybody comment on this?

Comment 7

[timeless]

nsIURIChecker has a channel property, nsIHttpChannel's have referrer properties
and explicit comments about setting cookie headers.

Comment 8

[Deomid Ryabkov]

thanks for pointing that out!

i think i have the proper solution now.

the patch attached is against v1.0, but should apply to cvs version as well.

Comment 9

[Deomid Ryabkov]

Attachment: the proposed patch

Comment 10

[Deomid Ryabkov]

c'mon, guys, the patch is real simple (8 lines!) and it does fix rather annoying
bug.
people keep asking me (on one of the forums) why can they not save links (to
files that check the referring page).
explorer appears to have this bug too. i could tell people to go get firefox if
this fix made it into the release version :)

Comment 11

[Darin Fisher]

To what version of contentAreaUtils.js does this patch apply?  Are you sure
you're patching the Mozilla trunk?

Comment 12

[Darin Fisher]

In fact, I think the link checker code has been removed.  Has anyone tested a
trunk build of Firefox?  (Deer Park Alpha 1 would be a good candidate to test.)
If that code has been removed, then perhaps this bug is fixed or worksforme.

Comment 13

[Jaime Mitchell (use [e-mail] for email)]

(In reply to comment #12)
> In fact, I think the link checker code has been removed.  Has anyone tested a
> trunk build of Firefox?  (Deer Park Alpha 1 would be a good candidate to test.)

The code was changed after Deer Park Alpha 1, see bug 294759, test case in
comment 5 is WFM with current firefox trunk build.

Comment 14

[Jesse Ruderman]

Following the steps in comment 5, the file is now saved, but the file is "Your
client has not sent the Referer header."  So this bug still exists.

Comment 15

[Deomid Ryabkov]

the problem actually got worse since 1.0
I tested this on a recent daily build and now it happily saves the content of
the "403 Forbidden" error.
(tested on "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8b3) Gecko/20050710
Firefox/1.0+")

for reference, source of refcheck.cgi: http://rojer.pp.ru/misc/refcheck.cgi.txt

Comment 16

[Jesse Ruderman]

I have a patch, which I will attach as part of my patch for bug 303181.

Comment 17

[Jesse Ruderman]

Fixed.