196871 - Referrer downloads cause bad data page or invalid html on shift-click on link

Status: RESOLVED FIXED

(SeaMonkey :: UI Design, defect)

Description

[Darrin Maunders]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b; MultiZilla vundefinedundefined) Gecko/20030210
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b; MultiZilla vundefinedundefined) Gecko/20030210

The link to the file uses a a java referrer mechanism. IE downloads the file
correctly. Left clicking the file in Mozilla causes a page of garbage to load or
if you shift-click the link it downloada an HTML file with the following
information in it.

"Unauthorized download

We're sorry, but this downlod request cannot be authorized. There could be
several reasons for this:

1) You clicked on a link on a site that is not part of the Doom9 domains. In
this case the URL in your browser would not contain doom9.org or doom9.net. We
do not allow that kind of external hotlinking. This site creates a great deal of
traffic and it's hard to find a good hosting. We will not provide the bandwith
for other sites, too so that they can have a great software archive without
having to host and pay for it. Please visit Doom9.org and you can download all
the files you want.

2) You're using a download manager. Download managers usually offer to search
for alternative download locations. While this is nice for the user, it's bad
for the webmaster because it could be that a download which originated on
another site could theoretically still be downloaded from this site, without
anybody knowing. But there's a solution to this problem:

Set up your download manager to send a HTTP referer, and when you click on a
link on the Doom9 site the download will be authorized. Here's how you can do
this in Getright:

Go to Getright Configuration, under Advanced/Protocol check Send "Referer" in
HTTP Request and then select Generated from the download URL.

The setup should be similar for other download managers.

3) Your browser did not send a referer. This normally happens if you right click
on a link and select save-as, or if you copy and paste a link into a new browser
window and press Enter. It may also be that your browser is set up to not send
any referer, though I did not find any such option in the browser I'm using
(IE6). Unfortunately we must test for empty referer or we still have the
download manager problem mentioned in 2). But left clicking on a link works just
fine and has been tested in IE6, Netscape 6, Opera6 and Mozilla 1.0.

Some software firewalls, for instance the Norton Personal Firewall (which is
also part of Norton Internet Security) or ZoneAlarm remove the referer from HTTP
requests which will block any downloads on any site that does a referer check.
Hence such options should always be turned off. In Norton you have to go to the
Privacy Settings and disable browser privacy (which doesn't really protect your
privacy anyway as your IP address will always be logged regardless of that
setting), in ZoneAlarm you have to go to Privacy -> Cookie Control -> Custom and
uncheck "remove private header information".

Software like AdSubstract also blocks the referer and makes downloads impossible.

We're sorry for any inconvenience this may cause and hope you understand our
reasons for doing so. Bandwith doesn't come cheap and we have to do all we can
to keep the bandwith usage (roughly 600GB a month these days) down."

Reproducible: Always

Steps to Reproduce:
1. Click on link to download rar
2. Shift click on link to download HTML
3.

Actual Results:  
See above

Expected Results:  
To download rar file to hard disk. This does not occur.

Comment 1

[Jesiah S]

Linux 2003031105
Go to http://www.doom9.org/software.htm then find "Media Player Classic 6.4.3.1" 

Clicking the Link:
Causes a HTTP GET Request w/ a Referer of http://www.doom9.org/software.htm
Then the Server responds with HTTP/1.1 200 with a Content type:text/plain
mozilla downloads the rar file and displays it as a plain text file

Right clicking the link:
Causes a HTTP HEAD Request w/o a Referer 
Then the Server responds with HTTP/1.1 302 with a Location of
http://www.doom9.org/leach.htm
mozilla go and retrives http://www.doom9.org/leach.htm w/o problems.

Adding a Referer to the HEAD request would fix this so you can atleast right
click and save the file.

Comment 2

[Boris Zbarsky [:bzbarsky]]

This is a dup of us not sending the referrer on save operations (clicking on the
link is not a save operation -- different codepath).

Comment 3

[Matthias Versen [:Matti]]

that would be bug 170722 but the bug is marked fixed and it's still not working.
Any hints Boris ?

Comment 4

[Boris Zbarsky [:bzbarsky]]

The back end now sends a referrer if its given one.  The Save As front end still
needs to give it one.

Comment 5

[Jon Henry]

Unless I've missed something, this is a dupe of bug 188253.

*** This bug has been marked as a duplicate of 188253 ***

Comment 6

[Boris Zbarsky [:bzbarsky]]

The page descriptor is something that lets you get the page data from the cache.

The referrer is something you only need if you're going to end up hitting the
server.

The two are unrelated....

Comment 7

[Jon Henry]

Sorry, I didn't realize that. I was following a trail of bugs and saw that that
one was spun off of bug 170722 (which does deal with referrers) and I guess I
made a bad assumption.

Comment 8

[Asa Dotzler [:asa]]

->XP Apps

Comment 9

[Jesse Ruderman]

Bug 236194 is the same bug for Firefox.  It has a simple Firefox-specific patch.

Comment 10

[Christian :Biesinger (don't email me, ping me on IRC)]

taking. bug 160454 might make this a bit easier to fix, but this is doable
either way.

Comment 11

[Christian :Biesinger (don't email me, ping me on IRC)]

Attachment: patch

who knows... maybe this fixes some of the problems with "save link target as"

Comment 12

[Christian :Biesinger (don't email me, ping me on IRC)]

well that was pretty badly worded. I meant some of the problems that were blamed
on HEAD before...

Comment 13

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 143068 [details] [diff] [review]
patch

sr=bzbarsky, but I _really_ wish firefox hadn't forked this code... :(

Comment 14

[[e-mail]]

Comment on attachment 143068 [details] [diff] [review]
patch

>+    var refer = getReferrer(document);
Some naming consistency would be nice, var referrer perhaps?
I assume that unsuitable referrers will get screened out at some point?

Comment 15

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #14)
> Some naming consistency would be nice, var referrer perhaps?

ok, I'll change that

> I assume that unsuitable referrers will get screened out at some point?

yes, nsHttpChannel does that.

Comment 16

[Christian :Biesinger (don't email me, ping me on IRC)]

Checking in resources/content/contentAreaUtils.js;
/cvsroot/mozilla/xpfe/communicator/resources/content/contentAreaUtils.js,v  <--
 contentAreaUtils.js
new revision: 1.104; previous revision: 1.103
done

Comment 17

[OstGote!]

Question: Is a referrer sent if I do a "save link target as" with this pref?

user_pref("network.http.sendRefererHeader", 1);
// 0=don't send any, 1=send only on clicks, 2=send on image requests as well

Comment 18

[Jon Henry]

I haven't personally tried, but examining the code, it looks as if the
preference is respected.

Comment 19

[OstGote!]

(In reply to comment #18)
> I haven't personally tried, but examining the code, it looks as if the
> preference is respected.

Yes it is. I tried it with www.doom9.org. The only (nothing to do with this bug)
thing is, if you do a shift+click to download you will always get a .rar.html or
.zip.html served (wrong extension).