174984 - Prefetch requests should send Referer [was: Make Link prefetching work only with pages on the same server]

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Robert Marcano]

Link prefetching implemented in Bug #12274, must only prefetch files on the same
server, or at least make it a user preference. I think that trusting page
authors about what to prefetch is not right, If I visit a page, that has a link
tag to a pronography page that my company/school etc. proxy server bans, I can
be fired if the logs shows that it was requested using my proxy user

Comment 1

[Christopher Hoess (gone)]

It's already trivially possible for pages to do this, regardless of prefetching. 
<iframe src="http://www.example.com/very_naughty_page" style="display: 
none"></iframe>.

Comment 2

[Robert Marcano]

Sure but the Proxy log will show that is was refered form another page. The
prefetch implementation does not send the Referer header, so the user has no way
to demostrate that this wasn't his fault

Comment 3

[Christopher Hoess (gone)]

I don't see any deliberate decision in that bug not to send 
document.referer; perhaps that was an oversight in implementation, and 
we should send a Referer header when we prefetch?

Comment 4

[gordon]

Darin, where do we stand on document.referer?

Comment 5

[Darin Fisher]

will try to resolve this bug for moz 1.2 final, one way or the other.

Comment 6

[Daniel Veditz [:dveditz]]

As summarized this bug should be WONTFIX, limiting sites to the same server
would prevent large distributed sites from taking advantage of the feature for
no particularly good reason. If this bug has morphed into the referrer issue
then let's change the summary -- that does sound like a bug worth fixing since
many sites protect their content by checking referrers, and it would address the
original concern of this bug.

(If possible respect the referrer-spoofing prefs, or if not pre-emptively file
bugs about not doing so.)

Comment 7

[Boris Zbarsky [:bzbarsky]]

*** Bug 175118 has been marked as a duplicate of this bug. ***

Comment 8

[R.K.Aa.]

I am told this feature is ON by default. I was also pointed to the FAQ and see
from it that you're aware that in several countries people pay per downloaded
Kb. Even more ISP's offer billing shemes in that lane. I was considering one
myself. Depending on browsing habits, this might mean one ugly surprise in
increased Internet expenses for mozilla users who are unaware of this feature.

I'll certainly do my bit to issue warnings, but need to know how to disable the
feature. Anyone?

Comment 9

[Daniel Veditz [:dveditz]]

Speaking of extra downloading, is the pre-fetching smart enough to pay attention
to the image blocking prefs? I'd hate to waste time (and money in some locales)
prefetching an image I wouldn't display anyway.

Comment 10

[James Antill]

 Note that in the iframe case you can see that the browser is downloading
something, in the prefetch case ... the browser will just silently download ...
indeed I don't think it's possible for the user to find out what is being
downloaded on their behalf.

Comment 11

[Robert Marcano]

If the referer issue is changed, It would be nice to have a preference set from
which sites prefetch is enabled with this posible values: All, same server and
same domain., this way the concern expressed in comment 6 about large
distributed sites can be solved

Comment 12

[Darin Fisher]

R.K.Aa: remember, there is a balance here.  websites are not going to litter
their sites with tons of prefetch links because they probably can't afford the
server bandwidth for that.  perhaps the big problem is bad sites, but as has
been said before, bad sites can already cause a lot of trouble (without your
knowledge) once you visit them.  disabling prefetching doesn't protect you
against silent downloads. 

dveditz: there is no way to reliably determine if a href corresponds to an image
without seeing the response headers or the image data itself.  there is not
contextual information about where the href will actually appear in a page
(i.e., we don't know if it will be later referenced via an <img> tag).

james: an <iframe> can easily be hidden via the DOM, and the src of an <iframe>
can be easily changed via JS from an onLoad handler, which would cause the
<iframe> to be loaded silently without any visible changes in the browser UI. 
this is how websites currently prefetch content.  <link> tag prefetching
provides an explicit way of doing the same thing, and because the browser is
made aware of the fact that a site is doing prefetching, the browser can be
smart about prioritizing prefetch requests.

robert: what is most important is figuring out the default setting because
websites will base deployment of <link> prefetching on that.  if by default we
disable link prefetching across domains, then link prefetching will most likely
never be used by websites to fetch content across domains.  so, what good is UI
for such a pref if there is no content?

Comment 13

[Mitchell Stoltz (not reading bugmail)]

Do we need to worry about non-malicious but poorly constructed existing sites
that might inadvertently cause a lot of hits on other sites by accidentally
causing a lot of prefetching?

My guess is this won't be a problem, but I wanted to put it out there.

Comment 14

[Darin Fisher]

yes, and that's why we need to tag each prefetch request so sites can decide to
ignore prefetch requests if it becomes a burden.  if we did this and if we
included a Referer header then websites could filter out prefetch requests from
other domains if they so desired.

morphing bug summary to reflect reality ;-)

Comment 15

[Darin Fisher]

Attachment: v1 patch

ok, this patch causes moz to send out a Referer header when prefetching.  i'm
considering only allowing prefetching when the referrer is http: because some
referring URL types are not sent as a Referer header (in fact, we only allow
sending http:, ftp:, or gopher: as Referer on a http: load).

this patch also adds a "cache-control: x-moz-prefetch" header.	i'm totally up
for suggestions here... this is just my initial cut at such a header.

lastly, i made it so that we won't prefetch if the disk cache and memory cache
are set to zero size.

Comment 16

[James Antill]

Darin: There is a reason I don't enable JS, so random security holes/stupidity
in JS don't bother me ... if people start exploiting it then it'll just become
another one of those many don't let JS do X stupid thing preferences.
 So the fact remains that before prefetch, it was very obvious if the browser
was using the network ... which is how it should be. And it was also possible to
just hit ESC and have all network activity stop, which is also how it should be.

 I'm not trying to say prefetching is bad ... there are a lot of cases where
it'll be just want everyone wants. I can imagine there are a lot where the web
page writer thinks so and gets it wrong ... but that's not any different than
them having megton images everywhere.

 Keep it going to external servers if you want, but as well as passing the
referer to tell the website owner, the UI needs to tell _me_ and the UI for stop
etc. need to work.

Comment 17

[Darin Fisher]

> Darin: There is a reason I don't enable JS, so random security holes/stupidity
> in JS don't bother me ... if people start exploiting it then it'll just become
> another one of those many don't let JS do X stupid thing preferences.

yes, you are right.  disabling javascript protects the user against a lot of
stuff.  and this is probably the best argument for providing a user visible pref
to disable prefetching.


>  So the fact remains that before prefetch, it was very obvious if the browser
> was using the network ... which is how it should be. And it was also possible 
> to just hit ESC and have all network activity stop, which is also how it 
> should be.

"obvious if the browser [is] using the network" only when javascript is disabled.


> Keep it going to external servers if you want, but as well as passing the
> referer to tell the website owner, the UI needs to tell _me_ and the UI for 
> stop etc. need to work.

again, with javascript enabled, the UI doesn't always inform you that something
is being downloaded and the stop button may not always be enabled.  but that
said, i think it might be cool to have a little icon appear in the status bar
when the browser is prefetching documents.  it could provide easy access to stop
as well as view any active prefetches.

Comment 18

[Darin Fisher]

Attachment: v2 patch

same patch w/ strict referrer checking.  switched to sending 'X-Moz: prefetch'
request header instead of 'Cache-Control: x-moz-prefetch' -- i don't want to
trample on 'Cache-control' until we have something standardized!  i also added
a security check to avoid prefetching from mailnews, which i should have done a
long time ago :(

Comment 19

[Gagan]

Comment on attachment 103618 [details] [diff] [review]
v2 patch

r=gagan

Comment 20

[Mitchell Stoltz (not reading bugmail)]

Comment on attachment 103618 [details] [diff] [review]
v2 patch

r=mstoltz for security on the HTMLContentSink part.

Comment 21

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 103618 [details] [diff] [review]
v2 patch

sr=bzbarsky; please file a bug on docshell to create a reasonable replacement
for that apptype business....  What about other gecko-based apps that want that
level of security?  They should not get screwed...

Comment 22

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Comment on attachment 103618 [details] [diff] [review]
v2 patch

a=dbaron for trunk checkin

Comment 23

[Darin Fisher]

fixed-on-trunk

Comment 24

[John Unruh]

Darin, how can this be verified?

Comment 25

[Darin Fisher]

junruh: you will need to use a packet sniffer probably or inspect server logs to
verify that a Referer header is truly being sent in prefetch requests.  you'll
also need a sample page that uses prefetching.  i have some samples at:

  http://unagi.mcom.com/~darinf/prefetch/testcases.html

you can also use a packet sniffer to verify that we aren't prefetching from
mailnews.  that can be verified by sending yourself one of the pages (that does
prefetching) from testcases.html.  let me know if you need any help w/ this.

two good packet sniffers:
  www.ethereal.com (gui)
  ngrep.sourceforge.net (command line)

Comment 26

[John Unruh]

Verified based on the following. Tested on the 10/25 Win2000 trunk build. From 
this page - http://unagi/~darinf/prefetch/test1.html only one of 3 images 
(Screenshot1.png) should be downloaded, which was the case, and shown in the 
Ethereal log.

Partial source of the test page:
<head>
<link rel=next href="http://64.130.71.9/~darinf/test/Screenshot.png">
<link rel="prefetch" href="http://64.130.71.9/~darinf/test/Screenshot1.png">
<!--
<link rel="next alternate" 
href="http://64.130.71.9/~darinf/test/Screenshot2.png">
-->
<meta HTTP-EQUIV="link" 
CONTENT="&lt;http://64.130.71.9/~darinf/test/Screenshot2.png&gt;; rel=next">
</head>

Next, I emailed myself the page, opened it in a new profile, and nothing was 
prefetched, as evidenced in the Ethereal log.

Comment 27

[Daniel Veditz [:dveditz]]

adt+ for branch, please seek drivers approval before landing

Comment 28

[Darin Fisher]

yup, request to drivers sent last thursday.

Comment 29

[Judson Valeski]

 please checkin to the MOZILLA_1_0_BRANCH branch. once there, remove the
"mozilla1.0.2+" keyword and add the "fixed1.0.2" keyword.

Comment 30

[Darin Fisher]

fixed1.0.2

Comment 31

[John Unruh]

Verified on Win2000, OSX and Linux branch

Comment 32

[Daniel Veditz [:dveditz]]

unverifying for retest in 1.0.2 builds -- according to darin and barrowman
prefetching isn't working at all because this fix stumbles on bug 142135 which
was never fixed on the branch.

Comment 33

[Daniel Veditz [:dveditz]]

re-verifying -- I guess just because we don't use the pre-fetched data doesn't
mean we don't send the right headers. (I unverified the wrong bug)