175418 - Limit prefetching of rel="next" documents

Status: NEW

(Core :: Networking, defect, P5)

Description

[cmiller]

Problem:
--------

Due to the web not being a linear medium, most of the time the rel="next"
relationship is not relevant to browsing. While it's good to _have_ such a
relationship for those occasions you're reading something from start to finish,
most of the time you're not.

Blindly pre-fetching "next" documents makes the tag less useful. It changes the
definition from "logical ordering", to "pages you want prefetched", which is
already taken care of explicitly by rel="prefetch"

I'm already considering removing the "next" tags from my own site, as I use the
tag to define a chronological relationship between pages. People rarely browse
sites chronologically, and since my account has a bandwidth cap, I can't really
afford to have visitors downloading two pages for every one they request.

Possible Solution:
------------------

When loading a page, examine both the rel="prev" and rel="next" links. If the
referring page matches one, then prefetch the other. i.e., if you "came from"
the previous page, you should prefetch the next page, and vice versa. This way,
if someone _is_ following the sequence of pages, forwards or backwards, then
prefetching will kick in automagically, but if you're not following the
sequence, prefetching won't waste bandwidth on either side.

Comment 1

[Boris Zbarsky [:bzbarsky]]

to darin

Comment 2

[Darin Fisher]

hmm, well... i understand your concern, but the HTML spec does suggest that
browsers may prefetch rel=next.

from http://www.w3.org/TR/html4/types.html#type-links,

  "Next: Refers to the next document in a linear sequence of documents. User 
   agents may choose to preload the "next" document, to reduce the perceived
   load time."

a quick and dirty solution for your content would be to append a '?no-prefetch'
to the end of each url.  for static content, most servers will simply ignore the
query string, but mozilla prefetching is explicitly disabled when the URL
contains a query string, any query string (i just picked '?no-prefetch', but
'?xxx' would also work).  obviously, this is rather hackish, and we should
consider coming up with a cleaner mechanism.

perhaps this bug should be morphed into refining such a mechanism.

Comment 3

[cmiller]

The only places I've seen next-links use is in weblog archives
(http://simon.incutio.com/archive/2002/10/17/#mozillaPrefetching), and online
documentation (http://trainedmonkey.com/week/2002/42#n1053). For both of these,
linear reading is the exception rather than the rule. Asking sites to do
something hacky (and maybe search-engine/index confusing?) to prevent the
browser prefetching doesn't seem to be the best solution, even if it's
technically sanctioned by the spec. 

"May choose to prefetch", as opposed to just "may prefetch" implies the browser
can be smart in deciding which next-links to prefetch. Having the prefetching
trigger on previous user behaviour suggesting they are following the prev/next
path just seemed the most logical implementation. I'm sure there are others.

Comment 4

[Hixie (not reading bugmail)]

I read specs linearly, and this helps immensely. I like the idea of checking to
see if we matched one of the 'prev' links before fetching the 'next' links
though, if there are any 'prev's. That's quite a cool idea.

Comment 5

[Darin Fisher]

hmm... it might be possible to implement this by simply comparing the referrer
of a HTML document against the href of a prev tag when a next tag is found.

hmm... and the HTMLContentSink is told its channel at construction time, so it
could easily snag a reference to the Referer header that was sent in the request
for the document currently being parsed.

this does sound like a really cool idea.

Comment 6

[Jesse Ruderman]

In addition to checking whether (referrer = this page's prev link), you could
check whether (referring page's next link = this).  That way, if a site uses
next links but doesn't use prev links, "smart prefetching" would still be able
to determine that you're following the sequence.  I don't know how common that is.

Comment 7

[Prognathous]

Checking that the previous page does match the current rel="prev" (before
activating link prefetching for rel="next") is a brilliant solution for this
problem, cmiller, I'm all for it.

Prog.

Comment 8

[Darin Fisher]

-> default owner

Comment 9

[Steffen Weber]

In addition to what has been suggested already: I think Firefox should not prefetch rel="next" links if there is a rel="prefetch" link present, too.

Rationale: In this case the web-developer has explicitly marked the ressource that should be prefetched and therefore Firefox should not try to outsmart the web-developer. For example, in a photo show it makes sense to prefetch the next image file, but it does not make much sense to prefetch the page that contains the next image file.

Comment 10

[Steffen Weber]

Please disable this so-called feature "network.prefetch-next"!

We have rel="next" links on many of our multi-page articles and slideshows. The pointless prefetch-requests sent by Firefox have accumulated to 15% (!) of total page requests. What a waste of resources! Our HTML pages are explicitly marked as uncacheable but there is no way to stop these requests from coming in.

We have started to block all "X-Moz: prefetch" requests but now some users behind a proxy-server report problems. Most likely because their proxy caches the denied requests. How can we block these requests reliably?

Comment 11

[Keith Pomakis]

I second Steffen's suggestion to not prefetch rel="next" links if there is a rel="prefetch" link present.

Please, somebody step up to the plate and do something about this.  I've been making compromises to my web pages for the last 10 years by leaving out "next" links that logically should be there, solely because I DON'T want them to be prefetched by the over-aggressive firefox browser.

Comment 12

[Steffen Weber]

In contrast to what I posted 2 years ago in comment 10, I now think that "network.prefetch-next" should be disabled completely. If the alternative (disabling it only if another rel="prefetch" link exists) gets implemented then I would add <link rel="next" href="blind.gif"> to every single page. =)

No other browser than Firefox tries to outsmart the developer with such a crazy feature, please just disable it! rel="prefetch" exists for cases where prefetching really makes sense.

Comment 13

[Steffen Weber]

Sorry, I of course meant <link rel="prefetch" href="blind.gif"> above.

Comment 14

[Keith Pomakis]

I'm certainly in favour of disabling "next" prefetching completely as well.

Wait, is there a configuration parameter specifically for controlling whether or not "next" links get prefetched (i.e., this "network.prefetch-next" of which you speak)?  If so, a reasonable (and extremely simple) resolution to this bug could be to simply change the default value to FALSE.  Then, a few years later, once the firefox development team is satisfied that this is the only reasonable value for the configuration parameter, it can be removed completely.  :-)

Comment 15

[Steffen Weber]

I had thought that "network.prefetch-next" would only disable the prefetching of rel="next" links (as its name suggests). But https://developer.mozilla.org/en/link_prefetching_faq says that this switch disables prefetching _completely_ (contrary to its name). So the solution does not seem to be that simple.

But still, I don't think that fixing this bug would be a huge effort.

Comment 16

[zanezane.net]

I found this bug report Googling for the exact issue.

This is my 2 cents on the problem: in a "listing" page such as this https://turbolab.it/ my site displays the latest articles. Based on Google SERP experience, I do rel="prefetch" on the first article of the list (the newest and most clicked for each day). This is the page I'd like to prefetch.

That page also has a rel="next" but its only purpose is to guide search engines to the next paginated listing (with previous articles): https://turbolab.it/home/2 .

As per reported issue, Firefox prefetches both, even if the next prefetched link is unuseful (only a few users actually go to that page).

I'm ok whit rel="next" being prefetched by default, but I also would like a markup-way to disable it.

Thanks.

Comment 17

[Steffen Weber]

This behaviour (prefetching of rel="next" documents) triggers another bug on websites using a Content Security Policy (CSP): https://bugzilla.mozilla.org/show_bug.cgi?id=1242902

Comment 18

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258