175418 - Limit prefetching of rel="next" documents

Status: NEW

(Core :: Networking, defect, P5)

Description

[cmiller]

Problem:
--------

Due to the web not being a linear medium, most of the time the rel="next"
relationship is not relevant to browsing. While it's good to _have_ such a
relationship for those occasions you're reading something from start to finish,
most of the time you're not.

Blindly pre-fetching "next" documents makes the tag less useful. It changes the
definition from "logical ordering", to "pages you want prefetched", which is
already taken care of explicitly by rel="prefetch"

I'm already considering removing the "next" tags from my own site, as I use the
tag to define a chronological relationship between pages. People rarely browse
sites chronologically, and since my account has a bandwidth cap, I can't really
afford to have visitors downloading two pages for every one they request.

Possible Solution:
------------------

When loading a page, examine both the rel="prev" and rel="next" links. If the
referring page matches one, then prefetch the other. i.e., if you "came from"
the previous page, you should prefetch the next page, and vice versa. This way,
if someone _is_ following the sequence of pages, forwards or backwards, then
prefetching will kick in automagically, but if you're not following the
sequence, prefetching won't waste bandwidth on either side.

Comment 1

[Boris Zbarsky [:bzbarsky]]

to darin

Comment 2

[Darin Fisher]

hmm, well... i understand your concern, but the HTML spec does suggest that
browsers may prefetch rel=next.

from http://www.w3.org/TR/html4/types.html#type-links,

  "Next: Refers to the next document in a linear sequence of documents. User 
   agents may choose to preload the "next" document, to reduce the perceived
   load time."

a quick and dirty solution for your content would be to append a '?no-prefetch'
to the end of each url.  for static content, most servers will simply ignore the
query string, but mozilla prefetching is explicitly disabled when the URL
contains a query string, any query string (i just picked '?no-prefetch', but
'?xxx' would also work).  obviously, this is rather hackish, and we should
consider coming up with a cleaner mechanism.

perhaps this bug should be morphed into refining such a mechanism.

Comment 3

[cmiller]

The only places I've seen next-links use is in weblog archives
(http://simon.incutio.com/archive/2002/10/17/#mozillaPrefetching), and online
documentation (http://trainedmonkey.com/week/2002/42#n1053). For both of these,
linear reading is the exception rather than the rule. Asking sites to do
something hacky (and maybe search-engine/index confusing?) to prevent the
browser prefetching doesn't seem to be the best solution, even if it's
technically sanctioned by the spec. 

"May choose to prefetch", as opposed to just "may prefetch" implies the browser
can be smart in deciding which next-links to prefetch. Having the prefetching
trigger on previous user behaviour suggesting they are following the prev/next
path just seemed the most logical implementation. I'm sure there are others.

Comment 4

[Hixie (not reading bugmail)]

I read specs linearly, and this helps immensely. I like the idea of checking to
see if we matched one of the 'prev' links before fetching the 'next' links
though, if there are any 'prev's. That's quite a cool idea.

Comment 5

[Darin Fisher]

hmm... it might be possible to implement this by simply comparing the referrer
of a HTML document against the href of a prev tag when a next tag is found.

hmm... and the HTMLContentSink is told its channel at construction time, so it
could easily snag a reference to the Referer header that was sent in the request
for the document currently being parsed.

this does sound like a really cool idea.

Comment 6

[Jesse Ruderman]

In addition to checking whether (referrer = this page's prev link), you could
check whether (referring page's next link = this).  That way, if a site uses
next links but doesn't use prev links, "smart prefetching" would still be able
to determine that you're following the sequence.  I don't know how common that is.

Comment 7

[Prognathous]

Checking that the previous page does match the current rel="prev" (before
activating link prefetching for rel="next") is a brilliant solution for this
problem, cmiller, I'm all for it.

Prog.

Comment 8

[Darin Fisher]

-> default owner

Comment 9

[Steffen Weber]

In addition to what has been suggested already: I think Firefox should not prefetch rel="next" links if there is a rel="prefetch" link present, too.

Rationale: In this case the web-developer has explicitly marked the ressource that should be prefetched and therefore Firefox should not try to outsmart the web-developer. For example, in a photo show it makes sense to prefetch the next image file, but it does not make much sense to prefetch the page that contains the next image file.

Comment 10

[Steffen Weber]

Please disable this so-called feature "network.prefetch-next"!

We have rel="next" links on many of our multi-page articles and slideshows. The pointless prefetch-requests sent by Firefox have accumulated to 15% (!) of total page requests. What a waste of resources! Our HTML pages are explicitly marked as uncacheable but there is no way to stop these requests from coming in.

We have started to block all "X-Moz: prefetch" requests but now some users behind a proxy-server report problems. Most likely because their proxy caches the denied requests. How can we block these requests reliably?

Comment 11

[Keith Pomakis]

I second Steffen's suggestion to not prefetch rel="next" links if there is a rel="prefetch" link present.

Please, somebody step up to the plate and do something about this.  I've been making compromises to my web pages for the last 10 years by leaving out "next" links that logically should be there, solely because I DON'T want them to be prefetched by the over-aggressive firefox browser.

Comment 12

[Steffen Weber]

In contrast to what I posted 2 years ago in comment 10, I now think that "network.prefetch-next" should be disabled completely. If the alternative (disabling it only if another rel="prefetch" link exists) gets implemented then I would add <link rel="next" href="blind.gif"> to every single page. =)

No other browser than Firefox tries to outsmart the developer with such a crazy feature, please just disable it! rel="prefetch" exists for cases where prefetching really makes sense.

Comment 13

[Steffen Weber]

Sorry, I of course meant <link rel="prefetch" href="blind.gif"> above.

Comment 14

[Keith Pomakis]

I'm certainly in favour of disabling "next" prefetching completely as well.

Wait, is there a configuration parameter specifically for controlling whether or not "next" links get prefetched (i.e., this "network.prefetch-next" of which you speak)?  If so, a reasonable (and extremely simple) resolution to this bug could be to simply change the default value to FALSE.  Then, a few years later, once the firefox development team is satisfied that this is the only reasonable value for the configuration parameter, it can be removed completely.  :-)

Comment 15

[Steffen Weber]

I had thought that "network.prefetch-next" would only disable the prefetching of rel="next" links (as its name suggests). But https://developer.mozilla.org/en/link_prefetching_faq says that this switch disables prefetching _completely_ (contrary to its name). So the solution does not seem to be that simple.

But still, I don't think that fixing this bug would be a huge effort.

Comment 16

[zanezane.net]

I found this bug report Googling for the exact issue.

This is my 2 cents on the problem: in a "listing" page such as this https://turbolab.it/ my site displays the latest articles. Based on Google SERP experience, I do rel="prefetch" on the first article of the list (the newest and most clicked for each day). This is the page I'd like to prefetch.

That page also has a rel="next" but its only purpose is to guide search engines to the next paginated listing (with previous articles): https://turbolab.it/home/2 .

As per reported issue, Firefox prefetches both, even if the next prefetched link is unuseful (only a few users actually go to that page).

I'm ok whit rel="next" being prefetched by default, but I also would like a markup-way to disable it.

Thanks.

Comment 17

[Steffen Weber]

This behaviour (prefetching of rel="next" documents) triggers another bug on websites using a Content Security Policy (CSP): https://bugzilla.mozilla.org/show_bug.cgi?id=1242902

Comment 18

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 19

[Brian Carpenter [:geeknik]]

Attachment: diff.patch

Attached is a patch that stops Firefox from prefetching rel=next targets and treats them as preconnect hints instead.

Why
The HTML Standard no longer singles out prefetch for rel=next. The current text
(https://html.spec.whatwg.org/multipage/links.html#link-type-next) says:

  "When the next keyword is used with a link element, user agents should
  process such links as if they were using one of the dns-prefetch, preconnect,
  or prefetch keywords. Which keyword the user agent wishes to use is
  implementation-dependent; for example, a user agent may wish to use the
  less-costly preconnect processing model when trying to conserve data, battery
  power, or processing power [...]"

Firefox currently always picks prefetch, which costs the site a full document
request for every page view that carries a rel=next link. That is the complaint
in comment 0, comment 10 and comment 16, and it is still being hit today:
Datasette removed its rel=next Link header from all HTML pages in 2026 to stop
Firefox fetching the next page of every table
(https://github.com/simonw/datasette/issues/3046).

Picking preconnect keeps us within the spec's "should" while sending no HTTP
request for the next document. Authors who do want the document fetched can
still say so with rel=prefetch, which is unchanged.

What the patch does

• HTMLLinkElement::TryDNSPrefetchOrPreconnectOrPrefetchOrPreloadOrPrerender:
  rel=next no longer reaches nsIPrefetchService; it takes the same path as
  rel=preconnect (Document::MaybePreconnect).
• nsContentSink::ProcessLinkFromHeader: same change for Link response headers.
• New mochitest netwerk/test/mochitests/test_rel_next.html (with rel_next.sjs)
  checks that both <link rel=next> and a Link: rel=next header trigger a
  speculative connection and do not enqueue a prefetch.

This also removes an inconsistency between the two paths. The element path
passed aExplicit=false for rel=next, so URLs with a query string were skipped
(the workaround suggested in comment 2). The header path went through
nsContentSink::PrefetchHref, which always passes aExplicit=true, so a
rel=next Link header was prefetched even with a query string. That is why the
Datasette URLs (?_next=...) were fetched.

Testing
Local Linux x86_64 build of current main:

• new test_rel_next.html and existing test_rel_preconnect.html pas
• existing prefetch mochitests pass: dom/base/test/test_link_prefetch.html,
  dom/base/test/test_bug1268962.html,
  dom/tests/mochitest/webcomponents/test_link_prefetch.html,
  netwerk/test/mochitests/test_prefetch_cache.html and
  test_prefetch_cache_no_headers.html
• browser tests pass: browser_page_load_event_prefetch.js,
  browser_prefetch_data_usage.js, browser_prefetch_delivery_type.js,
  browser_speculative_connection_link_header.js
• ./mach lint is clean on the changed files

Not done: no try push and no WPT run, and I have not confirmed the new test
fails on an unpatched build.

Follow-ups, not in this patch

• With this change every caller of nsIPrefetchService.prefetchURI
  aExplicit=true, so that parameter and the query-string check in
  nsPrefetchService::Prefetch are dead code. I left them alone to keep this
  patch to the behaviour change; happy to remove them here or in a follow-up.
• The network.prefetch-next pref name and its comment in all.js still describe
  rel=next prefetching. The pref continues to gate rel=prefetch.

Alternatives considered

• The Referer/rel=prev heuristic from comment 0 is also allowed by the spec
  ("heuristic analysis of past user behavior"), but it is more code and more
  ways to guess wrong for little gain over preconnect.
• Ignoring rel=next entirely (comment 12, comment 14) would go against the
  spec's "should".