1174921 - Reuse cached requests with different referrer policy

Status: RESOLVED DUPLICATE of bug 1642591

(Core :: DOM: Core & HTML, defect, P3)

Description

[Franziskus Kiefer [:franziskus]]

After internal discussions and with the working group I think we should reuse requests even if the referrer policy is different.

This can happen in two cases:
1) Speculative loads use a different policy than the one specified in the document.
2) Otherwise identical object have different referrer policy attributes.

Reloading in those cases creates unnecessary traffic. If the response actually depends on the referer header "Vary: Referer" might be used.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Speculative preloads, if reused, would be reused unconditionally even if they send "Vary: Referer".  So if you do want to support different content for different referer values, you can't reuse speculative preloads that don't match the referer policy.

Comment 2

[Franziskus Kiefer [:franziskus]]

After discussing this the consensus seemed to be that we should always reuse loads independent of the referrer policy (thereby also ignoring the vary header) as we do not want to create unnecessary traffic and websites should not rely on the referer header to send content. (There is no privacy benefit anyway as the request has been sent out already.) This should not break any websites as this behaviour is relatively new, but we should push that this is mentioned in the spec [1].

[1] https://w3c.github.io/webappsec/specs/referrer-policy/

Comment 3

[Boris Zbarsky [:bzbarsky]]

> and websites should not rely on the referer header to send content.

But... we 100% know that sites do just that.

> This should not break any websites as this behaviour is relatively new

Here's my question: will this lead to observably interoperable behavior between different preloading implementations in browsers?  Because if not, it sounds like a recipe for sites that work in one browser but not in another depending on which browser preloads what when.

Comment 4

[Thomas Nguyen (:tnguyen)]

Attachment: Reuse cached requests with different referrer policy

MozReview-Commit-ID: KZXIQytSU4Z

Comment 5

[Thomas Nguyen (:tnguyen)]

I made a simple test on Google Canary and it did not make a secondary request with different referrer-policy. I think it would be more reasonable.
Hi Franziskus.
Could you please take a look at the patch-removing cache referrer-policy check condition?
Thanks

Comment 6

[Boris Zbarsky [:bzbarsky]]

None of that addresses my concerns above...

Comment 7

[Thomas Nguyen (:tnguyen)]

(In reply to Boris Zbarsky [:bz] from comment #3)
> > and websites should not rely on the referer header to send content.
> 
> But... we 100% know that sites do just that.
> 
> > This should not break any websites as this behaviour is relatively new
> 
> Here's my question: will this lead to observably interoperable behavior
> between different preloading implementations in browsers?  Because if not,
> it sounds like a recipe for sites that work in one browser but not in
> another depending on which browser preloads what when.

imho, I think if the vary: referrer is not included in response (response content does not rely on the referrer header), it should be fine. What we are doing here is just ignoring referrer policy checking in cache, speculative loading works the same way w/o referrer policy.
We are concerning about a site relies on Referrer header to respond content (actually it does), uhm, in this case we need to get the exact response behavior from the site. Sending a new request with different referrer header (different referrer policy) wastes resource but is needed to have.
 We are considering the trade off between reusing cache and have the exact response from sites. I am not very sure how popular the Vary: referrer is used, but I guess not too many.

Comment 8

[Boris Zbarsky [:bzbarsky]]

> I am not very sure how popular the Vary: referrer is used

I would be shocked if the sorts of people who base their response on the referrer actually send the relevant Vary header.

The basic problem is this: say, for the same image, one browser does a speculative load and the other does not (because their speculation machinery is slightly different).  If the speculative load is not done with the same referrer as the real load, and is still reused, then you can have the image show up in one browser but not the other, right?  This is a potential compat issue, and the upshot will be to force minority browsers to align on the speculation behavior of whatever browsers are currently popular.  In particular, they will not be able to introduce _better_ speculation behaviors....

This should really have been thought through in the working group before they added mid-load mutation of referrer policy.

It's easy to say "sites shouldn't do that", but to the extent that they do this will basically create problems for all browsers that aren't Chrome, in today's environment.

Maybe the working group thinks we can assume that newer sites, which use referrer policies, won't have conditional responses based on referrer policy. But they why do we need all the different policies to start with, instead of just a boolean for whether to send a referrer or not?  Clearly something doesn't add up here in the reasoning, and I'd like to understand whether I'm missing something or whether the working group is just deluding itself.

Comment 9

[Franziskus Kiefer [:franziskus]]

Comment on attachment 8788105 [details] [diff] [review]
Reuse cached requests with different referrer policy

Review of attachment 8788105 [details] [diff] [review]:
-----------------------------------------------------------------

Cancelling this for now. It looks ok to me in general but we really have to decide first on what's the correct behaviour here.
Looking at the HTML spec we should re-use the cached entry. But I'm not sure if they're aware of this issue. This might have got lost when moving things from the referrer policy spec to html [1].

[1] https://github.com/whatwg/html/pull/1153

Comment 10

[Franziskus Kiefer [:franziskus]]

Anne, you were involved in the HTML change. Is this issue sufficiently addressed and can you point us to relevant parts in the HTML spec if it is? I don't find it.

Comment 11

[Anne (:annevk)]

The reason referrer can change is because it's set using markup. That's simply not going to be as deterministic as using HTTP headers. So yeah, browsers might have to align on heuristics over time. I'm not sure why that is a bad thing. Either way if you're not careful there will be non-determinism here solely due to network speeds and such.

Comment 12

[Boris Zbarsky [:bzbarsky]]

> there will be non-determinism here solely due to network speeds

There shouldn't be non-determinism in terms of what data is actually received if the requests are identical, unless the data actually changes due to time passing.

Comment 13

[Anne (:annevk)]

I was thinking that if requests before a <meta name=referrer> were asynchronously constructed it depends on network speeds and packet boundaries whether the <meta name=referrer> is taken into account. I don't think we currently accurately define when requests are constructed time-wise.

Comment 14

[Boris Zbarsky [:bzbarsky]]

> I don't think we currently accurately define when requests are constructed time-wise.

If we don't, it's a bug in the specs; we should.

Comment 15

[Thomas Nguyen (:tnguyen)]

(In reply to Anne (:annevk) from comment #13)
> I was thinking that if requests before a <meta name=referrer> were
> asynchronously constructed it depends on network speeds and packet
> boundaries whether the <meta name=referrer> is taken into account. I don't
> think we currently accurately define when requests are constructed time-wise.

FWIW, this is exactly the case I was using to verify Firefox behavior. Putting <meta name=referrer> after <script> (which contains a request) in speculative loading may have no effect for the first request but affect the later request.

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Moving to p3 because no activity for at least 1 year(s).
See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information

Comment 17

[Anne (:annevk)]

FWIW, https://html.spec.whatwg.org/#the-list-of-available-images does not consider the referrer policy as a cache key and I don't think it should. (Edit: it's being discussed again in https://github.com/whatwg/html/issues/5541 though.)

Comment 18

[Emilio Cobos Álvarez (:emilio)]

Let's do this in bug 1642591

Comment 19

[Emilio Cobos Álvarez (:emilio)]

We had to change rel=preload behavior to deal with perf issues in bug 1642325, so we're already kinda inconsistent here, fwiw.