1893842 - stale-while-revalidate causes revalidation of the wrong page

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect, P2)

Description

[coolcat_the_best2]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0

Steps to reproduce:

The setting

I have a setup that uses the same URL for normal requests and for HTMX requests. This works perfectly, since I can branch on the presence of the HX-Request header on the server side.

Today I started playing with the preload extension. I artificially increased the server's response time to 1 second and saw that the preloading caused the page to be requested from the server twice.

No problem there. It just meant I had to specify a Cache-Control header in the response and settled on Cache-Control: max-age=60. This however exposed my next issue: when first loading the page through a normal request and then triggering a hx-get HTMX request, the target element of the call suddenly contained a copy of the whole webpage, instead of just the element that the HTMX request was supposed to load.

This actually made sense, because the normal request and HTMX request were sharing the same cache instance. To fix this, I added the Vary: HX-Request header to the response, so that the browser knew to cache the normal and HTMX requests separately. So far, so good.

The issue

While researching caching and the Cache-Control header, I found the stale-while-revalidate value.

The stale-while-revalidate response directive indicates that the cache could reuse a stale response while it revalidates it to a cache.

In the name of a responsive website, I of course had to try it out. It works well for normal requests. The problem however came when I tried to use this for HTMX requests. This was especially true when using the preload extension.

Actual results:

Actual

When I trigger a preload on a link that has a hx-get attribute, I get 2 network events:

The second one however isn't triggered by HTMX, but is triggered by the browser to revalidate the request. The request headers for this request don't contain the HX-Request header:

The result

When the link is actually clicked, the cache is still stale and the browser will actually send the HTMX request, rather than using the stale cached value.

Also, since the revalidation triggered the caching for the normal http request, which returns the whole page, it wastes precious bandwidth and server/client time, since only the revalidation of the HTMX request was necessary.

Expected results:

Expected

• Whenever a stale page is requested by HTMX within the stale-while-revalidate time window, I expect HTMX to load the response from cache. After that, I expect a revalidation to be triggered for that HTMX request (with the HX-Request header).
• Whenever a stale page is preloaded by the HTMX Preload extension within the stale-while-revalidate time window, I expect the browser to trigger a revalidation (with or without "loading" the page from cache) for that HTMX request (with the HX-Request header).

Possible solution

It would be nice if any request headers that were provided in the Vary response header (here HX-Request) would also be used when revalidating that request.

HTMX version: 1.9.12
Browser: Mozilla Firefox 125.0.1
Example repo: https://github.com/CC007/htmx-poc

It seems that MS Edge does correctly send along the headers from the previous request (and more than just the headers from the Vary response header):

Using Microsoft Edge 124.0.2478.51

Comment 1

[coolcat_the_best2]

It seems the images didn't load properly. You can find them here: https://github.com/bigskysoftware/htmx/issues/2517

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Untriaged' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

Example repo: https://github.com/CC007/htmx-poc

I don't have any experience with htmx. Could you provide some instructions of how to run the demo?
Thanks!

Comment 4

[coolcat_the_best2]

I added a README.md file to the repo, that explains how to build and run the application

Comment 5

[Sergiu Prodan]

I have a similar issue with Firefox 133. The background requests made during the stale-while-revalidate period are done without the custom headers I send in the original request.

Example flow:
T0 - Request:
GET /resource
...
x-custom-header: somevalue
...
T0 - Response:
204
...
Vary: x-custom-header
cache-control: max-age=5,stale-while-revalidate=30
...

T0 + 7seconds Request:
GET /resource
...
x-custom-header: somevalue
...
T0 +7seconds - Response:
204 No content
...
Vary: x-custom-header
cache-control: max-age=5,stale-while-revalidate=30
...

At this point Firefox issues a background request without the x-custom-header which then results in a different than expected outcome.

Comment 6

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

The revalidation is triggered from nsHttpChannel::OnCacheEntryCheck.
nsHttpChannel::PerformBackgroundCacheRevalidationNow should also be copying the headers, ideally by calling HttpBaseChannel::SetupReplacementChannel. (not sure if calling nsHttpChannel::SetupReplacementChannel is appropriate here)

We also need a test for this. We can probably add to the existing XPC or WPT tests https://searchfox.org/mozilla-central/search?q=stale-while-reval&path=&case=false&regexp=false

Comment 7

[Sunil Mayya]

Attachment: Bug 1893842 - extend test_stale-while-revalidate_positive.js to include custom headers. r=#necko

Comment 8

[Sunil Mayya]

Attachment: Bug 1893842 - background validation requests should include headers from original requests. r=#necko

Comment 9

[Sunil Mayya]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #6)

nsHttpChannel::PerformBackgroundCacheRevalidationNow should also be copying the headers, ideally by calling HttpBaseChannel::SetupReplacementChannel. (not sure if calling nsHttpChannel::SetupReplacementChannel is appropriate here)

I think we could simply copy the headers using CopyNonDefaultHeaderVisitor

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d38a54de08c6
extend test_stale-while-revalidate_positive.js to include custom headers. r=necko-reviewers,valentin
https://hg.mozilla.org/integration/autoland/rev/9c29aa2968f2
background validation requests should include headers from original requests. r=necko-reviewers,valentin

Comment 11

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/d38a54de08c6
https://hg.mozilla.org/mozilla-central/rev/9c29aa2968f2