1648075 - HLS playlist *.m3u8 requests are cached despite a no-cache header

Status: RESOLVED INVALID

(Core :: Networking: HTTP, defect)

Description

[Marc]

Attachment: player.zip

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36

Steps to reproduce:

1. Wipe the Firefox cache in the preferences menu.
2. Add a *.m3u8 file (HLS playlist) to a webspace (file attached) or a Google Cloud Storage
3. Assure that the webserver adds the "Cache-Control: max-age:0, no-cache" header to the response
4. Paste the *.m3u8 file URL in another static HTML page and use it in a JS-based video player (like VideoJS, Bitmovin Player) or use the attached script example file in player.html (creating only a HTTP GET request with jQuery).

Actual results:

The player or script sends the first GET request to retrieve the *.m3u8 file URL.
The request fetches the file from the server the first time. Fine.
But the following requests of this file are served from the cache despite the provided HTTP header with cache-control: max-age:0, no-cache).

In the network tab in the web console the requests are shown as retrieved from cache (see screenshot.png).

This behaviour causes HLS livestream playbacks to fail.
Because the player is not able to get the updated playlist file (containing the new chunk files) from the server. Only the cached first version of the playlist file is used.

The first time I noticed this issue the *.m3u8 file was served from a Google Cloud Storage. This is an example for the response header:

HTTP/1.1 200 OK
X-GUploader-UploadID: <id>
Date: Wed, 24 Jun 2020 11:43:53 GMT
Cache-Control: max-age:0, no-cache
Expires: Thu, 24 Jun 2021 11:43:53 GMT
Last-Modified: Thu, 18 Jun 2020 15:36:58 GMT
ETag: "etag"
x-goog-generation: 1592494618956903
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 346534
Content-Type: application/x-mpegURL
x-goog-hash: <removed>
x-goog-hash: <removed>
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 346534
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length, Content-Type, Date, Server, Transfer-Encoding, X-GUploader-UploadID, X-Google-Trace
Server: UploadServer

Expected results:

The XHR requests generated by the player or the script are not cached as required by the provided HTTP response header (cache-control: max-age:0, no-cache).

With Firefox 76 this issue is not reproducible.
This issue seems to be related to the release v77.0.
Also current Chrome v83 does not show this behaviour.

Some other HLS live streaming services may be not be affected by that.
Maybe due to additonal query parameters in the m3u8 URL.
Some add the "Pragma: no-cache" instead of Cache-Control.
Services that use MPEG DASH are not affected too.

All tests with following user agent:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Marc]

I was able to replicate this behaviour on firefox 78.0.1.
The first GET request of the *.m3u8 file is retrieved from the server. All subsequent requests are served from the cache.
Therefore HLS live streams suddenly stop, because the *.m3u8 file is not updated due to caching.

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Does v77.0 introduced some changes affecting the interpretation of the "Cache-Control: max-age:0, no-cache" header?
Maybe in connection with the Expires header?

According to the documentation the Expires header should be ignored if a there is a Cache-Control header in the response.
But due to the wrong Expires header (1 year in the future) the file is loaded from the cache.
And not retrieved from the server.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expires

Comment 3

[John Lin [:jhlin][:jolin]]

Sounds like a HTTP protocol cache control issue. I'll move it to that component.

Comment 4

[Marc]

Yes it seems to be related to the HTTP caching.
Yesterday I ran further tests with a local Nginx web server and different HTTP response header combinations.
It seems that following combination (future Expires timestamp!) forces Firefox to ignore the Cache-Control: max-age:0 header.

Cache-Control: max-age:0, no-cache
Expires: Thu, 24 Jun 2021 11:43:53 GMT

I would have expected following behaviour.
If both Expires and max-age are set, the Cache-Control: max-age will take precedence.

Using a correct Expires timestamp and e.g. Cache-Control: max-age=1 seems to solve the problem.
All requests are forwarded to the server as expected.
So the Expires header is evaluated despite the existing Cache-Control: max-age.

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 02 Jul 2020 12:49:03 GMT
Content-Type: application/x-mpegURL
Content-Length: 346534
Last-Modified: Thu, 02 Jul 2020 12:49:00 GMT
Connection: keep-alive
ETag: "5efdd7bc-549a6"
Expires: Thu, 02 Jul 2020 12:49:04 GMT
Cache-Control: max-age=1
Access-Control-Allow-Origin: *
Accept-Ranges: bytes

Comment 5

[Dragana Damjanovic [:dragana]]

Honza, can you take a look?

Comment 6

[Honza Bambas (:mayhemer)]

The correct form is max-age=0. max-age:0, no-cache will make us break at the : and then fail as we won't find neither of EOF or ,. We started to use this parser for responses in 77 in bug 1542293.

A possible fix to ignore the broken directive and allow parsing of the no-cache token would be to add : to word char list here.

Comment 7

[Honza Bambas (:mayhemer)]

https://tools.ietf.org/html/rfc7234#section-5.2
: is not allowed in a token:

Cache-Control   = 1#cache-directive

     cache-directive = token [ "=" ( token / quoted-string ) ]

token          = 1*tchar

     tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                    / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                    / DIGIT / ALPHA
                    ; any VCHAR, except delimiters

VCHAR = any visible [USASCII](https://tools.ietf.org/html/rfc7230#ref-USASCII) character