1687569 - cache partitioning causes web page saving extensions to waste more time and data

Status: UNCONFIRMED

(WebExtensions :: General, task, P3)

Description

[Nijaz]

Attachment: cache partition.jpg

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0

Steps to reproduce:

I downloaded page saving extensions from AMO like SingleFile, SingleFileZ, and Save Page WE.

Actual results:

Then I noticed slow time and double resources downloads using 3rd party apps like HTTP Debugger, Charles Proxy, Telerik FIddler, BurpSuite...
For example I save some site, and I see 2 images being reuested instead of one, 20 images instead of 10, 10 MB instead of 5 MB.
I went to about:debugging#/runtime/this-firefox and Inspect the 3 extensions each.
In Network tab of Developer tools that open after clicking Inspect button I monitored network requests.
They do not show to use cache, except if:
SingleFile or SingleFileZ is used more than once on same url, for example by saving same page twice or by cancelling first save, and resaving.
If we change code in two files of extension Save Page WE to remove cache-control whole code line, because author added Cache-Control: no-store header. Author said such header will be removed in next version. Then Save Page WE behaves just like SingleFIle.
In Chrome I noticed slowdown from about 1 seconds to 1 minute to save page, because Chrome would previously allow Save Page WE to reuse cache.
Chrome invented something called "cache partitioning" and that is 2 months ago https://developers.google.com/web/updates/2020/10/http-cache-partitioning, and that's when I noticed slowdown with page saving extensions.
You invented it too in Firefox version 70. https://bugzilla.mozilla.org/show_bug.cgi?id=1536058

Expected results:

You should allow page saving extensions to use already existing cache as they could in versions before Firefox 70. Otherwise anybody using such extensions will have time and data about double.

Comment 1

[Nijaz]

In attached image I show entries in about:cache for same resource from wikihow article when page is loaded normally and when loaded via Save Page WE (modified to remove Cache-Control: no-store code lines). As you can see there is double cached entry, separately cached by extension and web page. You should allow extensions, at least those for saving pages to access already existing cache. Also preference in about:config browser.cache.cache_isolation is already set to false, so how do I turn off cache partitioning? Either that is not same as cache isolation and there is some secret preference or it is prevented secretly from changing just like preference for installing unsigned addons in release version of Firefox?

Comment 2

[Nijaz]

bug 1536058

Comment 3

[Nijaz]

Just tested this version: https://ftp.mozilla.org/pub/firefox/releases/68.12.0esr/win64/en-US/Firefox%20Setup%2068.12.0esr.exe
and it allows extensions to access cache and SingleFile, SingleFileZ, Save Page WE (for current version you have to remove code line in two files mentioning "no-store") to download files extremely fast. That is last ESR (because allows unsigned addons) version before you invented cache partitioning. So I'll use that method until you fix or improve newer versions.

Comment 4

[Nijaz]

In case somebody wants to test here is modified extension with no-store removed: https://drive.google.com/file/d/1oiAIaToSOCli-brJ0v687xydfW7cEz3V/view?usp=sharing
Just remember to disable xpinstall.signatures.required in about:config

Comment 5

[Nijaz]

I solved all my problems by disabling (setting to false) these in about:config :
privacy.partition.network_state
network.http.rcwn.enabled

The 2nd one is potential speed improver but data waster.
Hope addons developers can disable those two settings via extensions just like uBlock origin can disabled prefetching.

Comment 6

[Anne (:annevk)]

As far as I can tell extensions already have the privilege to access the partitioned cache, so this is likely something that would have to change in the extension.

Shave, Tomislav, thoughts?

Comment 7

[Nijaz]

New version of Save Page WE is released with removed cache-control: no-store header, so no need to use my unsigned extension from google drive anymore.

Comment 8

[Shane Caraveo (:mixedpuppy)]

(In reply to Anne (:annevk) from comment #6)

As far as I can tell extensions already have the privilege to access the partitioned cache, so this is likely something that would have to change in the extension.

Shave, Tomislav, thoughts?

We have no tests illustrating that any API works with partitioned cache, one test where it is disabled so apparently some problem there.

Anne, Correct me if I am misunderstanding something:

I would expect an image to be downloaded twice due to cache partitioning, that is the point isn't it? Origin A gets it's own cache, Origin B gets it's own. If A is a webpage and B is an extension background page, they would end up with partitioned caches. I'm not certain what the bug is here, other than that the report assumes that an extension should bypass its own partition and avoid a second download.

The only way I can see an addon possibly working around that is to handle the download in a content script on the page, so the partition used is that of the page and not the extension. I'm not sure that works as we don't have a test for it, but it's the only way that makes sense. Given that works, this would be an outreach bug to the downloaders to update how they work.

On the other hand, if there is a way to configure XHR/Fetch to operate as-if they are in the context of a given origin, then an extension could do this from the background page. If there were something like XHR.usePartition (only in privileged code, or extension origins) then the origin of the request url could be used as the partition to pull from. That could avoid extraneous downloads.

Comment 9

[Anne (:annevk)]

Thanks Shane, that seems correct. If there's a need for such a bypass we could offer it I think, it seems consistent with the extensions security model.

Comment 10

[Shane Caraveo (:mixedpuppy)]

Ok, so this is not a defect, but the side effect is perhaps not really desirable.

I think that this bug can move to extensions and that the next step is to verify whether handling downloads in a content script allows an extension to avoid the double download caused by partitioning. If that is possible, the next step would be to consider whether extensions should have a way to "impersonate" an origin to effectively use that origins cache. The latter sounds reasonable, but we need to think through that from an extension perspective first.

We'll re-triage next week or week after.

Comment 11

[dw-dev]

I am the developer of Save Page WE.

I have tried using XHR in a content script to fetch resources cached by the browser's page load, but this does not work. The resources are still fetched over the network. The resources fetched by XHR are saved in the cache, but without a partition key.

From my perspective, I would much prefer to have a way to configure XHR/Fetch to operate as-if they are in the context of a given origin, as suggested by Shane in Comment 8. Then an extension could fetch resources from the browser cache from either a content script or the background script.

Comment 12

[Shane Caraveo (:mixedpuppy)]

We're going to look into a combination of issues with requests made by extensions where various types of "partitions" are causing issues (cache, containers, etc).

Comment 13

[Rob Wu [:robwu]]

Tim, is this the same as bug 1691907?

Comment 14

[Rob Wu [:robwu]]

Tim, I meant to needinfo you - is this the same as bug 1691907 ? (and more generally fixed by bug 1698863)

Comment 15

[Tim Huang[:timhuang]]

Yeah, I believe this is the same issue here.

Comment 17

[Nijaz]

Looks like this is not fixed in 89.0a1 (20210410091448). Neither is 1691907

Comment 18

[Rob Wu [:robwu]]

Coul you provide one precise (ideally minimal) test case that demonstrates the issue?

Bug 1691907 supposedly fixed a common case. For other cases bug 1698863 would be the general solution.

Comment 19

[Nijaz]

1. Make new Firefox profile by using something like ""C:\Program Files\Firefox Nightly\firefox.exe" -P" depending on location of Firefox executable. Make new profile from there.

2. Open Firefox and install "Download All Images" extension (https://addons.mozilla.org/en-US/firefox/addon/save-all-images-webextension/)

3. Open URL: https://www.ghacks.net/2021/04/07/mozilla-decides-to-hide-compact-mode-in-firefox-for-new-users-but-keep-it-for-existing-ones/

4. Click on "Download All Images" toolbar icon. In deep "search" choice in top right corner change 1 to 0 so that extension will only capture images already loaded by current page. Press Cancel button in bottom right corner.

5. Clear cache: ctrl+shift+del, time range to clear: everything, select cache from choices, nothing else needed.

6. Open URL: https://www.ghacks.net/2021/04/07/mozilla-decides-to-hide-compact-mode-in-firefox-for-new-users-but-keep-it-for-existing-ones/ either by copy pasting it into address bar or by clicking with pointer once into that bar and pressing enter key. Do not use Refresh button!

7. Open network monitor (ctrl+shift+e).

8. Repeat step 6. Scroll page from top to bottom and back to top to load lazy loading images, either with touchpad/mouse slowly, or by holding Page Down keyboard key.

9. You will notice that all entries in network monitor show "cached" or "raced" in transferred column (except main maybe one of very few files which are normally not cached).

10. Open in new tab (next to existing one, do not close existing one) URL: about:debugging#/runtime/this-firefox

11. Inspect Download All Images extension, by pressing Inspect button in that tab, and make sure that network tab is active in developer tools that open.

12. Click on "Download All Images" toolbar icon. You will see in developer tools (network monitor) of that extension that all resources are downloaded from beginning, meaning no word "cached" or "raced" in "transferred" column, but shows just size of each resource (in "transferred" column). After few seconds click Cancel button in bottom right corner.

13. Open URL in new tab: about:cache?storage=disk
    Search for this using CTRL+F keyboard shortcut: https://mk0ghacksnety2pjrgh8.kinstacdn.com/wp-content/uploads/2021/04/Firefox-88-Open-Image-in-new-tab-300x300.jpg
    You will see 2 results, the only difference being that one has partition key below url, while other entry (next one in search results) does not: O^partitionKey=%28https%2Cghacks.net%29,

To repeat tests: never refresh ghacks.net page but load it by clicking once into url with cursor and pressing enter key, clear network monitor of extension, and repeat steps 5 - 13 only.

If you set preferences in about:config "network.http.rcwn.enabled" and "privacy.partition.network_state" to false and repeat steps above (5-13 only) you will notice only one entry for some image url in about:cache?storage=disk, as well as word "cached" in network monitor (developer tools) of extension too.

Tests would not be valid if you press refresh button because or if you use extension's toolbar button twice, pressing cancel in extension window to be able to do that, and clearing network monitor of extension between tries, then even extension would have it's resources used from cached which it previously stored under separate (actually without) partition key. Sorry, that is the shortest way to explain all of this.

Comment 20

[Nijaz]

Forgot to mention in step 6 too: Scroll page from top to bottom and back to top to load lazy loading images, either with touchpad/mouse slowly, or by holding Page Down keyboard key.

Comment 21

[Rob Wu [:robwu]]

ni me to look into comment 19