647391 - Increase maximum size for objects stored in disk cache

Status: RESOLVED FIXED

(Core :: Networking: Cache, enhancement)

Description

[Matthew Middleton (:zzxc)]

In Firefox 4, there is a hard limit of 5MB per cached file.  This prevents users on slow/metered connections, who often increase the cache size, from storing large images and videos in cache.

The maximum size should be configurable to larger sizes when the total cache limit is sufficiently large.

Comment 1

[Jo Hermans]

I totally agree. I noticed that the limit was imposed because of fear that large files might push out most or all files in the cache. 5MB was appropriate for a cache-size of 0MB. But a few days later, the cache was also increased up to 1 GB.

Comment 2

[Lozzy]

I also find this to be detrimental to my browsing experience. I've explained my thoughts here; http://support.mozilla.com/en-US/questions/808767

The outline of it is that I hate it I have to wait for the video to load again due to it not being cached. This can be caused by a number of things, including; crashes, losing net connection and navigating away from the page.

I found the same issue just as irritating with Flash, and was hoping that Firefox's implementation of the video tag would not be a repeat of that.

Overall, being able to cache videos has various benefits, and I'm not short on space, so I'd like to be able to cache them please!

Comment 3

[Jason Duell]

Yes, we should change this.  It may have to wait until we've got a better algorithm than plain LRU for the disk cache (ie. bug 578541).

Note that the media cache does store >5MB media files, which handles the common case of watching a Youtube video twice, etc.  Is that not working for you?

Comment 4

[Jason Duell]

On some more thought, I think this is a sensible short-term plan (since bug 578541 is not going to happen very soon):  

1) Start honoring the "browser.cache.disk.max_entry_size" pref (bug 650995)

2) Bump the default size of the pref for Firefox to something bigger than 5 MB (20 MB?  50 MB?  The number will be overridden if it's larger than 1/8th of the cache).

Step #1 is needed because fennec is going to want a much smaller limit (less than 5 MB, in fact).  It will also allow power users to set the limit as high as they like.

Comment 5

[Lozzy]

(In reply to comment #3)

> Note that the media cache does store >5MB media files, which handles the common
> case of watching a Youtube video twice, etc.  Is that not working for you?

I'm afraid not, no. Using Firefox 4.0, refreshing the page will trigger a
redowload of the file. This is the same for every video I have tried, not just
Youtube. As a point of reference, Chromium loads the video from cache on a
refresh or after closing the tab and later revisiting the page.

Nightly versions can't be tested due to bug 649500 (Youtube does UA sniffing
and serves the video using flash to 6.0pre anyway)

Incidentally, to test Youtube in HTML5/Webm mode quickly without having to set
the preference, you can add "&webm=1&html5=1" to the end of the URI. For
example; http://www.youtube.com/watch?v=MlqLU_tYx7A&webm=1&html5=1

I also think your proposed short term solution sounds good. It would settle my use case at least, but not non power users.

Comment 7

[Lozzy]

Which reminds me; media cache seems to be working better now in the nightlies. There's still a couple of peculiarities though; there seem to be a few instances where the cached version of the file is ignored.

I haven't found a solid STR, but I know that certain combinations of reloading/closing tab/changing Youtube quality settings can cause a redownload. Often, simply changing the quality setting a couple of times could cause it.

Also, it seems to only cache the video sequentially from the start rather than caching all the chunks received, but that's not important.

Comment 8

[Jason Duell]

Attachment: Increase max size of objects stored in disk cache to 50 MB from 5 MB

OK, so now that we've got prefs for max_cache_object_size (bug 650995), we should go ahead and raise the max size of files cached on disk from 5 MB to something substantially bigger.

Data to support this:  The Download Video Helper extension has 94 *million* downloads.  

  https://addons.mozilla.org/en-US/firefox/addon/video-downloadhelper/

If we store vids in cache, it doesn't have to re-download them over the net, and instead gets them from cache.  Very noticeable for large downloads.

We're also getting lots of flack in our Help forums for not caching video files any more as of FF 4:

  http://support.mozilla.com/en-US/questions/769414  
  http://support.mozilla.com/en-US/questions/807132 
  http://forums.mozillazine.org/viewtopic.php?f=38&t=2141531
  http://support.mozilla.com/en-US/questions/809086

On the other hand, until we have a size-aware eviction algorithm for the disk cache, allowing truly huge files does mean we run the risk of scenarios like "watch 4 250 MB videos in a row == blow away all the other entries in your 1 GB cache".  But it's unlikely that users will watch that many videos of that length in a row.

This patch raises the limit to 50 MB (1/20th of the max cache size for desktops).  It also fixes some comments.   I'm open to suggestions to use some other size.  Youtube videos use about 3.6 MB/minute, so this is enough to store 13 minutes of video.

Comment 9

[Lozzy]

50MB does sound quite reasonable as a default. I'm considering a value of between 250-500 myself ... will I need to set browser.cache.memory.max_entry_size to the same, or will it be sufficient to just increase browser.cache.disk.max_entry_size?

Also, what is the status of the media cache? Has it been decided that it would be simpler to handle large/media files in the standard cache?

Comment 10

[Jason Duell]

> I'm considering a value of between 250-500 myself

Note that we have a "hard" limit of never storing anything bigger than 1/8th of the total cache size, which is 1 GB max, so the effective limit is 125 MB, not matter what you set the pref to.   Perhaps that needs fixing too :(

> what is the status of the media cache? Has it been decided that it would be simpler to handle large/media files in the standard cache?

A very good question.  I'll ask the media cache folks, to make sure we're not doing anything stupid/redundant.

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

See http://mxr.mozilla.org/mozilla-central/source/content/media/nsMediaCache.h#51

The media cache is not persistent, it only caches media during a browser session. And normal HTTP loads don't use it, only media elements use it. Also, when the server supports byte-range requests we frequently use them, so as I understand it, those loads don't go into the normal cache anyway.

So I think double-caching media files is not likely to be a big problem in practice.

Comment 12

[Lozzy]

(In reply to comment #10)
> > I'm considering a value of between 250-500 myself
> 
> Note that we have a "hard" limit of never storing anything bigger than 1/8th
> of the total cache size, which is 1 GB max, so the effective limit is 125
> MB, not matter what you set the pref to.   Perhaps that needs fixing too :(

Yes, I've just been bitten by this limit myself accidentally, it was a regular video which I had thought was going to be cached, but turned out it wasn't since it was 219MB. All things considered - videos gaining more ubiquity on the web and increasing in quality/size - it does seem like a good idea to look at Firefox's cache limits. After all, we can only expect the trend to rise, not fall.

By the way, I've found it convenient to have these larger size objects stored in the general cache. It's only small perks, but don't think I would get the same benefits if I was relying on the media cache.

Comment 13

[Jason Duell]

http://hg.mozilla.org/integration/mozilla-inbound/rev/52702d275995

Comment 14

[Matt Brubeck (:mbrubeck)]

Backed out for xpcshell test failures: http://hg.mozilla.org/integration/mozilla-inbound/rev/29232fea6e19

http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1312332890.1312334816.24742.gz

TEST-INFO | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | running test ...
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>
### XPCOM_MEM_LEAK_LOG defined -- logging leaks to /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpM1biJ-/runxpcshelltests_leaks.log
pldhash: for the table at address 0x466bb8, the given entrySize of 48 probably favors chaining over double hashing.

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-PASS | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | [sync_with_cache_IO_thread : 34] null == null

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | (xpcshell/head.js) | running event loop

TEST-PASS | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | [oCEA : 50] 2152398909 == 2152398909

TEST-PASS | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | [sync_with_cache_IO_thread : 34] null == null

TEST-PASS | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | [oCEA : 50] 2152398909 == 2152398909

TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | undefined - See following stack:
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_throw :: line 445
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js :: append_datafile :: line 104
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: <TOP_LEVEL> :: line 157

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js | undefined - See following stack:
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: do_throw :: line 445
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/tests/netwerk/test/unit/test_bug654926.js :: append_datafile :: line 111
JS frame :: /Users/cltbld/talos-slave/test/build/xpcshell/head.js :: <TOP_LEVEL> :: line 157

TEST-INFO | (xpcshell/head.js) | exiting test

Comment 15

[Michal Novotny [:michal]]

Attachment: patch v2 - fixed test_bug654926.js

Pushed to try server

https://tbpl.mozilla.org/?tree=Try&usebuildbot=1&rev=e5e8ee415d3f

Comment 16

[Michal Novotny [:michal]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/d13840232672

Comment 17

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/d13840232672

Comment 19

[Jason Duell]

Not sure if this needs documentation somewhere:

Firefox/Gecko will now cache objects on disk up to 50 MB in size (from previous FF 4+ limit of 5 MB).  (to be precise: there is still a 1/8th of total cache size limit, so if your cache is 400 MB or less--which is rare--you'll have a lower limit).

Comment 20

[fireattack [:fireattack]]

Er, are you sure?

I just tried neweast Nightly and seems this problem is still there.

Comment 21

[Tony Mechelynck [:tonymec]]

(In reply to Benjamin Peng from comment #20)
> Er, are you sure?
> 
> I just tried neweast Nightly and seems this problem is still there.

The latest nightly has a "build ID" datestamp of 20110914030906, i.e., about 4 hours before comment #17, so it doesn't include the fix.

You can get a Firefox build with the fix from http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ then the mozilla-central-* subdirectory corresponding to your software platform, and then the most recent subdirectory with a "numeric" name.

Comment 22

[fireattack [:fireattack]]

(In reply to Tony Mechelynck [:tonymec] from comment #21)
> (In reply to Benjamin Peng from comment #20)
> > Er, are you sure?
> > 
> > I just tried neweast Nightly and seems this problem is still there.
> 
> The latest nightly has a "build ID" datestamp of 20110914030906, i.e., about
> 4 hours before comment #17, so it doesn't include the fix.
> 
> You can get a Firefox build with the fix from
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ then the
> mozilla-central-* subdirectory corresponding to your software platform, and
> then the most recent subdirectory with a "numeric" name.

Oh, I see. Thanks.

Comment 23

[Eric Shepherd [:sheppy]]

Docs updated:

https://developer.mozilla.org/en/Mozilla_Networking_Preferences#Cache

Mentioned on Firefox 9 for developers.