443067 - Disk Cache overflow when downloading large files.

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect)

Description

[cn]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0

When using 200MB disk cache and downloading files (tested with files >90MB), downloaded content will be added to cache, increasing the in-use cache value ( - why is a download added to cache in the first place by the way?). However, once the download has finished, the file that was created in the cache directory gets deleted BUT the in-use disk cache value stays the same. So after downloading several big files, the in-use cache value will far exceed the maximum cache size, resulting in the cached-files "Number of entries" value to display "0" and no more files to be cached at all until the cache is cleared manually.

Reproducible: Always

Steps to Reproduce:
1.Download several big files while watching about:cache (Disk Storage in use)
2.
3.


Expected Results:  
Basically i did expect that download would not be cached to begin with, but that is a minor issue. That aside, the in-use cache should obviously not exceed the maximum cache value when downloading big files, and it also should never just become "stuck" if this happens (requiring manual cache-clearing).

I am not using any plugins that should be interfering with the caching system.

Comment 1

[cn]

note also that while all cached files do get deleted once the in-use cache exceeds the maximum value, about:cache still reports the in-use value greater than the maximum value (despite no files being currently cached at all).

Comment 2

[Glenn Maynard]

I see this constantly, or at least the same end result:

Disk cache device

Number of entries: 	0
Maximum storage size: 	256000 KiB
Storage in use: 	655365 KiB

The actual cache file is about 50 megs, and nothing is ever cached (even loading Google's front page reloads all of the script files every time).  I've seen this behavior for a long time (currently in 3.5.3 and I remember it from 3.0.x, at least).

After clearing the cache, it starts working again (temporarily).

See bug 531700 (probably others, too) for the issue about caching downloaded files.

Comment 5

[Boris Zbarsky [:bzbarsky]]

Confirming due to the multiple reports of this.  Bjarne, Michal, could one of you take a look at this?

Comment 7

[Michal Novotny [:michal]]

I'll take a look at it.

Comment 8

[Michal Novotny [:michal]]

Attachment: patch v1

Maximum filesize was reduced from 65536Kib to 65535Kib. Size in KiB is rounded up, so for range 0x3FFFC01-0x4000000 it is 0x10000 and the filesize in nsDiskCacheRecord overflows.

Comment 9

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 439198 [details] [diff] [review]
patch v1

r=bzbarsky

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 439198 [details] [diff] [review]
patch v1

Per reed and Jesse in #developers we don't really need sr with this.

(Bug 569709 did not have sr at all)

Comment 11

[Justin Wood (:Callek)]

http://hg.mozilla.org/mozilla-central/rev/eb8e32b47158 unbitrotted too

Comment 12

[Alfred Kayser]

Instead of using a hex number 0x3FFFC00, one could also do:
#define kMaxDataFileSize   (65535 * 1024)   // 65535 KiB (see bug #443067)

Comment 15

[Radek 'sysKin' Czyz]

Verified fixed with hourly, changeset 7d3e81dd1018. Downloading a ~100 MB file with 250 MB disk cache no longer advances cache "Storage in use" counter at all. Nice~

Comment 16

[LuceferAB]

STILL NOT FIXED! 

about:cache?device=disk
Disk cache device
Number of entries: 	0
Maximum storage size: 	819200 KiB
Storage in use: 	1376241 KiB
Cache Directory: 	C:\Documents and Settings\Lucefer\Local Settings\Application Data\Mozilla\Firefox\Profiles\mwe0jyv2.default\Cache
------

Comment 17

[LuceferAB]

about:buildconfig

Source

Built from http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c07c6c1b5071
Build platform
target
i686-pc-mingw32

Build tools
Compiler 	Version 	Compiler flags
cl 	14.00.50727.762 	-TC -nologo -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
cl 	14.00.50727.762 	-GR- -TP -nologo -Zc:wchar_t- -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1

Configure arguments
--enable-application=browser --enable-update-channel=release --enable-update-packaging --enable-jemalloc --enable-official-branding --enable-tests

Comment 18

[Justin Wood (:Callek)]

(In reply to comment #17)
> about:buildconfig
> 
> Source
> 
> Built from http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c07c6c1b5071

I'll note that this was fixed on trunk. aka Firefox 4/Gecko 2.0 not on the 1.9.2 branch.

I have no idea how invasive/hard this fix would be for the Firefox version you are using, but it did not land there, and would need approval to do so (if the patch even applies cleanly).

If you feel it is worth it to land there, first see if the patch applies cleanly and fixes the problem; if both are true, request approval and provide a clear risk assessment (if you can't assess the risk cleanly, convince someone who can to). If it does NOT apply cleanly, either update the patch yourself, or convince someone else to, and then follow the same approval request dance as I outlined above.  This *IS* fixed on trunk.

Comment 20

[Michal Novotny [:michal]]

Comment on attachment 439198 [details] [diff] [review]
patch v1

I've tested this patch on 1.9.2 branch and it applies cleanly and works correctly.

Comment 21

[Daniel Veditz [:dveditz]]

Comment on attachment 439198 [details] [diff] [review]
patch v1

non-blocking code-freeze for 1.9.2.17 was last week, try next time.

Comment 22

[al_9x]

(In reply to comment #20)
> Comment on attachment 439198 [details] [diff] [review]
> patch v1
> 
> I've tested this patch on 1.9.2 branch and it applies cleanly and works
> correctly.

Could you try for 3.6.18?

Comment 23

[al_9x]

(In reply to comment #20)
> Comment on attachment 439198 [details] [diff] [review] [review]
> patch v1
> 
> I've tested this patch on 1.9.2 branch and it applies cleanly and works
> correctly.

Please try to get this into 3.6.18

Comment 24

[Daniel Veditz [:dveditz]]

Comment on attachment 439198 [details] [diff] [review]
patch v1

code-freeze for 1.9.2.18 was June 6. I'd move the approval forward to .19 myself but this wait-til-the-last-minute pattern makes me dubious it would land in time or that there's really much interest in this bug.

Comment 25

[al_9x]

This is (In reply to comment #24)
> Comment on attachment 439198 [details] [diff] [review] [review]
> patch v1
> 
> code-freeze for 1.9.2.18 was June 6. I'd move the approval forward to .19
> myself but this wait-til-the-last-minute pattern makes me dubious it would
> land in time or that there's really much interest in this bug.

This is a fairly serious bug in core functionality, it effectively disables the cache and all the user knows is that Fx feels a bit slower, but would not know why or what to do about it.  Increasing the cache and downloading large files fairly regularly, which is all it takes to run into this, is not an unusual usage pattern.

@Michal Novotny - can you do this now for .19?

Comment 26

[al_9x]

https://bugzilla.mozilla.org/show_bug.cgi?id=614039#c20

Comment 28

[LuceferAB]

FF 3.6.20 - Bug still not fixed!!!

Disk cache device
Number of entries: 	0
Maximum storage size: 	1048576 KiB
Storage in use: 	1093882 KiB
----
To reproduce: Clean install, then browsing and saving some files from http://www.fileserve.com, http://www.filesonic.com/, youtube....

Comment 29

[James May [:fowl2]]

The current release version of Firefox is 6.0. If you can still reproduce the problem in that please file a new bug.