Disk Cache overflow when downloading large files.

RESOLVED FIXED

Status

()

RESOLVED FIXED
10 years ago
7 years ago

People

(Reporter: vexicate, Assigned: michal.novotny)

Tracking

({perf})

Trunk
x86
Windows Vista
Points:
---

Firefox Tracking Flags

(blocking2.0 beta1+)

Details

Attachments

(1 attachment)

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0

When using 200MB disk cache and downloading files (tested with files >90MB), downloaded content will be added to cache, increasing the in-use cache value ( - why is a download added to cache in the first place by the way?). However, once the download has finished, the file that was created in the cache directory gets deleted BUT the in-use disk cache value stays the same. So after downloading several big files, the in-use cache value will far exceed the maximum cache size, resulting in the cached-files "Number of entries" value to display "0" and no more files to be cached at all until the cache is cleared manually.

Reproducible: Always

Steps to Reproduce:
1.Download several big files while watching about:cache (Disk Storage in use)
2.
3.


Expected Results:  
Basically i did expect that download would not be cached to begin with, but that is a minor issue. That aside, the in-use cache should obviously not exceed the maximum cache value when downloading big files, and it also should never just become "stuck" if this happens (requiring manual cache-clearing).

I am not using any plugins that should be interfering with the caching system.
(Reporter)

Updated

10 years ago
Version: unspecified → 3.0 Branch
(Reporter)

Comment 1

10 years ago
note also that while all cached files do get deleted once the in-use cache exceeds the maximum value, about:cache still reports the in-use value greater than the maximum value (despite no files being currently cached at all).

Comment 2

9 years ago
I see this constantly, or at least the same end result:

Disk cache device

Number of entries: 	0
Maximum storage size: 	256000 KiB
Storage in use: 	655365 KiB

The actual cache file is about 50 megs, and nothing is ever cached (even loading Google's front page reloads all of the script files every time).  I've seen this behavior for a long time (currently in 3.5.3 and I remember it from 3.0.x, at least).

After clearing the cache, it starts working again (temporarily).

See bug 531700 (probably others, too) for the issue about caching downloaded files.
(Reporter)

Updated

9 years ago
Version: 3.0 Branch → 3.5 Branch

Updated

9 years ago
Duplicate of this bug: 552467
Duplicate of this bug: 538549
Confirming due to the multiple reports of this.  Bjarne, Michal, could one of you take a look at this?
Status: UNCONFIRMED → NEW
Component: General → Networking: Cache
Ever confirmed: true
Product: Firefox → Core
QA Contact: general → networking.cache
Version: 3.5 Branch → Trunk

Updated

9 years ago
Duplicate of this bug: 556251
(Assignee)

Comment 7

9 years ago
I'll take a look at it.

Updated

9 years ago
Keywords: perf
(Assignee)

Comment 8

9 years ago
Created attachment 439198 [details] [diff] [review]
patch v1

Maximum filesize was reduced from 65536Kib to 65535Kib. Size in KiB is rounded up, so for range 0x3FFFC01-0x4000000 it is 0x10000 and the filesize in nsDiskCacheRecord overflows.
Assignee: nobody → michal.novotny
Attachment #439198 - Flags: review?(bzbarsky)
Comment on attachment 439198 [details] [diff] [review]
patch v1

r=bzbarsky
Attachment #439198 - Flags: review?(bzbarsky) → review+
blocking2.0: --- → ?
(Assignee)

Updated

9 years ago
Attachment #439198 - Flags: superreview?(cbiesinger)

Updated

9 years ago
blocking2.0: ? → beta1+
Comment on attachment 439198 [details] [diff] [review]
patch v1

Per reed and Jesse in #developers we don't really need sr with this.

(Bug 569709 did not have sr at all)
Attachment #439198 - Flags: superreview?(cbiesinger)
http://hg.mozilla.org/mozilla-central/rev/eb8e32b47158 unbitrotted too
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED

Comment 12

9 years ago
Instead of using a hex number 0x3FFFC00, one could also do:
#define kMaxDataFileSize   (65535 * 1024)   // 65535 KiB (see bug #443067)

Updated

9 years ago
Duplicate of this bug: 218391

Updated

9 years ago
Duplicate of this bug: 230125

Comment 15

9 years ago
Verified fixed with hourly, changeset 7d3e81dd1018. Downloading a ~100 MB file with 250 MB disk cache no longer advances cache "Storage in use" counter at all. Nice~

Comment 16

8 years ago
STILL NOT FIXED! 

about:cache?device=disk
Disk cache device
Number of entries: 	0
Maximum storage size: 	819200 KiB
Storage in use: 	1376241 KiB
Cache Directory: 	C:\Documents and Settings\Lucefer\Local Settings\Application Data\Mozilla\Firefox\Profiles\mwe0jyv2.default\Cache
------

Comment 17

8 years ago
about:buildconfig

Source

Built from http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c07c6c1b5071
Build platform
target
i686-pc-mingw32

Build tools
Compiler 	Version 	Compiler flags
cl 	14.00.50727.762 	-TC -nologo -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
cl 	14.00.50727.762 	-GR- -TP -nologo -Zc:wchar_t- -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1

Configure arguments
--enable-application=browser --enable-update-channel=release --enable-update-packaging --enable-jemalloc --enable-official-branding --enable-tests
(In reply to comment #17)
> about:buildconfig
> 
> Source
> 
> Built from http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c07c6c1b5071

I'll note that this was fixed on trunk. aka Firefox 4/Gecko 2.0 not on the 1.9.2 branch.

I have no idea how invasive/hard this fix would be for the Firefox version you are using, but it did not land there, and would need approval to do so (if the patch even applies cleanly).

If you feel it is worth it to land there, first see if the patch applies cleanly and fixes the problem; if both are true, request approval and provide a clear risk assessment (if you can't assess the risk cleanly, convince someone who can to). If it does NOT apply cleanly, either update the patch yourself, or convince someone else to, and then follow the same approval request dance as I outlined above.  This *IS* fixed on trunk.
status1.9.2: --- → ?
(Assignee)

Updated

8 years ago
Duplicate of this bug: 614039
(Assignee)

Comment 20

8 years ago
Comment on attachment 439198 [details] [diff] [review]
patch v1

I've tested this patch on 1.9.2 branch and it applies cleanly and works correctly.
Attachment #439198 - Flags: approval1.9.2.17?
status1.9.2: ? → ---
Comment on attachment 439198 [details] [diff] [review]
patch v1

non-blocking code-freeze for 1.9.2.17 was last week, try next time.
Attachment #439198 - Flags: approval1.9.2.17? → approval1.9.2.17-

Comment 22

8 years ago
(In reply to comment #20)
> Comment on attachment 439198 [details] [diff] [review]
> patch v1
> 
> I've tested this patch on 1.9.2 branch and it applies cleanly and works
> correctly.

Could you try for 3.6.18?

Comment 23

8 years ago
(In reply to comment #20)
> Comment on attachment 439198 [details] [diff] [review] [review]
> patch v1
> 
> I've tested this patch on 1.9.2 branch and it applies cleanly and works
> correctly.

Please try to get this into 3.6.18
(Assignee)

Updated

8 years ago
Attachment #439198 - Flags: approval1.9.2.18?
Comment on attachment 439198 [details] [diff] [review]
patch v1

code-freeze for 1.9.2.18 was June 6. I'd move the approval forward to .19 myself but this wait-til-the-last-minute pattern makes me dubious it would land in time or that there's really much interest in this bug.
Attachment #439198 - Flags: approval1.9.2.18? → approval1.9.2.18-

Comment 25

8 years ago
This is (In reply to comment #24)
> Comment on attachment 439198 [details] [diff] [review] [review]
> patch v1
> 
> code-freeze for 1.9.2.18 was June 6. I'd move the approval forward to .19
> myself but this wait-til-the-last-minute pattern makes me dubious it would
> land in time or that there's really much interest in this bug.

This is a fairly serious bug in core functionality, it effectively disables the cache and all the user knows is that Fx feels a bit slower, but would not know why or what to do about it.  Increasing the cache and downloading large files fairly regularly, which is all it takes to run into this, is not an unusual usage pattern.

@Michal Novotny - can you do this now for .19?
(Assignee)

Updated

7 years ago
Duplicate of this bug: 614039

Comment 28

7 years ago
FF 3.6.20 - Bug still not fixed!!!

Disk cache device
Number of entries: 	0
Maximum storage size: 	1048576 KiB
Storage in use: 	1093882 KiB
----
To reproduce: Clean install, then browsing and saving some files from http://www.fileserve.com, http://www.filesonic.com/, youtube....

Comment 29

7 years ago
The current release version of Firefox is 6.0. If you can still reproduce the problem in that please file a new bug.
You need to log in before you can comment on or make changes to this bug.