Closed Bug 230125 Opened 18 years ago Closed 12 years ago

about:cache disk Storage In Use value is random once a download passes the 64MB mark - possible overflow?

Categories

(Core :: Networking: Cache, defect)

defect
Not set
normal

Tracking

()

RESOLVED DUPLICATE of bug 443067

People

(Reporter: wd, Unassigned)

References

()

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7a) Gecko/20040103
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7a) Gecko/20040103

Once more than 64MB of a large file has been downloaded, the "Storage in use" in
the disk section of about:cache will display a random incorrect value

Reproducible: Always

Steps to Reproduce:
1.View about:cache
2.start downloading a file larger than 64MB (from http://www.linuxiso.org for
example)
3.Keep on reloading about:cache

Actual Results:  
The Storage in Use value constantly grows until it reaches around 64000 KiB
Once it reaches that amount, the value displayed is some random value.   For
example, mine is currently 1900544 KiB which is impossible.  (I don't have that
much disk space where my profile is)

Expected Results:  
Display correct value
Blocks: 229984
Confirmed on Mac OS X 10.2.8 w/ build 2004010505 - so all/all

See bug 229984 comment 4 for my test-setup
OS: Windows 2000 → All
Hardware: PC → All
Note: This is for HTTP downloads only
To reproduce, try an HTTP link from this URL:
http://www.knopper.net/knoppix-mirrors/index-en.html
Possibly related to bug 218391.
Once an HTTP download passes the 64MB mark, I get a stream of the following asserts:

###!!! ASSERTION: data size out of range: 'sizeK < USHRT_MAX', file 
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/src/nsDiskCacheDevice.cpp#688
###!!! ASSERTION: data size out of range: 'newSizeK < USHRT_MAX', file 
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/src/nsDiskCacheDevice.cpp#689
###!!! ASSERTION: disk cache size negative?: 'mHeader.mDataSize >= 0', file 
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/src/nsDiskCacheMap.h#469
###!!! ASSERTION: disk cache size negative?: 'mHeader.mDataSize >= 0', file 
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/src/nsDiskCacheMap.h#460

(File references changed to LXR links)
Depends on: 218391
Assignee: darin → nobody
I am having a similar issue here (Firefox 3, Windows Vista 32bit SP1). 
When using 200MB disk cache and downloading files >64MB, downloaded content will be added to cache, increasing the in-use cache value ( - why is a download added to cache in the first place by the way?). However, once the download has finished, the file that was created in the cache directory gets deleted BUT the in-use disk cache value stays the same. So after downloading several big files, the in-use cache value will far exceed the maximum cache size, resulting in the cached-files "Number of entries" value to display "0" and no more files to be cached at all until the cache is cleared manually.
Suggestion 1.  Please alter criteria to remove file from disk cache.  Files are removed if they are larger than half the total cache size or if the disk file is larger than kMaxDataFileSize [nDiskCacheDevice.cpp:687].  For users with a large enough cache size (>128 MiB), the hard-coded limit of 64 MiB per file [nDiskCacheMap.h:94] is unnecessary and inconvenient.

Suggestion 2.  Make kMaxDataFileSize [nDiskCacheMap.h:94] user-definable (about:config).  Does this have side-effects?



[nDiskCacheMap.h]
93    #define kSeparateFile      0
94    #define kMaxDataFileSize   0x4000000   // 64 MiB
95    #define kBuckets           (1 << 5)    // must be a power of 2!
96    
97    #class nsDiskCacheRecord {


[nDiskCacheDevice.cpp]
666    nsresult
667    nsDiskCacheDevice::OnDataSizeChange(nsCacheEntry * entry, PRInt32 deltaSize)

...

682        PRUint32  newSize = entry->DataSize() + deltaSize;
683        PRUint32  newSizeK =  ((newSize + 0x3FF) >> 10);
684
685        // If the new size is larger than max. file size or larger than
686        // half the cache capacity (which is in KiB's), doom the entry and abort
687        if ((newSize > kMaxDataFileSize) || (newSizeK > mCacheCapacity/2)) {
688            nsresult rv = nsCacheService::DoomEntry(entry);
689            NS_ASSERTION(NS_SUCCEEDED(rv),"DoomEntry() failed.");
690            return NS_ERROR_ABORT;
691        }
692    
693        PRUint32  sizeK = ((entry->DataSize() + 0x03FF) >> 10); // round up to next 1k
694
695        NS_ASSERTION(sizeK < USHRT_MAX, "data size out of range");
696        NS_ASSERTION(newSizeK < USHRT_MAX, "data size out of range");


Could someone please fix this?  There are many sites which contain streaming flash videos and the only way to download them is to copy the video from the Firefox cache.  When Firefox has a hardcoded 64M limit on the cache file size, that means that it is impossible to download many flash videos.  It can't be that hard to make kMaxDataFileSize configurable.
So can we increase this limit altogether might make sense in this day.  The default disk cache size has just been bumped to 256M.

For example, some WebM videos on high resolution even pushes past the current media cache size of 50M.
Probably fixed by bug 443067?
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 443067
That sounds like it might be a duplicate the original issue--getting strange results when the cache reaches close to its maximum.  However, the issue of not being able to set a limit higher than 64M remains.  There are some videos which can be downloaded *only by copying them from the cache* and if Firefox refuses to allow a higher cache size these videos can't be downloaded.  Period.  This should be kept open for the issue of raising the 64M limit.
Correction: I meant to say cache file size, of course.  It's the file size that's causing the problem, not the total cache size.
As written in comment #9, this is fixed by bug 443067.

Can you please retest the issue with a nightly build after June 14, 2010.
So first update to the latest nightly to be sure that this fix is included.
(In reply to comment #10)
> However, the issue of not  being able to set a limit higher than 64M remains. 

Only the 64M overflow limit issue has been fixed from comment 0 and rightfully so.  

Another bug should be filed specifically to bump the file cache size or make it configurable and it will probably will be better addressed in a new bug (i haven't seen an open one yet).

I only mentioned the disk cache and large files from the web because some are so big that i agree we need to address your point.  BTW, default Disk Cache size is back to 50M.
You need to log in before you can comment on or make changes to this bug.