I have no idea how or when this started, but I noticed that linux was slowing down progressively when running the page loader (i.e., each successive round was a bit slower, until "supposedly cached" times were ~equal to "uncached" times). I created a new profile on Linux and got settings of 5MB disk, 3MB mem cache, which are the values listed in 'unix.js'. I don't get this. Were these always set like this? [I don't think so, but lxr says yes]. (As a side observation, I noticed that the eviction didn't appear to be "balanced" among the pages. The first pages visited in the cycle pretty much kept all their images in the cache, and the last pages visited kept pretty much none. I thought that it would have been more FIFO than it looked. I'll show you what I mean if you want).
see bug 77218
er, yeah, so it was always set to these values (and since hyatt only changed the value in all.js, these have stayed here all along. Don't know why it took me till now to notice this). I'm surprised that a hardcore Unix hacker like Dave would have missed this. ;-]
I do hope you folks realize that 50MB may well be far too large for some Unix installations. Think about quotas. One size does not fit all, particularly when that size comes from Windows.
cc'ing hyatt, since he was involved in the original proposal to change the default settings.
Also, someone made the point that 50MB should take into account internal fragmentation overhead, which can be quite severe on FAT filesystems if the cache uses lots of little files. That 50MB should not be based on net sizes of files. tenthumbs: care to propose a better default? 5MB was considered too small. How would 10MB do? Any data on average quota size? /be
I agree that the disk cache should use disk space as criteria for reporting/ respecting limits rather than the data size of the entries stored. Is there an XP way of determining file block sizes?
I have no hard data but every quotad system I've seen has had small quota (<10MB). That would make the old 5MB about right which is probably why it is that value. Hoever, it's also true that these systems tend to have nearby proxy servers which alleviate the problem. That does bring up another issue. On Linux/Unix one can have an industrial strength proxy server available. I have squid on my own machine so I can serve all my browsers. It has a 100MB cache so a large mozilla cache is rather useless. I don't think there's an XP way to measure block size. In Unix, struct stat has the st_blksize and st_blocks fields which tell you what you need but st_size may be meaningless. In Windows it's the other way around. And no one seems to have considered performance with 50MB of small files in one directory. That's a lot of files and that's generally considered a Very Bad Thing on Unix. I suppose I could see 10MB for a default size but I'm wondering whether something else isn't going on here. If you assume a typical page is about 500K of data then 5MB should store 10 pages. If mozilla has trouble managing 10 pages then there's a lot more wrong than the cache size. Finally, 50MB makes mozilla's disk cache into a true caching proxy but mozilla's just not up to that yet. You folks should look at squid's config file. It considers things that mozilla hasn't even thought of.
It's not that no one has considered the performance issues of many small files in one directory, it's that it's a separate problem. I'm working on changes to the disk cache to store chunks of data smaller than a certain threshold in a single file. After that is done, we will tune the number of cache subdirectories for the remaining files (the old cache used 16 subdirectories).
I wasn't clear. What I meant to say was that mozilla should _not_ bump up the default size on Unix until the performance issue is addressed. Unix mozilla is slow enough as it is. There's no reason to make it worse. That's particularly true for a milestone release.
Using block files is scheduled for this release (bug 72507).
But bug 72507 talks in terms of a flat file system which is an issue. I've been running some tests to see if, and how much, a performance penalty there is on a Linux ext2 fs. So far, the kernel's memory management and swapping policies are creating confusing results. In any event, this issue needs testing on all the Unix systems mozilla makes any pretensions of supporting.
I think it's important for the 0.9.1 milestone to come to some consensus on the value of these preference settings. Setting target milestone to 0.9.1.
Target Milestone: --- → mozilla0.9.1
Our (backup'ed, hence expensive) home partition is limited in size. Without manual intervention, the disk cache is created in the user's home. 5 MB per user is just the maximum that can be tolerated without major headache. If the default cache size is increased significantly, there has to be a very-well documented, easy way for a unix sysadmin to either a) change the default cache size back to a reasonable value, or b) change the default cache location for new profiles from, say, /home/dummyuser/.netscape/cache to /lots-of-space/no-backup/dummyuser/netscape-cache for all users with a single small change. Personally, I don't like the idea of hiding 50 MB in an "invisible" directory.
a common solution to this problem is to configure your backup scripts to skip over any cache directories. if this is inconvenient, you could also edit all.js (in the mozilla distribution) and set the pref: pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb
The "common solution" is not applicable here: backup cost is calculated from partition size, and even if the cache dirs were skipped they would take up space on our small home partition. The pref in all.js may work, thanks for the hint. So in case you increase the default setting, please advertise this method widely (maybe in the release notes, plus putting a note in the README where instructions & docs for sysadmins can be found).
New defaults for unix have been checked in. Assigning bug to Steve Rudman (as per discussion with PDT) so issues can be added to release notes.
Assignee: gordon → rudman
-->Jatin. I asked Gordon to clarify what the issues are, look for additional comments in the bug.
Assignee: rudman → jatin
I'm going to mark this fixed. do we have a release note key word we can hang this on now?
Status: NEW → RESOLVED
Last Resolved: 18 years ago
Resolution: --- → FIXED
Whiteboard: release note only now..
adding relnote keyword.
Andreas Franke asked us to document the "common solution" that Darin describes (about 6 comments) above.
VERIFIED fixed on a new profile on linux 2001060713 build.
Status: RESOLVED → VERIFIED
Did this get added to release notes? (The issue about 50MB for disk cache being more than some shared environments would want?).
Just in case there was any confusion, I'll quote my relnote request again: > > You can edit all.js (in the mozilla distribution) and set the pref: > > > > pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb > > This pref may work, thanks for the hint. So in case you increase the > default setting, please advertise this method widely (maybe in the release > notes, plus putting a note in the README where instructions & docs for > sysadmins can be found).
Does this still need to be in the 0.9.3 and RTM release notes? If so, please add a comment to the tracking bug (Bug 90577) with a one line description of the bug and the workaround.
+relnote - I messed this up. I read the status but not the entire bug when I was triaging networking bugs that were not mine.
You need to log in before you can comment on or make changes to this bug.