80890 - Linux cache settings for new profile: 5000KB disk; 3000KB mem

Status: VERIFIED FIXED

(Core :: Networking: Cache, defect)

Description

[John Morrison]

I have no idea how or when this started, but I noticed that linux was slowing
down progressively when running the page loader (i.e., each successive round 
was a bit slower, until "supposedly cached" times were ~equal to "uncached" times).

I created a new profile on Linux and got settings of 5MB disk, 3MB mem cache,
which are the values listed in 'unix.js'. I don't get this. Were these always
set like this? [I don't think so, but lxr says yes].


(As a side observation, I noticed that the eviction didn't appear to be
"balanced" among the pages. The first pages visited in the cycle pretty 
much kept all their images in the cache, and the last pages visited kept
pretty much none. I thought that it would have been more FIFO than it looked.
I'll show you what I mean if you want).

Comment 1

[Gilles Durys]

see bug 77218

Comment 2

[John Morrison]

er, yeah, so it was always set to these values (and since hyatt only changed
the value in all.js, these have stayed here all along. Don't know why it took
me till now to notice this).

I'm surprised that a hardcore Unix hacker like Dave would have missed this. ;-]

Comment 3

[tenthumbs]

I do hope you folks realize that 50MB may well be far too large for some Unix
installations. Think about quotas. One size does not fit all, particularly when
that size comes from Windows.

Comment 4

[gordon]

cc'ing hyatt, since he was involved in the original proposal to change the 
default settings.

Comment 5

[Brendan Eich [:brendan]]

Also, someone made the point that 50MB should take into account internal
fragmentation overhead, which can be quite severe on FAT filesystems if the
cache uses lots of little files.  That 50MB should not be based on net sizes of
files.

tenthumbs: care to propose a better default?  5MB was considered too small.  How
would 10MB do?  Any data on average quota size?

/be

Comment 6

[gordon]

I agree that the disk cache should use disk space as criteria for reporting/
respecting limits rather than the data size of the entries stored.  Is there an 
XP way of determining file block sizes?

Comment 7

[tenthumbs]

I have no hard data but every quotad system I've seen has had small
quota (<10MB).  That would make the old 5MB about right which is
probably why it is that value. Hoever, it's also true that these systems
tend to have nearby proxy servers which alleviate the problem.

That does bring up another issue. On Linux/Unix one can have an
industrial strength proxy server available. I have squid on my own
machine so I can serve all my browsers. It has a 100MB cache so a large
mozilla cache is rather useless.

I don't think there's an XP way to measure block size. In Unix, struct
stat has the st_blksize and st_blocks fields which tell you what you
need but st_size may be meaningless. In Windows it's the other way
around.

And no one seems to have considered performance with 50MB of small files
in one directory. That's a lot of files and that's generally considered
a Very Bad Thing on Unix.

I suppose I could see 10MB for a default size but I'm wondering whether
something else isn't going on here. If you assume a typical page is
about 500K of data then 5MB should store 10 pages. If mozilla has
trouble managing 10 pages then there's a lot more wrong than the cache
size.

Finally, 50MB makes mozilla's disk cache into a true caching proxy but
mozilla's just not up to that yet. You folks should look at squid's
config file. It considers things that mozilla hasn't even thought of.

Comment 8

[gordon]

It's not that no one has considered the performance issues of many small files in 
one directory, it's that it's a separate problem.  I'm working on changes to the 
disk cache to store chunks of data smaller than a certain threshold in a single 
file.  After that is done, we will tune the number of cache subdirectories for 
the remaining files (the old cache used 16 subdirectories).

Comment 9

[tenthumbs]

I wasn't clear. What I meant to say was that mozilla should _not_ bump
up the default size on Unix until the performance issue is addressed.
Unix mozilla is slow enough as it is. There's no reason to make it
worse. That's particularly true for a milestone release.

Comment 10

[gordon]

Using block files is scheduled for this release (bug 72507).

Comment 11

[tenthumbs]

But bug 72507 talks in terms of a flat file system which is an issue.
I've been running some tests to see if, and how much, a performance
penalty there is on a Linux ext2 fs. So far, the kernel's memory
management and swapping policies are creating confusing results. In any
event, this issue needs testing on all the Unix systems mozilla makes
any pretensions of supporting.

Comment 12

[gordon]

I think it's important for the 0.9.1 milestone to come to some consensus on the 
value of these preference settings.  Setting target milestone to 0.9.1.

Comment 13

[Andreas Franke (gone)]

Our (backup'ed, hence expensive) home partition is limited in size. Without
manual intervention, the disk cache is created in the user's home. 5 MB per user
is just the maximum that can be tolerated without major headache. If the default
cache size is increased significantly, there has to be a very-well documented,
easy way for a unix sysadmin to either 
a) change the default cache size back to a reasonable value, or
b) change the default cache location for new profiles from, say,
   /home/dummyuser/.netscape/cache to 
   /lots-of-space/no-backup/dummyuser/netscape-cache
   for all users with a single small change.
Personally, I don't like the idea of hiding 50 MB in an "invisible" directory.

Comment 14

[Darin Fisher]

a common solution to this problem is to configure your backup scripts to skip
over any cache directories.  if this is inconvenient, you could also edit
all.js (in the mozilla distribution) and set the pref:

pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb

Comment 15

[Andreas Franke (gone)]

The "common solution" is not applicable here: backup cost is calculated from
partition size, and even if the cache dirs were skipped they would take up space
on our small home partition.
The pref in all.js may work, thanks for the hint. So in case you increase the
default setting, please advertise this method widely (maybe in the release
notes, plus putting a note in the README where instructions & docs for sysadmins
can be found).

Comment 16

[gordon]

Attachment: Patch for unix.js as per discussion with PDT.

Comment 17

[gordon]

New defaults for unix have been checked in.  Assigning bug to Steve Rudman (as 
per discussion with PDT) so issues can be added to release notes.

Comment 18

[rudman]

-->Jatin. I asked Gordon to clarify what the issues are, look for additional 
comments in the bug.

Comment 19

[chris hofmann]

I'm going to mark this fixed.  do we have a release note key word we can 
hang this on now?

Comment 20

[Asa Dotzler [:asa]]

adding relnote keyword.

Comment 21

[gordon]

Andreas Franke asked us to document the "common solution" that Darin describes 
(about 6 comments) above.

Comment 22

[Bradley Baetz (:bbaetz)]

VERIFIED fixed on a new profile on linux 2001060713 build.

Comment 23

[benc]

-relnote, fixed.

Comment 24

[John Morrison]

Did this get added to release notes? (The issue about 50MB for disk cache being 
more than some shared environments would want?).

Comment 25

[Andreas Franke (gone)]

Just in case there was any confusion, I'll quote my relnote request again:

> >  You can edit all.js (in the mozilla distribution) and set the pref:
> > 
> >	pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb
>
> This pref may work, thanks for the hint. So in case you increase the
> default setting, please advertise this method widely (maybe in the release
> notes, plus putting a note in the README where instructions & docs for 
> sysadmins can be found).

Comment 26

[Jatin Billimoria]

Does this still need to be in the 0.9.3 and RTM release notes? If so, please add
a comment to the tracking bug (Bug 90577) with a one line description of the bug
and the workaround. 

Comment 27

[benc]

+relnote - I messed this up. I read the status but not the entire bug when I was
triaging networking bugs that were not mine.