bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Linux cache settings for new profile: 5000KB disk; 3000KB mem

VERIFIED FIXED in mozilla0.9.1

Status

()

Core
Networking: Cache
--
major
VERIFIED FIXED
17 years ago
7 years ago

People

(Reporter: John Morrison, Assigned: Jatin Billimoria)

Tracking

({perf, relnote})

Trunk
mozilla0.9.1
x86
Linux
perf, relnote
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: release note only now..)

Attachments

(1 attachment)

(Reporter)

Description

17 years ago
I have no idea how or when this started, but I noticed that linux was slowing
down progressively when running the page loader (i.e., each successive round 
was a bit slower, until "supposedly cached" times were ~equal to "uncached" times).

I created a new profile on Linux and got settings of 5MB disk, 3MB mem cache,
which are the values listed in 'unix.js'. I don't get this. Were these always
set like this? [I don't think so, but lxr says yes].


(As a side observation, I noticed that the eviction didn't appear to be
"balanced" among the pages. The first pages visited in the cycle pretty 
much kept all their images in the cache, and the last pages visited kept
pretty much none. I thought that it would have been more FIFO than it looked.
I'll show you what I mean if you want).
(Reporter)

Updated

17 years ago
Keywords: mozilla0.9, nsbeta1, perf

Comment 1

17 years ago
see bug 77218
(Reporter)

Comment 2

17 years ago
er, yeah, so it was always set to these values (and since hyatt only changed
the value in all.js, these have stayed here all along. Don't know why it took
me till now to notice this).

I'm surprised that a hardcore Unix hacker like Dave would have missed this. ;-]

Comment 3

17 years ago
I do hope you folks realize that 50MB may well be far too large for some Unix
installations. Think about quotas. One size does not fit all, particularly when
that size comes from Windows.

Comment 4

17 years ago
cc'ing hyatt, since he was involved in the original proposal to change the 
default settings.
Also, someone made the point that 50MB should take into account internal
fragmentation overhead, which can be quite severe on FAT filesystems if the
cache uses lots of little files.  That 50MB should not be based on net sizes of
files.

tenthumbs: care to propose a better default?  5MB was considered too small.  How
would 10MB do?  Any data on average quota size?

/be

Comment 6

17 years ago
I agree that the disk cache should use disk space as criteria for reporting/
respecting limits rather than the data size of the entries stored.  Is there an 
XP way of determining file block sizes?

Comment 7

17 years ago
I have no hard data but every quotad system I've seen has had small
quota (<10MB).  That would make the old 5MB about right which is
probably why it is that value. Hoever, it's also true that these systems
tend to have nearby proxy servers which alleviate the problem.

That does bring up another issue. On Linux/Unix one can have an
industrial strength proxy server available. I have squid on my own
machine so I can serve all my browsers. It has a 100MB cache so a large
mozilla cache is rather useless.

I don't think there's an XP way to measure block size. In Unix, struct
stat has the st_blksize and st_blocks fields which tell you what you
need but st_size may be meaningless. In Windows it's the other way
around.

And no one seems to have considered performance with 50MB of small files
in one directory. That's a lot of files and that's generally considered
a Very Bad Thing on Unix.

I suppose I could see 10MB for a default size but I'm wondering whether
something else isn't going on here. If you assume a typical page is
about 500K of data then 5MB should store 10 pages. If mozilla has
trouble managing 10 pages then there's a lot more wrong than the cache
size.

Finally, 50MB makes mozilla's disk cache into a true caching proxy but
mozilla's just not up to that yet. You folks should look at squid's
config file. It considers things that mozilla hasn't even thought of.

Comment 8

17 years ago
It's not that no one has considered the performance issues of many small files in 
one directory, it's that it's a separate problem.  I'm working on changes to the 
disk cache to store chunks of data smaller than a certain threshold in a single 
file.  After that is done, we will tune the number of cache subdirectories for 
the remaining files (the old cache used 16 subdirectories).

Comment 9

17 years ago
I wasn't clear. What I meant to say was that mozilla should _not_ bump
up the default size on Unix until the performance issue is addressed.
Unix mozilla is slow enough as it is. There's no reason to make it
worse. That's particularly true for a milestone release.

Comment 10

17 years ago
Using block files is scheduled for this release (bug 72507).

Comment 11

17 years ago
But bug 72507 talks in terms of a flat file system which is an issue.
I've been running some tests to see if, and how much, a performance
penalty there is on a Linux ext2 fs. So far, the kernel's memory
management and swapping policies are creating confusing results. In any
event, this issue needs testing on all the Unix systems mozilla makes
any pretensions of supporting.

Comment 12

17 years ago
I think it's important for the 0.9.1 milestone to come to some consensus on the 
value of these preference settings.  Setting target milestone to 0.9.1.
Target Milestone: --- → mozilla0.9.1
Our (backup'ed, hence expensive) home partition is limited in size. Without
manual intervention, the disk cache is created in the user's home. 5 MB per user
is just the maximum that can be tolerated without major headache. If the default
cache size is increased significantly, there has to be a very-well documented,
easy way for a unix sysadmin to either 
a) change the default cache size back to a reasonable value, or
b) change the default cache location for new profiles from, say,
   /home/dummyuser/.netscape/cache to 
   /lots-of-space/no-backup/dummyuser/netscape-cache
   for all users with a single small change.
Personally, I don't like the idea of hiding 50 MB in an "invisible" directory.

Comment 14

17 years ago
a common solution to this problem is to configure your backup scripts to skip
over any cache directories.  if this is inconvenient, you could also edit
all.js (in the mozilla distribution) and set the pref:

pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb
The "common solution" is not applicable here: backup cost is calculated from
partition size, and even if the cache dirs were skipped they would take up space
on our small home partition.
The pref in all.js may work, thanks for the hint. So in case you increase the
default setting, please advertise this method widely (maybe in the release
notes, plus putting a note in the README where instructions & docs for sysadmins
can be found).

Comment 16

17 years ago
Created attachment 35718 [details] [diff] [review]
Patch for unix.js as per discussion with PDT.

Comment 17

17 years ago
New defaults for unix have been checked in.  Assigning bug to Steve Rudman (as 
per discussion with PDT) so issues can be added to release notes.
Assignee: gordon → rudman

Comment 18

17 years ago
-->Jatin. I asked Gordon to clarify what the issues are, look for additional 
comments in the bug.
Assignee: rudman → jatin

Comment 19

17 years ago
I'm going to mark this fixed.  do we have a release note key word we can 
hang this on now?
Status: NEW → RESOLVED
Last Resolved: 17 years ago
Resolution: --- → FIXED
Whiteboard: release note only now..

Comment 20

17 years ago
adding relnote keyword.
Keywords: relnote

Comment 21

17 years ago
Andreas Franke asked us to document the "common solution" that Darin describes 
(about 6 comments) above.
VERIFIED fixed on a new profile on linux 2001060713 build.
Status: RESOLVED → VERIFIED

Comment 23

17 years ago
-relnote, fixed.
Keywords: relnote
(Reporter)

Comment 24

17 years ago
Did this get added to release notes? (The issue about 50MB for disk cache being 
more than some shared environments would want?).
Just in case there was any confusion, I'll quote my relnote request again:

> >  You can edit all.js (in the mozilla distribution) and set the pref:
> > 
> >	pref("browser.cache.disk_cache_size", 5000); // set cache size to 5Mb
>
> This pref may work, thanks for the hint. So in case you increase the
> default setting, please advertise this method widely (maybe in the release
> notes, plus putting a note in the README where instructions & docs for 
> sysadmins can be found).
(Assignee)

Comment 26

17 years ago
Does this still need to be in the 0.9.3 and RTM release notes? If so, please add
a comment to the tracking bug (Bug 90577) with a one line description of the bug
and the workaround. 

Comment 27

17 years ago
+relnote - I messed this up. I read the status but not the entire bug when I was
triaging networking bugs that were not mine.
Keywords: relnote
You need to log in before you can comment on or make changes to this bug.