123080 - Cache data storage in profile bloats and slows roaming profiles

Status: RESOLVED DUPLICATE of bug 74085

(Core :: Networking: Cache, defect)

Description

[Kevin Lange]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.7) Gecko/20011221
BuildID:    2001122106

Currently, mozilla stores expendable cache data in a subdirectory under the
mozilla application data folder (C:\Documents and
Settings\<username>\Application Data\Mozilla\Profiles\<Mozilla Profile
Name>\9kb608th.slt\Cache).  For a Windows roaming profile, all that cache data
is then copied to the networked roaming profile directory on the profile server
when the user logs out, (and copied back when they log back in).  Copying all
that data across a network can take 5-10 minutes given a large number of cached
files across a couple of mozilla profiles (not windows profiles).  I had two
mozilla profiles that contributed about 80-85% of a 200MB windows profile. 95%
of that data is useless cache data. Every time I log out and log in, that data
is transferred across the network.  Some shops put quota limits on the profiles,
and this would cause further problems.

I would assume (have not tested) that this is a problem for any version of
Windows where users' Windows profiles are stored on a profile server (roaming
profile).  I currently have tested and reproduced this problem on win NT4.0,
NT2000, WinXP.  I'm not sure about Windows 95/98/ME and how they handle the profile.

Reproducible: Always
Steps to Reproduce:
1.  Create a Windows NT user with a roaming Windows profile on a networked
profile server
2.  Login to a networked workstation as that user.
3.  Create two fresh Mozilla profiles
4.  Create one POP3/IMAP mail account for each Mozilla profile
5.  Launch mozilla for each profile, loading sufficient pages to fill cache
directory to maximum
6.  Connect to mail server per each mozilla profile, (say a 200-300 msg inbox)
7.  Close mozilla
8.  Examine summary size of the "C:\Documents and
Settings\<username>\Application Data\Mozilla" directory (folder properties).

Actual Results:  A very large mozilla directory (50-200mb) depending on mailbox
size and type as well as maximums on the browser's cache sizes.  Consequently, a
long time for users to log off/log on to Windows when they are using a roaming
windows profile (5-10 minutes) as the profile is synchronized to the profile
server. Our profile server is a dual CPU Compaq with 1GB memory and RAID
storage.   The workstations are 1GHz pentium III each with 256MB memory, 100mb
ethernet full-duplex connectivity through two Cisco switches.

Expected Results:  I would Suggest storing the cache data in the more
appropriate local TIF directory (C:\Documents and Settings\<Username>\Local
Settings\Temporary Internet Files).

Also, consider using the C:\Documents and Settings\<Username>\Local
Settings\Application Data directory to store data that isn't expendable  but
doesn't need to be shared via roaming profile. (perhaps mailbox index files)?

I have been able to work around this shortcoming with a couple of procedures:

1. Set my mail server settings "Local Directory" to store data outside of my
profile. ie instead of storing mail server data into "C:\Documents and
Settings\<Username>\Application Data\Mozilla\Profiles\<Profile
Name>\9kb608th.slt\Mail\<mail account name>", I specify a folder outside of the
profile location, like a local disk or such.  This helps to reduce the size of
the windows profile due to mail indexes, etc.  This is an extremely complex
procedure, and not practical for shops with novice users.

2. Flush all caches before logging off.  This is a bit impractical to say the
least.  But it keeps the profile size down.

Comment 1

[Matthias Versen [:Matti]]

dupe of bug 46490


*** This bug has been marked as a duplicate of 46490 ***

Comment 2

[Conrad Carlen (not reading bugmail)]

Only partial dup. Bug 46490 is about the necko cache. This bug brings up another
valid complaint about mail storage which is a different component. Kevin, can
you see if there's a bug against the mailnews component for your problem? If
not, take the portion on this bug about mailnews and file a new bug.

Comment 3

[K'Trina Medina]

Verified as a dupe of bug 46490 (re: necko cache)

Reminder: 
Reporter, please submit a separate bug for your mailnews component problems.

Comment 4

[benc]

REOPEN:
I'm particularly interested in making this feature work well in this environment.

Kevin, have you made this work for you?

Comment 5

[Kevin Lange]

Ok...It's true that both the Mail and Browser cache directories are user
configureable and at that, they satisfy my requirements at margin.  

It would be much more preferable if, when on an Windows platform that fully
supports the notion of Profiles, and more specifically the C:\Documents and
Settings\<username>\Local Settings\ (not sure which version of Windows
introduced this), that Mozilla Browser cache was automatically placed in the
Local Settings directory (as in c:\Documnets and Settings\<username>\Local
Settings\Application Data\).  Anything under 'Local Settings' is not propogated
to a profile server when sychronizing a Roaming Profile.  For instance,
Microsoft stores its cache data under C:\Documents and Settings\<username>\Local
Settings\Temporary Internet Files.  They also store History under there as
well...it's a judgement call depending on the potential amount of data.  Whether
to store data in the Local Settings or not is similar to designing an LDAP
directory store...you don't want to burden the LDAP directory to the point where
it's bloated with data that could be stored in a database somewhere.

Comment 6

[Jason M. Abels]

I agree that this is something that needs to be addressed. This isn't really a
bug so much as it is an RFE. I'm not going to change that status though. Just
confirming that it is, indeed, a problem we need to fix.

Comment 7

[benc]

conrad:

Are there other bugs about other files that would benefit from this logic
(mailnews mbox indexes?)

kevin:
Is there an API that tells us we using a roaming profile?

Comment 8

[Kevin Lange]

I'm not aware of a specific API call (though I would think there would be one)
that determine if the profile is local or roaming.  I'm not a heavy duty MS
developer by any means (I'm a UNIX admin).  I'm not certain that you would need
to make that determination though; regardless of whether a profile is roaming or
not, just by placing transient cache data in the 'local settings' directory
would suffice.  If the profile is roaming the data won't be copied to the
profile server.  If the profile is local (not roaming), it won't really matter
anyway.

With regards to use of the local settings and what could benefit....Any
transient data (such as cache, indexes, etc) would be better suited in the
'local settings' directory.  

Comment 9

[David Schwarz]

I'd like to add the the following should be placed in the "local data" 
directory.

CACHE:
all files should be moved to "local settings"

IMAPMAIL:
everything in IMAPmail directory, Execept the mailfilter rules file

MAIL:
its arguable that files in the mail directory should not be moved (as this 
data can not be recreated)

NEWS:
I am not sure what should go from the NEWS directory, but all the directories 
that are created should not be there.

With thse changes corporate use will be easier

Comment 10

[David Schwarz]

*** Bug 195150 has been marked as a duplicate of this bug. ***

Comment 11

[Jon Henry]

*** Bug 225706 has been marked as a duplicate of this bug. ***

Comment 12

[Ian Tommins ('thelem')]

I would argue that the cache should actually be stored in a non-propogating
folder *that is not part of any profile*. For example, in the mozilla folder in
program files.

Advantages
 * Single cache for all users - no duplication of cached pages so cache can be
stored longer or made smaller
 * Does not propogate to the network

Disadvantages
 * Size & Location cannot be configured on a per-profile basis.

Can anyone think of any others?

Comment 13

[Darin Fisher]

> For example, in the mozilla folder in program files.

No good.  In some cases, Mozilla may be installed readonly.

Comment 14

[benc]

I'd like to keep the mail and cache considerations separate, esp. since it isn't
clear to me that a profile will function well if various mail storage folders
are pushed outside the profile directory. Bug 195150 is the only bug I can find
on the topic, so I've REOPENed it in MailNews.

*** This bug has been marked as a duplicate of 74085 ***