157627 - 'registry.dat' grows by 45KB! on _every_ start of win32/commercial/branch

Status: VERIFIED FIXED

(Core Graveyard :: Profile: BackEnd, defect, P1)

Description

[John Morrison]

Steps to reproduce:

1) Install current branch build
2) Shut down mozilla. 
3) Delete (or rename) the '...\Application Data\Mozilla' directory
4) Start mozilla. Note size of '...\Application Data\Mozilla\registry.dat'
5) Repeat step 4.
6) Repeat step 4.
 ...

Actual results:   'registry.dat' grows by 45KB on each start of browser

Expected results: 'registry.dat' should not need to grow in size, if 
                  nothing in environment has changed. (And if it did 
                  need to grow, obviously 45KB is way too much

Reproducible 100% in current 07/15 branch build on win2k. (Haven't checked 
other platforms, or trunk, yet).

Not entirely sure who owns this, so spreading the love among dveditz, dougt,
ccarlen. Reassign as appropriate.

Comment 1

[Nicholas Allen]

Are you seeing bug 109739?

Comment 2

[Matthias Versen [:Matti]]

*** This bug has been marked as a duplicate of 109739 ***

Comment 3

[John Morrison]

This is essentially the same problem as bug 109739, but there is 
a particular aspect to this bug report that makes it something that
we can't just shrug our (Netscape) shoulders and say "well, end users
won't see this".

The 'registry.dat' file grows...:
                       Mozilla                Netscape
-----------------------------------------------------------------
1.1 07/15 trunk     When a different       When a different   
		    binary is started  	   binary is started  
-----------------------------------------------------------------
1.0 07/15 branch    When a different       EVERY time the same        
		    binary is started      build is started


This should really move to a bugscape bug for the commercial build. 
Hopefully we can find the cause of this happening only in the commercial
build, and get that build to only add to registry.dat when a different
build is started. 

[I'll note that the large file size doesn't appear to add to startup time, at 
least on win2k with a SCSI drive. (I'm less optimistic about Mac performance
not being affected, although this bug still needs to be confirmed on Mac 
and Linux for the commercial branch).]

Comment 4

[John Morrison]

I opened http://bugscape.mcom.com/show_bug.cgi?id=17790 for the commercial only 
bug.

Comment 5

[John Morrison]

...and put the summary back. Duh.

Comment 6

[Asa Dotzler [:asa]]

jrgm, I've heard reports at mozillazine that this is happening for people using
mozilla builds too.

Comment 7

[John Morrison]

Well, yeah. Bug 109739 covers it for the general case (i.e., someone who 
downloads new mozilla builds on a regular basis will have registry.dat 
grow huge over time). But that's (comparatively) not as bad as growing
every time the browser is started.

Or are you saying that someone can reproduce the above in a mozilla build.
If so, we can reopen this bug, but as far as as I can see, the every time
thing is only in commercial branch.

Comment 8

[Daniel Veditz [:dveditz]]

Mozilla builds happen on the branch, too -- why would it be a commercial-only
problem? Are you blaming activation? 109739 was plugins, and that's common code.

Comment 9

[Daniel Veditz [:dveditz]]

er, never mind. And I had just read your chart, too. D'oh!

Comment 10

[Asa Dotzler [:asa]]

I think we should reopen this bug.
http://www.mozillazine.org/talkback/read.php?f=2&i=2084&t=2084 reports it
growing with every launch. 

Comment 11

[K'Trina Medina]

Reopening per Asa, 'registry.dat' growing with every launch

Comment 12

[Pascal Chevrel:pascalc (PTO until April 26)]

FYI, there is a long thread about this problem in netscape.netscape7.windows, a
NS7 user is reporting that he discovered he had a 68MB registry.dat file after a
few months using NS7PR1 !

Comment 13

[John Morrison]

To quote from the comment referenced by asa: "I have recently found out that
my registry.dat file get 30k larger each time I switch from running
mozilla.exe from the installed version to the debug version and vise-versa."

Okay, that is column one from my table above, and that is a duplicate of bug
109739, no? I'll note again that I opened bugscape bug 17790 to cover the
Netscape-specific aspect of this bug, which is that in the commercial build,
starting the _same_ build will write the registry.dat _every_ time (i.e., 
not just when switching builds).

But that 68MB number makes me feel ill :-(. It suggests that we must implement
Pack (or equivalent), but I'd hate to have to call that on every startup.

Comment 14

[Peter Lubczynski]

When running different versions of the browser, there could be a version
mis-match in the registry cache and cause the cache to invalidate. Because it
has no Pack() method, it grows out of control. :(

Comment 15

[Peter Lubczynski]

--->Andrei's got this

Comment 16

[av (gone)]

I think we can dup this to bug 109739, given that bug is being really taken care
of, which it seems to be as it got ADT1.

Comment 17

[av (gone)]

Can anybody please try today's Netscape build? I am using Mozilla/5.0 (Windows;
U; Windows NT 5.1; en-US; rv:1.1b) Gecko/20020719 Netscape/7.0b1+ and cannot
repro the problem any more.

I do have npcdt.dll in the Components folder, although it is not listed in the
about:plugins page for some reason.

Comment 18

[John Morrison]

Yes, I also think that this bug should be duped to bug 109739. (The Netscape
specific aspects are being dealt with in bugscape bug 17790).

av: I believe that user agent string is for a trunk commercial build. The 
'every time you launch' aspect of this bug is specific to the _branch_ 
commercial build. Refer to the table in comment #3.

I can reproduce the comm. branch bug in today's builds on both a win98 and 
a win2k machine.

Further comments in http://bugscape.mcom.com/show_bug.cgi?id=17790

Comment 19

[Peter Lubczynski]

Attachment: patch v.1

On a side note, setting NSPR_LOG_MODULES=all:8 and running with -console, I'm
surprised at how much a Netscape commercial build outputs. Does this add bloat
or reduce performance when not used?

I think this problem is caused by nsIModule-type "new" XPCOM plugins. npcdt.dll
is one of these kinds of plugins (can be verified with depends.exe) and only
ships with the Netscape commercial builds. Copying that DLL to my components
folder and somehow getting it to "register", I was able to reproduce the
problem in a Mozilla branch debug build. To verify a true XPCOM plugin is
registered, set a breakpoint in the middle of
|nsPluginHostImpl::LoadXPCOMPlugins|. Linux users may also be seeing this
regularly as I think JRE 1.4 is one of these 'components'.

First off let me say that I think there are *several* problems here. This patch
attempts of the fix the problem of the growing registry on each start on
Netscape commercial builds because of XPCOM style plugin/components (may help
Linux w/JRE 1.4 too). The (ApplicationRegistry) registry grows because we
detect plugins have changed each time we scan:

    // count plugins remained in cache, if there are some, that means some
plugins were removed;
    // while counting, we should ignore unwanted plugins which are also present
in cache
    PRUint32 cachecount = 0;
    for (nsPluginTag * cachetag = mCachedPlugins; cachetag; cachetag =
cachetag->mNext) {
      if (!(cachetag->mFlags & NS_PLUGIN_FLAG_UNWANTED))
	cachecount++;
    }
    // if there is something left in cache, some plugins got removed from the
directory
    // and therefor their info did not get removed from the cache info list
during directory scan;
    // flag this fact
    if (cachecount > 0) 
      *aPluginsChanged = PR_TRUE;
  }


When they change, we nuke the subtree and re-write the info. There is no Pack()
method so it grows endless. We think plugins have changed because we are not
marking or removing the cached plugin when it's a 'component' and loaded from
the ApplicationComponentRegistry. The flag is not set in the loop above and
cachecount is equal to my plugin/components. There may be several ways to fix
this specific problem and this patch takes one stab at it by simply removing
the cached tag for the XPCOM plugin. This list is cleaned up shortly thereafter
anyway post scanning. We probably should not cache these plugins twice anyway
in the registry but I thought just removing it may be least risky. Thoughts?
Reviews?

Comment 20

[John Morrison]

(Re: NSPR logging: A PR_LOG call, in an optimized build with FORCE_PR_LOG set,
but NSPR_LOG_MODULES not defined, adds three instructions (including a branch)
to the line of execution. So, that's pretty minimal, but I'll check if there
is a perceivable impact on startup or pageload (where it is called a lot) or
on bloat (although unless FORCE_PR_LOG is widespread, I doubt that this has
a major impact on code size). On the other hand, I think that having logging 
available has in the past helped to diagnose end user problems "in the field").

Comment 21

[av (gone)]

Peter, although I still cannot see the problem with npcdt plugin, I see now that
this can happen. But instead of not caching XPCOM plugins why not just treat
them as unwanted in the code you cited above? Something along the following lines:

  if (!*aPluginsChanged) {
    // count plugins remained in cache, if there are some, that means...
    // while counting, we should ignore unwanted plugins which are...
    PRUint32 cachecount = 0;
    for (nsPluginTag * cachetag = mCachedPlugins; cachetag; cachetag =
cachetag->mNext) {
      if ((cachetag->mFlags & NS_PLUGIN_FLAG_UNWANTED) ||
          !(cachetag->mFlags & NS_PLUGIN_FLAG_OLDSCHOOL))
        continue;

      cachecount++;
    }

I think the reason I want XPCOM plugins still be in our cache is that if such a
plugin lives in the Plugins folder rather than in the Components folder we are
going to be screwed again because every time we will think it has just been
added. But given this patch is supposed to go to the branch only, I am fine with
yours.

Comment 22

[Peter Lubczynski]

Attachment: patch v.2

Andrei,

You may need to delete components.reg and restart several times to get the CDT
plugin to get picked up when |LoadXPCOMPlugins| is called. Use the branch. Do
not have another copy of the CDT plugin/component in any plugins folder. You
may have to use regxpcom as last resort.

NS_PLUGIN_FLAG_OLDSCHOOL can not be used here because it is only valid after
mEntryPoint has been set which is typically in |GetPluginFactory|.

That's fine if we shouldn't remove the cache tag. We can just as well mark it
as "unwanted" to avoid cachecount++. This patch does that and it also stops the
appreg from growing on each startup. It still may grow at other times when
plugins have changed.

However, you do bring up a valid point if the same copy of the plugin is in
both a plugins folder and a components folder. Currently, |LoadXPCOMPlugin|
does not set |tag->mLastModifiedTime|, it is Zero. It may also cause us to
always think new plugins have been installed. This may be yet another bug.

Comment 23

[av (gone)]

Comment on attachment 92191 [details] [diff] [review]
patch v.2

I feel better, thanks, Peter, r=av

Comment 24

[av (gone)]

I was able to reproduce and test it eventually. The patch fixes things neatly.
The problem with the registry growth has even more impact than it was originally
thought (thanks jrgm for pointing this out): it grows on every about:plugins too
which essentially means on every javascript:navigator.plugins.refresh. The patch
fixes this too.

I think we must check it in as it solves the original problem and is simple and
safe. It will not interfere with more profound solution which dougt and serge
are working on in bug 109937.

Comment 25

[av (gone)]

Sorry, 'bug 109937' should read 'bug 109739'.

Comment 26

[Daniel Veditz [:dveditz]]

Comment on attachment 92191 [details] [diff] [review]
patch v.2

sr=dveditz

Comment 27

[Paul Wyskoczka]

Adding adt1.0.1+ on behalf of the adt.  Please check into the Mozilla branch
after getting approval from drivers.

Comment 28

[Asa Dotzler [:asa]]

Comment on attachment 92191 [details] [diff] [review]
patch v.2

a=asa (on behalf of drivers) for checkin to 1.1

Comment 29

[chris hofmann]

a=chofmann for the branch. add the fixed1.0.1 keyword after checking in on the
branch.

Comment 30

[av (gone)]

Formally marking fixed as the patch is on the trunk now.

Comment 31

[Jaime Rodriguez, Jr.]

marking as "mozilla1.0.1+" per Comment #29 From chris hofmann.

Comment 32

[av (gone)]

On the branch now. Marking fixed1.0.1

Comment 33

[shrirang khanzode]

fix looks great, the registry has stopped growing. verified on today's build 
0723. made sure that I had the CDT plugin present.

Comment 34

[shrirang khanzode]

by ' the registry has stopped growing' , i meant, rigistry.dat does not increase
if I repeatedly launch the same branch build (0723).

Comment 35

[K'Trina Medina]

Thanks Shrirang! 
Verified

Comment 36

[Mel Reyes]

I'm not really sure if this has been "Fixed" maybe it's due to other changes,
but I was one of the original multi-meg registry.dat file victims and just
recently I checked and yes it was wan't growing, but it also wasn't optimized
either.

I guess with parsing out the pluginreg.dat file the registry.dat and the
pluginreg.dat aren't cleaned out or something.  Here's my example after already
having cleaned out my registry.dat file back in August or September and using
those build I had no problem with the file "growing", but here's what I found
yesterday

Steps to reproduce (at least on Windows Systems):
- have mozilla open or in Quick Launch mode
- go to the blah-blah\Application Data\Mozilla and rename the pluginreg.dat and
the registry.dat files
- close mozilla or exit out of quick launch
- new files are created and here is what I've seen in size:
       old registry.dat  173k         New  1.04kb
       old pluginreg.dat 7.67kb       New  5.78kb

Somewhere there is some cleanup that should be done, not really sure where
though, thanks.

Comment 37

[K'Trina Medina]

Reopened per Comment #36 

Comment 38

[Daniel Veditz [:dveditz]]

The growth problem has been fixed. If you want to open a new bug on cleaning out
the old cruft go ahead, but I doubt it'll get much action (in fact there may be
such a bug, search for "registry packing" or similar). Effort by the profile
folks is focusing on other problems such as portability, and may replace
registry.dat entirely in the future.

Comment 39

[Mel Reyes]

Sorry about getting testy below, but here goes:

Is this really fair to just dismiss this, since there could very well be users
that have huge registry.dat files out there?  This patch should really be
marked, 'incomplete' or 'insufficient', cleaning this file should have been part
of the solution, not spunoff for a future conceptualized rewrite.

Oh well, I'll make sure to check my .dat files renegade files.

Comment 40

[Mel Reyes]

Attachment: Registry.dat file

I just checked one of my other computers (one that I don't even use that much)
and found a 1.1 meg registry.dat file, fun for the whole family!!!

Comment 41

[Mel Reyes]

Also, can someone guage the load and startup impact of such a large file?  Just
wondering.

Comment 42

[John Morrison]

re: startup impact, see comment #3 (and reconfirmed today). Despite the large
size, this does not impact startup on win2k (and likely linux, although I'm not
sure about mac [particularly os/9]). The app does not try to read this entire 
file; it just opens, maps, checks a header, seeks, reads, closes and unmaps 
(from what I can tell).

I'd have preferred (in a perfect world) that this had been cleaned up. But 
that would be bug 10210 as dveditz points out.

Comment 43

[Mel Reyes]

I had posted a write up on my site about clearing this and just updated it
include Netscape 7.01 based on this email that I got:

"Re: Mozilla registry.dat files.  A week ago I installed Netscape 7.01.  It is
based on Mozilla build 1.0.2, the latest stable release. I followed your
instructions for renaming the registry.dat file, then restarting Netscape. 
Registry.dat file went from 2407KB to 25KB on restart, with no noticable change
in settings or performance. Hopefully there will be a fix because I seriously
like the browser. Good work on your finding!"

Maybe Netscape should apply the patch soon or release 7.1 based on 1.2a, hope
this helps.

http://www.mrtech.com/news/messages/2466.html

Comment 44

[John Morrison]

um, and what if the users to whom you are recommending this procedure have 
more than one profile and/or have a profile that doesn't have the default 
profile name and/or that profile is not located in the default profile 
directory. How do they recover their existing profile(s)?

[Actually, I'm going to defer to somebody from the installer group for an 
authoritative explanation about the above. But I thought you might like to 
know that deleting that file can have unintended consequences. You should 
warn your readers that they risk dataloss if they do this, and, at minimum,
that they should back up their 'Application Data\Mozilla' before trying any
of this.]

Comment 45

[Grace Bush]

The registry.dat file stores the locations of all profiles used by the
application as well as plugin information used by the application.  
People who regularly install and test are the ones most l likely to see this
problem and the ones most able to be aware of the pitfalls of deleting this file.
Mel's instructions seem to alert people to these issues.

It would not be recommended for other users

Comment 46

[Grace Bush]

verified-