46000 - Component Registry not pre-installed on MacOS, VERY slow first startup and runs out of memory quickly

Status: VERIFIED FIXED

(Core :: XPCOM, defect, P3)

Description

[Mike Pinkerton (not reading bugmail)]

- delete your component registry to force a rebuild
- launch mozilla (it will re-register everything)
- go to a couple of webpages, like my.netscape.com and foxnews.com

You'll hit a sysError 25 (out of memory) after only a couple of webpages. This 
only happens after the components are re-registered, probably because we load all 
the libs but never unload them. Subsequent runs are fine, memory wise.

This is pretty bad, as we crash very quickly on first run of the app. A great way 
to make a first impression!

Comment 1

[Simon Fraser [no longer active]]

It seems that we never unload all the DLLs we load when autoregistering. Most of 
the heap is thus being used by data (TOC etc) associated with loaded libraries on 
Mac, which never go away. The rest of the heap is sucked up with the double-
buffering GWorld back buffer.

Comment 2

[Ray Whitmer]

Perhaps this is related to slowness issues of autoregistration on Mac.

A stop-gap solution might be to require autoregistration to always occur 
separately on the Mac.  We certainly need to solve the issue of unloadability, 
but it probably will not be a quick one.

Comment 3

[Jay Patel [:jay]]

Putting on [nsbeta2-] radar.  Not critical to beta2.  Adding "relnote" keyword 
for PR2 release. Adding nsbeta3 keyword.

Comment 4

[Mike Pinkerton (not reading bugmail)]

relnote2 keyword

Comment 5

[Ray Whitmer]

I disagree that the ability to unload DLLs is a nsbeta3 function.

Comment 6

[Mike Pinkerton (not reading bugmail)]

whoa, so you're telling me that it's ok that we're running out of memory on 
registration? ray?

Comment 7

[Ray Whitmer]

There are lots of things that XPCOM does that are not OK.  But solving the whole
problem of being able to unload DLLs in nsbeta3 does not seem likely to happen.

There are ways you can work around the problem.  I could suggest a few if you 
like.  None are perfect, but it is possible to restart one or multiple times if 
registration has loaded too many things into memory.

Comment 8

[leger]

Putting on [nsbeta3+] radar.

Comment 9

[Ray Whitmer]

A very simple work-around is to pre-build the component.reg file.

Unloading DLLs will not happen in beta3.

I am marking it "Future" and anyone who would like to come discuss this further 
may do so.  I am leaving it on the nsbeta3 for a few days so that those who 
marked it beta3 can decide what they want to do instead of unloading DLLs.  
There are many possible approaches.  After that, with no more input, I will 
remove nsbeta3.

Comment 10

[Mike Pinkerton (not reading bugmail)]

What would some of those approaches be, pray tell, since the obvious one is being 
summarily dismissed?

Comment 11

[Mike Pinkerton (not reading bugmail)]

cc'ing other smart people

Comment 12

[Scott MacGregor]

Sorry to jump in here. But this bug has already been given a beta3+ by PDT.

Engineers are not allowed to future beta3 approved bugs. Ray if you really don't
think this should be fixed in beta3 then you should make your case to PDT (they
are the ones that said we need to fix this problem for the Mac for beta3). But
we can't future beta3 approved bugs at our level. Although just by scanning the
bug this sounds like a pretty severe problem IMHO...

anyway, just thought I'd point out that this bug shouldn't be marked future
until PDT decides they gave the beta3+ nomination in error (if they decide that)

Comment 13

[Chris Waterson]

Hold the phone.

We are *not* going to implement DLL unloading by nsbeta3. There are far, far too
many issues with DLL loading for us to even consider it. None of the modules
we've written (and I mean *none*) actually reference count the objects they've
created. So they don't even know if they can be safely unloaded or not. If we
start unloading modules, we will literally introduce *hundreds* of "random"
crashes.

But let's take a step back here. Look at "step one" that is required to
reproduce the bug:

  - delete your component registry to force a rebuild

We will ship the bits with a precompiled component registry. Why on earth would
a user delete their component registry? If they *do* delete it, then next time
they run the browser, it's going to gobble up memory. And maybe crash. Too bad.
It'll be okay after that. (Compare this to deleting any other "system" file
that's associated with an application: go ahead, I dare you to delete random
files out of /etc, /usr/bin,  your Windows system directory, or your Mac system
folder. You are asking for trouble.)

If the problem is that we're not shipping/installing a precompiled component
registry, then let's change the bug to mean that.

I'm tempted to mark this bug INVALID as reported, but I won't. Instead I'll
clear the nsbeta3+ that was assigned to it, and ask for this to be reconsidered.

Comment 14

[Mike Pinkerton (not reading bugmail)]

according to dp the last time I asked him about this, we are _not_ going to ship 
a preinstalled component registry. While this was bantered around at porkjockeys, 
and everyone there thought it was a done deal, no one outside those meetings ever 
took this idea seriously.

Since we're not doing that, this bug is perfectly valid. If we are doing that,  
where is the bug? Who is tasked with the work? When is it scheduled to land?

cc'ing dveditz. he might know.

Comment 15

[Mike Pinkerton (not reading bugmail)]

Here's the real problem, there is supposed to be a pre-installed component 
registry (according to waterson), but for some reason it's either ignored or the 
date checking is buggy on macOS.

This is bad bad bad, however you slice it. What's up here? Who would know? Who is 
supposed to own the pre-install of the component registry?

Comment 16

[Mike Pinkerton (not reading bugmail)]

updating summary again

Comment 17

[Scott MacGregor]

Given this new information it sounds like we want to recommend an nsbeta3+ to
PDT for making sure the MAC comes with a pre-installed components.reg. This
works on windows and linux but as pinkerton says is aparently not working on the
Mac.

I think the summary of this bug now reflects that. Thanks pink.

Comment 18

[Chris Waterson]

I'm not convinced this is working on Linux or Windows. I just untar'd the
commercial Linux release bits from this AM, and there was no component.reg file
in there. I would love to be proven wrong.

Comment 19

[Chris Waterson]

cc'ing warren & dp.

Comment 20

[Warren Harris]

Marking nsbeta3+

Comment 21

[leger]

adding [nsbeta2-]...this is work for nsbeta3

Comment 22

[Ray Whitmer]

I never suggested I would change the whiteboard status, but only the keywords 
and target milestone, which I understood should reflect my own analysis as the 
current owner of the bug.  I feel that it is my responsibility to do so.  The 
whole point is that the conflicting values should raise a flag with those who 
maintain the whiteboard status.

I will change the target milestone from "Future" when the bug becomes clearly 
about something that should be deliverable in a more-concrete time frame (or 
perhaps a new owner of the bug will).

Comment 23

[Simon Fraser [no longer active]]

This is why people will delete their component registry on Mac:
1. Rumors will start that deleting this file, and getting it rebuilt, will solve
   instability problems. This is inevitable.
2. Users see this stupid file with a generic icon in the Netscape folder, and 
   wonder why it is there. They will delete it, "but it keeps coming back. Is it
   a virus?"

My point is that this file is in the users face, it has no icon, and generally 
looks 'lost'. Naive users will be happy to delete it regularly.

Comment 24

[Warren Harris]

The file should probably live in the cache directory (or a writable portion of 
"Essential Files"), along with all of hyatt's manifest.rdf files. They all 
should get the right icons, creator codes, etc.

Bug 42184 talks about the need to move these files for unix write permission 
issues.

Reassigning to myself.

Comment 25

[Simon Fraser [no longer active]]

un-future this, since warren took it.

Comment 26

[Mike Pinkerton (not reading bugmail)]

we're less than a week away from b3, what's the status on this?

Comment 27

[Warren Harris]

It's all up to the build team to make sure it goes out with the bits, right?

Comment 28

[Mike Pinkerton (not reading bugmail)]

then maybe we should assign this bug to someone on the build team, perhaps?
--> leaf

Comment 29

[Daniel (Leaf) Nunes]

Once upon a time, we ran regxpcom to generate this registry on mac; when it
started crashing the build machine and preventing deliveries of any kind
whatsoever from happening, we turned it off.

We still run this as part of the unix and windows automation, because it hasn't
crashed on these platforms.

Simon Fraser spent a lot of time trying to figure out why this was happening,
and it seemed to stop for a while. Unfortunately, it started happening again.

Any mac heads want to help debug this?

Comment 30

[Warren Harris]

We don't need to debug regxpcom on the mac to fix this bug for beta3, nor do we 
have time. We simply need to launch the browser, and it will automatically build 
the components.reg file to ship. We just have to make sure we do that before the 
bits go out the door.

Comment 31

[Daniel (Leaf) Nunes]

fine, whatever. marking resolved remind, leaving in this milestone.

Comment 32

[Simon Fraser [no longer active]]

Reopening. Having Mac do autoreg on first run after an installation still sucks, 
and has the side-effect of causing serious low memory problems.

Comment 33

[Simon Fraser [no longer active]]

Giving to JJ for some mac release machine loving.

Comment 34

[Mike Pinkerton (not reading bugmail)]

adding rtm keyword. this is very important that we fix!

Comment 35

[Simon Fraser [no longer active]]

Reasons for fixing this are described in bug 57835.

Comment 36

[Jon Granrose]

build release would be happy to do this as soon as you fix regxpcom.

rtm- til regxpcom is fixed.

Comment 37

[J.J. Enser]

Until RegXPCOM gets fixed (not by me), Release could use the following workaround 
for RTM.

since the automation is what prevents us from delivering a prebuilt component.reg 
on mac, I could to run Netscape by hand and add the newly generated component 
registry to the installer... I am willing to do that for RTM candidates -only-.
(and maybe set the file type & creator so that the registry has a Netscape icon 
and doesn't seem 'lost')

For those of you who have missed the first few episodes of this long story, take 
a look at related bugs 13129, 31858, and mainly 24312.

Comment 38

[Warren Harris]

jj: I recall discussing this with Leaf, and agreeing to run mozilla by hand for 
RTM candidate builds to generate the component.reg file. I think it would be 
great if you could do this!

Comment 39

[J.J. Enser]

working on a "half-automated" solution where the component registry gets 
generated after the build is complete, then added to the browser.xpi installer 
module.

Comment 40

[Simon Fraser [no longer active]]

FWIW, I've testing RegXPCOM, and it still crashes in the same way.

Comment 41

[J.J. Enser]

Since 11/03, the packaging process launches mozilla/netscape after the build and 
delivers the component registry generated file.
This has proved to be a viable solution... as long as we don't have DOA builds ;-
)
This way, Release would discover and report crashes on startup before the bits 
even get to the somketesters!

Comment 42

[Jon Granrose]

verified fixed.

Comment 43

[J.J. Enser]

Reopening. Crashes have re-occured when launching Netscape at the end of the 
build, so this part of the automation has been turned off again since around 
March 20 :-(
I have 2 stack traces captured with Netscape 6, build 2001-05-30-11, coming up.

changing target milestone from M18 to ... 0.9.1?

Comment 44

[J.J. Enser]

Attachment: crash on startup stack trace with build 20001053011 (2)

Comment 45

[Simon Fraser [no longer active]]

That log shows 2 crashes; which should we be looking at?

Comment 46

[J.J. Enser]

I attached both crashes (hence the (2) in the attachment name) because they both 
occured during test autoreg tests I did yesterday, but I now realize that the 
first one might be unrelated:

crash 1 occured after I did the following:
1. started up NS6
2. observed all components registration
3. waited until idle on www.netscape.com
4. pressed Cmd-Q
5. Boom!
(hence the domenu/menuSelected calls in the first stack trace)

This might be a totally different bug that might not affect the release 
automation since NS gets launches & quit using AppleEvents, not a Cmd key :-)

crash 2 occured a few minutes later as I was trying again.
1. Deleted previous Components Registry file
2. Launched NS 6
3. Observed Components registration in the splash screen
4. Boom!

Comment 47

[Simon Fraser [no longer active]]

jj: please test and see if you can reproduce these crashes.

Comment 48

[J.J. Enser]

I ran a basic script to see if this worked in an "automated environment":
	set myNetscape to "disk:mozilla tree:ns:dist:viewer:Netscape 6"
	tell application myNetscape to activate
	delay 5
	tell application myNetscape to quit

This ran just fine on both 06-01-11-trunk and 06-01-14-0.9.1 builds (tested on 
the branch right after the normal packaging automation ended), so I will flip the 
switch back on and pray that this is the last one I need to change this.

Comment 49

[J.J. Enser]

patch to BuildCentral Default Settings (applied on the branch verif. build mac 
only)

< #mz_make_comp_reg	false
< #ns_make_comp_reg	false
---
> #mz_make_comp_reg	true
> #ns_make_comp_reg	true

Comment 50

[J.J. Enser]

Yeah! branch builds from the weekend and today's didn't crash. Looking at the 
build machines directories, the component registry was successfully created for 
all these builds.
Grace, please verify it's also installed by the installer.

Applying the changes to the trunk automation now.

Comment 51

[Grace Bush]

verified with RTM  build 2001072621 and trunk build 2001081704