64978 - RegXPCOM crashes

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Simon Fraser [no longer active]]

I'm seeing a crash on Mac after the following steps. I believe this to be an old 
problem, but the steps have now become clear.

1. Delete the Mozilla registry and Netscape registry from the prefs folder (keep
   the Netscape registry to one side to preserver 4.x profiles if you must).

2. Delete the 'Mozilla' folder from the documents folder.

3. Delete the xpti.dat and xptitemp.dat files from the Components folder.

4. Delete the Components registry file

(Maybe all these steps are not necessary. 4. certainly is).

5. Reboot the machine.

6. Run mozilla. It will crash as it gets to the end of autoreg, as it starts
   to register JS components.

7. Marvel at how this bug has exactly the same symptols as bug 24312 (the
   RegXPCOM crash).

Comment 1

[Simon Fraser [no longer active]]

These steps also show it:
1. Trash the Components registry.
2. Restart the machine
3. Run mozilla.

This bug is serious. It's a non-obvious cause of crashes, and leaves you with a 
trashed component registry. To get around it, you have to trash the component 
registry, and run mozilla again before restarting to get things to work.

This is exactly the same as the RegXPCOM bug 24312. However, what's failing here 
is a GetService call, instead of a crash down in some OS file system call. So I 
think we have a better chance of finding the bug in this case. I'm thinking it 
might be registry-related.

Comment 2

[Simon Fraser [no longer active]]

Dumping console output to a file, I notice that the times that it crashes, it 
fails to get all the way through autoreg. It misses the last 2-3 C++ components, 
meaning that xpconnect never gets registered:

Good launch:
...
*** Registering nsWalletModule components (all right -- a generic module!)
*** Registering nsWalletViewerModule components (all right -- a generic module!)
*** Registering Browser Embedding Module components (all right -- a generic 
module!)
*** Registering xpconnect components (all right -- a generic module!)
**************************************************
nsNativeComponentLoader: SelfRegisterDll(Development:Mozilla 
tree:src:mozilla:dist:viewer_debug:Components:xpinstallDebug.shlb) Load FAILED 
with error: error 0 [note: this always happens]
**************************************************
nsNativeComponentLoader: autoregistering succeeded

Crash launch:
...*** Registering nsWalletModule components (all right -- a generic module!)
*** Registering nsWalletViewerModule components (all right -- a generic module!)
nsNativeComponentLoader: autoregistering succeeded
nNCL: registering deferred (0)

Another crash launch:
...
*** Registering nsWalletViewerModule components (all right -- a generic module!)
*** Registering Browser Embedding Module components (all right -- a generic 
module!)
nsNativeComponentLoader: autoregistering succeeded

Comment 3

[Simon Fraser [no longer active]]

Allocating more memory to the app does not help with this failure to finish 
component reg. Debugging now.

Comment 4

[Simon Fraser [no longer active]]

Debugging some more, here is what I have found.

This problem occurs because a call to ResolveAliasFile down in nsLocalFileMac 
fails with a -192 (resource not found) error. This call is made from component->
IsDirectory() in the autoreg loop, which means that we skip files for which such 
an error occured. Thus, we can end up not registering some components.

I've no idea why this error happens.

Comment 5

[Simon Fraser [no longer active]]

With a little more debugging, I verified that once we hit the -192 error when 
trying to tell if an item in the components folder is a directory, then we'll get 
it for every subsequent IsDirectory() test, so the registeration of all 
subsequent DLLs fails. XpConnect comes last, so we are pretty hosed if anything 
fails at all here.

However, before writing this off as an OS bug, note the following.
In nsNativeComponentLoader::RegisterComponentsInDir, I duplicated the loop that 
goes through all the files in the Components folder and tests IsDirectory on each 
one, and put it before the existing loop. We always get through this loop fine 
(and it doesn't affect where the failure occurs in the second loop). So it's 
something that we're doing during component registration (CFM loading etc) that 
tweaks the resource error. And recall that this is only after you've rebooted the 
machine. Note also that it doesn't always fail on the same file in the second 
loop, but it's always one of the last few files that fails (somewhere after 
ucvtw2Debug.shlb).

Since we stay away from WaitNextEvent for a long time during autoreg, I thought 
that might be a problem. So I added a call to SystemTask(); in teh second 
registration loop. That does indeed fix the problem (but is not something I can 
check in). It also makes RegXPCOM work.

Comment 6

[Simon Fraser [no longer active]]

More data on the crashing case: we always die after loading 60 component DLLs. 
The next Resource Manager call that we make after that will return with an error, 
so we'll die resolving the alias which comes after the 60 DLL that is loaded 
(ucvtwDebug.shlb for me).

If I replace the aliases in Components with actual files, the story is pretty 
much the same. When we try to open the 61st DLL, we get a -199 error (mapReadErr) 
from the FSpOpenResFile used to get at the 'cfrg' resource. DLL loading failure 
ensues.

Comment 7

[Simon Fraser [no longer active]]

cc beard.

Comment 8

[Ray Whitmer]

Reassigning to default owner of xpcom, who also has a mac so he can dup the
bug.

Comment 9

[Ray Whitmer]

Apparently default owner of XPCOM registry has not been properly assigned.

Reassigning to sfraser.  Please find an appropriate person to assign to,
who has a mac and is responsible for this.

Comment 10

[Simon Fraser [no longer active]]

*** Bug 65017 has been marked as a duplicate of this bug. ***

Comment 11

[rubydoo123]

reassigning to build team, Chris please assign to the appropraite person on your 
team

Comment 12

[Simon Fraser [no longer active]]

Interesting. When I hit the first error, 'file 0' in MacsBug says:

  #256 FCBs, #255 in use (including #74 fonts not listed), #1 free

after doing a 'log outfile', 'file 0' says:

  #384 FCBs, #256 in use (including #74 fonts not listed), #128 free

so it looks like we're not giving the OS a chance to grow it's FCB table.

Comment 13

[Steve Dagley]

You'd think the file system would grow the FCB table without requiring idle time 
to do it but apparently not.  So it sounds like the SystemTask() call in the reg 
loop may not be such a bad idea after all, despite being in XP code.  Patrick, 
what do you think about that as a fix?

Comment 14

[Daniel Veditz [:dveditz]]

"reassigning to build team"? why?

Comment 15

[Simon Fraser [no longer active]]

Beats me

Comment 16

[chris hofmann]

simon, any ideas who should own this or can own this?
how about scc, can he help with xpcom issues that might
be specific to mac?

looks like avoiding the corruption might be there if
I'm reading the bug right.

also looks like avoiding installation/registration
of X .dll's might be another workaround/solution????

how about recovery? anyone have ideas for installer
changes that might help a future installation attempt
to try and correct the problem?

Comment 17

[J.J. Enser]

I also believe this should not be in our (my) hands. Chris, I sent an email about 
this, suggesting to reassign to sdagley, on Simon's advice.

Comment 18

[Steve Dagley]

Simon and I discussed this bug and a fix tonight and he has a proposed fix I'm in 
agreement with so I'm going to reassign the bug to him rather than myself.  We 
just want comments from beard or scc on it.  And how common is this bug?  Is it 
common enough that we should fight to get it into 6.01?

Comment 19

[sean echevarria]

>how common is this bug?

This doesn't only affect autoreg.  It affects installation of xpcom plugins (on 
machines with lots of fonts).  We had one machine here that *always* had 
difficulties getting the Beatnik player installed under n6.  Turns out it used 
to be used by a graphic designer and has quite a few fonts installed.  Removed 
some fonts and now installation of the same plugin consistently works properly.

Comment 20

[Steve Dagley]

I've discussed this with beard and he agrees with the call to SystemTask() to 
allow the FCB list to grow but feels that rather than calling it in a loop we 
should just call it when we detect the error.

Comment 21

[Simon Fraser [no longer active]]

Attachment: nsNativeComponentLoader patch

Comment 22

[Simon Fraser [no longer active]]

I attached a proposed patch. This adds a call to SystemTask just before we 
autoregister a DLL. I think this is acceptable because:
1. It is only called per-DLL load, not per file in components.
2. It's only called when we do autoreg, which, after installation, should be 
never.

Calling SystemTask() just when we detect the error will be hard. The error can 
happen in a number of places (which depends on whether we have aliases or not), 
and we'd have to write extra code to re-try the DLL load on failure. It's also 
not clear whether we can recover after the error has taken place.

Beard, please sr=, sdagley please r=

Comment 23

[Simon Fraser [no longer active]]

[e-mail]: will this fix affect the XPCOM plugins issue? I'm not clear 
how their installation works, and whether they go through autoreg.

Comment 24

[Simon Fraser [no longer active]]

*** Bug 24312 has been marked as a duplicate of this bug. ***

Comment 25

[Steve Dagley]

I started looking at where the error could be caught and I agree it rapidly got 
ugly handling the potential code paths.  Since this patch only applies to autoreg 
startup and not normal startup I'd say we can live with any launch slowdown it 
might impose in that case.

r=sdagley

Comment 26

[Simon Fraser [no longer active]]

Comment from Mark Day of Apple in the newsgroup:

In article <[e-mail]>, Simon
Fraser <[e-mail]> wrote:

> Interesting. When I hit the first error, 'file 0' in MacsBug says:
> 
>   #256 FCBs, #255 in use (including #74 fonts not listed), #1 free
> 
> after doing a 'log outfile', 'file 0' says:
> 
>   #384 FCBs, #256 in use (including #74 fonts not listed), #128 free
> 
> so it looks like we're not giving the OS a chance to grow it's FCB table.
> I know this changed in 9.0, but I'm surprised that it's possible to
> open sufficient files in succession that the file system runs out
> of FCBs.

Growing the FCB table requires allocating memory.  Since the File
Manager uses the plain old Memory Manager, this has to happen at a time
when the File Manager knows it is safe to call the Memory Manager.  If
the Open call was made synchronously (which should be true in your
case), then the File Manager assumes it is safe to allocate memory.

> If I throw a SystemTask() call into our directory iteration loop,
> then the problem does not occur.

The File Manager has some code that runs during SystemTask time that
checks whether you're getting close to running out of free FCBs, and
grows the table if so.  If you can't find the underlying problem, this
is a good workaround.

So, the question becomes: why is SystemTask able to grow the FCB table,
but an ordinary Open call can't?  I'm betting it's not the File Manager
itself since I've got an app that sits in a tight loop opening files
synchronously.  I'm pretty sure it's not calling SystemTask,
GetNextEvent, etc.  It's able to get to the full 8169 open files
supported by Mac OS 9.

One possibility is that the System heap can't grow because there is a
locked block in the Process Manager's heap, right after the end of the
System heap.  I suspect this is not the case because calling SystemTask
fixes the problem (unless the locked block is getting unlocked or moved
as a result of the SystemTask call, before the File Manager tries to
grow the FCB table).

It's probably more likely to be some patch on the File Manager.  Does
it happen if you only use the Mac OS 9 Base or Mac OS 9 All extension
sets?  Are File Sharing or Multiple Users turned on?  If Multiple Users
is on, are you logged in as the owner or some other user?

-Mark

Comment 27

[Simon Fraser [no longer active]]

OK, I need to get this sucker in. Can I have an sr please? Beard? Conrad?

Comment 28

[Conrad Carlen (not reading bugmail)]

Seems like the way to go. This is more certain and less trouble then trying to
catch the error and deal with it in nsLocalFileMac. r=ccarlen

Comment 29

[Scott Collins]

sr=scc

Comment 30

[Simon Fraser [no longer active]]

Fix checked in.

Comment 31

[J.J. Enser]

flipped the switch in the mac automation settings to include a Component Registry 
with release builds again.
I'll close or reopen (oh no) this bug after tomorrow's verification build cycle.

Comment 32

[J.J. Enser]

still crashing when trying to launch Mozilla at the end of the build. reopening.

Comment 33

[Simon Fraser [no longer active]]

Stack please jj.

Comment 34

[Simon Fraser [no longer active]]

Over to jj for stack info. This is no longer the FCBs problem.

Comment 35

[J.J. Enser]

I just turned autoreg back on in the release build automation. Based on what 
happens in the next couple of days, I'll mark this fixed or attach a new stack 
trace if Macsbug lets me do so.

Comment 36

[lchiang]

sdagley notes in bug http://bugzilla.mozilla.org/show_bug.cgi?id=70543 that it
may be related to this bug.   jj - is this now fixed for you?
 

Comment 37

[J.J. Enser]

Unfortunately not. I had turned autoreg back on in the Mac automation and it 
seems like it worked fine for a while, but it crashed again today.
I attach a stack trace - but I don't think it's very useful since the app's heap 
was bad.

Simon, Pink, sorry but I'll have to turn autoreg off again until it's more 
stable.

Comment 38

[J.J. Enser]

Attachment: Mozilla stack trace while doing autoreg (03/13)

Comment 39

[J.J. Enser]

Autoreg turned back on for both 0.9.1 branch and trunk on 06/04. No crash 
observed; Component Registry included in installer.

Marking this one fixed (one more time) and adding dependency with bug 46000 which 
also tracks this issue for future reference.