516113 - Startup crash [@ PL_DHashTableOperate | free | nsEventListenerManager::AddEventListenerByType(nsIDOMEventListener*, nsAString_internal const&, int, nsIDOMEventGroup*)]

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Michael Kraft [:morac]]

I installed a program which opened a locally stored .htm file.  This launched Firefox which promptly crashed.  The .htm file opened fine when I restarted from the Crash Reporter so it's not reproducible.  This is the first time I've seen it.  The only changes to my system made lately were upgrading to Firefox 3.5.3 and also upgrading from Norton Internet Security 2009 to 2010.

http://crash-stats.mozilla.com/report/index/4e0cefed-bd97-4e5f-9c6d-95f212090911

Comment 1

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Olli, can you have a look at this topcrash? It appears to be happening on startup, and is possibly simply a nullpointer deref.

Comment 2

[Olli Pettay [:smaug][[e-mail]]]

The stack traces like http://crash-stats.mozilla.com/report/index/3a829991-c85d-42cf-8d3b-2eb892091019 have something strange in #10/#11.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

Attachment: guess fix

Though, I'm not sure how this could be happening.
Does something in #11 shutdown gklayout?

Comment 4

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comment on attachment 407239 [details] [diff] [review]
guess fix

jst can hook you up with minidumps which should help you tell is sEventTable is null.

Would be good to verify that that is in fact the case.

r=me on this in any event.

Comment 5

[Olli Pettay [:smaug][[e-mail]]]

The stack traces are a bit strange, so it is also possible that
the hashtable isn't initialized. Though, in that case sEventTable
really must be null too, if I read nsContentUtils correctly.

Comment 6

[Olli Pettay [:smaug][[e-mail]]]

(In reply to comment #4)
> (From update of attachment 407239 [details] [diff] [review])
> jst can hook you up with minidumps which should help you tell is sEventTable is
> null.
Minidumps do show that the table is null when PL_DHashTableOperate is called.

But I still don't understand why. Is the gklayout not yet initialized properly?

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

Ah, hmm, the stack something strange.

...
xul!nsAppShellService::CreateTopLevelWindow+0x46
xul!nsAppStartup::CreateChromeWindow2+0x75
xul!nsWindowWatcher::OpenWindowJSInternal+0x404
xul!nsWindowWatcher::OpenWindow+0x150
xul!NS_InvokeByIndex_P+0x27
xul!XPCWrappedNative::CallMethod+0x4c0
mozcrt19!arena_malloc_small+0x125
js3250!js_DefineNativeProperty+0x19c

Why on earth do we have the js part there, and what is malloc doing?

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

Looks like  Bug 524256 is needed.

Comment 9

[Damon Sicore (:damons)]

Any progress here?

Comment 10

[Olli Pettay [:smaug][[e-mail]]]

This is blocked by bug 524256. So when some new release with that fixed
gives reasonable minidumps, I can re-try.
Currently I can't see this in 3.5.6pre or 3.6.b2pre crash list.

Jst, could you check if there are 3.5.6pre or 3.6b2pre minidumps for this?

Comment 11

[Jesse Ruderman]

Bug 524256 is fixed now (for 1.9.2 and trunk), but I'm not sure that helps.

Comment 12

[Johnny Stenback (:jst)]

For the record, I found one (and only one) minidump for smaug for the requested version(s).

Comment 13

[Olli Pettay [:smaug][[e-mail]]]

The stack doesn't still go all the way up.

...
nsWindowWatcher::OpenWindow
NS_InvokeByIndex_P
XPCWrappedNative::CallMethod
js_DefineNativeProperty


Ted, do we have stack size limit or something like that for crash dumps?
I would expect main() to be somewhere in the stack, but that is not what I
get from minidumps in this case.

What is really needed here is to understand who calls the method OpenWindow
Somehow OpenWindow is called either before gklayout is initialized, or after
it has been shutdown.

Or, is it possible that js_DefineNativeProperty does something wrong :/

Comment 14

[(not currently active) Ted Mielczarek]

We just call MinidumpWriteDump from Microsoft's dbghelp.dll on Windows. I don't know what that method does internally, but I would expect that it simply dumps the raw contents of stack memory. (this is what Breakpad's implementation on Mac and Linux does)

Is the dump you're using from a 3.5.x release? bug 524256 only landed on trunk and 1.9.2 so far (I just requested approval for 1.9.1.7). You can probably manually reconstruct the stack, there's a MSDN article on how to do that with WinDBG:
http://msdn.microsoft.com/en-us/library/cc267826.aspx

Comment 15

[Olli Pettay [:smaug][[e-mail]]]

(In reply to comment #14)
> Is the dump you're using from a 3.5.x release? bug 524256 only landed on trunk
> and 1.9.2 so far (I just requested approval for 1.9.1.7).
Oh, I thought it landed to 1.9.1 too, since there is the comment
"Pushed changeset e3756eea61f0 to releases/mozilla-1.9.1"

Comment 16

[(not currently active) Ted Mielczarek]

That is confusing, I'll ask Neil what happened there.

Comment 17

[Jeff Muizelaar [:jrmuizel]]

So here's a manually unwound stack part of the way. I'll see if I can get the stuff that comes after XPC_WN_CallMethod

Thread 0 (crashed)
FUNC 56030 2f3 4 PL_DHashTableOperate
 0  xul.dll + 0x56039
    eip = 0x10056039   esp = 0x0012e64c   ebp = 0x0012e738   ebx = 0x00000000
    esi = 0x0093ba00   edi = 0x00000000   eax = 0x01ba119c   ecx = 0x00000000
    edx = 0x01ba119c   efl = 0x00010206
    Found by: given as instruction pointer in context

FUNC 158016 e1 0 nsWindowRoot::nsWindowRoot(nsIDOMWindow *)
1  xul.dll + 0x1580a4
    eip = 0x101580a5   esp = 0x0012e740   ebp = 0x0012e76c
    Found by: previous frame's frame pointer

FUNC 1b8569 3f 8 NS_NewWindowRoot(nsIDOMWindow *,nsPIDOMEventTarget * *)
 2  xul.dll + 0x1b8580
    eip = 0x101b8581   esp = 0x0012e774   ebp = 0x0012e79c
    Found by: previous frame's frame pointer

FUNC 14be0 166 4 nsDocShell::EnsureScriptEnvironment()
 3  xul.dll + 0x14cef
    eip = 0x10014cf0   esp = 0x0012e7a4   ebp = 0x0012e7d4
    Found by: previous frame's frame pointer

FUNC 16e60 4af c nsDocShell::GetInterface(nsID const &,void * *)
 4  xul.dll + 0x16ff8
    eip = 0x10016ff9   esp = 0x0012e7dc   ebp = 0x0012e830
    Found by: previous frame's frame pointer

FUNC 51a00 30 4 nsCOMPtr_base::assign_from_helper(nsCOMPtr_helper const &,nsID const &)
 5  xul.dll + 0x51a16
    eip = 0x10051a17   esp = 0x0012e838   ebp = 0x0012e874
    Found by: previous frame's frame pointer

FUNC 1bf350 68 20 nsAppShellService::CreateTopLevelWindow(nsIXULWindow *,nsIURI *,unsigned int,int,int,nsIAppShell *,nsIXULWindow * *)
 6  xul.dll + 0x1bf395
    eip = 0x101bf396   esp = 0x0012e87c   ebp = 0x0012e894
    Found by: previous frame's frame pointer

 7  xul.dll + 0x1a0f1f
    eip = 0x101a0f20   esp = 0x0012e89c   ebp = 0x0012e8d4
    Found by: previous frame's frame pointer

FUNC 114494 918 1c nsWindowWatcher::OpenWindowJSInternal(nsIDOMWindow *,char const *,char const *,char const *,int,nsIArray *,int,nsIDOMWindow * *)
 8  xul.dll + 0x114886
    eip = 0x10114887   esp = 0x0012e8dc   ebp = 0x0012eb24
    Found by: previous frame's frame pointer

FUNC 167346 161 1c nsWindowWatcher::OpenWindow(nsIDOMWindow *,char const *,char const *,char const *,nsISupports *,nsIDOMWindow * *)
 9  xul.dll + 0x167484
    eip = 0x10167485   esp = 0x0012eb2c   ebp = 0x0012eb8c
    Found by: previous frame's frame pointer

FUNC 21134a 2b 10 NS_InvokeByIndex_P
10  xul.dll + 0x211370
    eip = 0x10211371   esp = 0x0012eb94   ebp = 0x0012ebc8
    Found by: previous frame's frame pointer

FUNC 58e00 1a78 8 XPCWrappedNative::CallMethod(XPCCallContext &,XPCWrappedNative::CallMode)
11  xul.dll + 0x592d7
    eip = 0x100592d8   esp = 0x0012ebd0   ebp = 0x0012eda8
    Found by: previous frame's frame pointer

FUNC 5c970 1c6 14 XPC_WN_CallMethod(JSContext *,JSObject *,unsigned int,int *,int *)
12  xul.dll + 0x5ca86
    eip = 0x1005ca87   esp = 0x0012edb0   ebp = 0x00000000
    Found by: previous frame's frame pointer

13  xul.dll + 0x8b78df
    eip = 0x108b78e0   esp = 0x0012edd4   ebp = 0x00000000
    Found by: stack scanning

Comment 18

[Jeff Muizelaar [:jrmuizel]]

Here's some stuff that comes after that:

1005c970
FUNC 5c970 1c6 14 XPC_WN_CallMethod(JSContext *,JSObject *,unsigned int,int *,int *)

1008bd16
FUNC 8bbd0 14e 8 nsXPCWrappedJS::nsXPCWrappedJS(XPCCallContext &,JSObject *,nsXPCWrappedJSClass *,nsXPCWrappedJS *,nsISupports *)

FUNC 5f150 109b 2c XPCConvert::NativeInterface2JSObject(XPCCallContext &,int *,nsIXPConnectJSObjectHolder * *,nsISupports *,nsID const *,XPCNativeInt
1005f3ef

FUNC 5f150 109b 2c XPCConvert::NativeInterface2JSObject(XPCCallContext &,int *,nsIXPConnectJSObjectHolder * *,nsISupports *,nsID const *,XPCNativeInt
1005f4f2

FUNC 5f150 109b 2c XPCConvert::NativeInterface2JSObject(XPCCallContext &,int *,n
1005f494

FUNC 5d920 12a2 14 nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
1005df64

FUNC bcb20 53 c nsLocalFile::Equals(nsIHashable *,int *)
100bcb6c

1005694e

FUNC 2b830 48c c mozJSComponentLoader::LoadModule(nsILocalFile *,nsIModule * *)
1002bc8d

FUNC 2114d6 138 10 PrepareAndDispatch
102115bd

1003bb9e

FUNC 28f102 19 c Substring(nsACString_internal const &,unsigned int,unsigned int)
1028f116


More to come...

Comment 19

[Jeff Muizelaar [:jrmuizel]]

Here's the rest. Note: these stacks can be pretty inaccurate but should still be useful.

1003bb9e

FUNC 28f102 19 c Substring(nsACString_internal const &,unsigned int,unsigned int)
1028f116

FUNC 72bc0 3b 10 nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
10072bf8

FUNC 2114d6 138 10 PrepareAndDispatch
102115bd

FUNC 6e7a0 22e 4 nsXPCWrappedJS::Release()
1006e7e7

FUNC 52760 557 10 nsComponentManagerImpl::GetServiceByContractID(char const *,nsID const &,void * *)
10052c03

FUNC 21160e 26 0 SharedStub
10211624

FUNC 1e02bc f c EnumValidate
101e02ca

FUNC 132bb2 20d c nsCommandLine::EnumerateHandlers(unsigned int (*)(nsICommandLineHandler *,nsICommandLine *,void *),voi
d *)
10132d39

FUNC 1e02bc f c EnumValidate
101e02bc

FUNC 1c27f5 3d 4 nsCommandLine::Run()
101c2821

FUNC 1e02bc f c EnumValidate
101e02bc

FUNC 6ccea2 24a c nsNativeAppSupportWin::HandleCommandLine(char const *,nsIFile *,unsigned int)
106cd0ce

FUNC 4861bb 8e c nsNativeAppSupportWin::ParseDDEArg(unsigned short const *,int,nsString &)
1048620c

FUNC 4861bb 8e c nsNativeAppSupportWin::ParseDDEArg(unsigned short const *,int,nsString &)
10486232

FUNC 4861bb 8e c nsNativeAppSupportWin::ParseDDEArg(unsigned short const *,int,nsString &)
10486245

FUNC 3eaaa3 19 c nsAString_internal::Replace(unsigned int,unsigned int,nsAString_internal const &)
103eaab9

FUNC 73e403 5bd 20 nsNativeAppSupportWin::HandleDDENotification(unsigned int,unsigned int,HCONV__ *,HSZ__ *,HSZ__ *,HDDEDATA__ *,unsigned long,unsign
1073e97e

FUNC 73e403 5bd 20 nsNativeAppSupportWin::HandleDDENotification(unsigned int,unsigned int,HCONV__ *,HSZ__ *,HSZ__ *,HDDEDATA__ *,unsigned long,unsign
1073e403

FUNC a04c0 c5 4 nsAppShell::ProcessNextNativeEvent(int)
100a056e

FUNC 88850 2cc 10 nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal *,int,unsigned int)
100888ea

FUNC 872f0 45e c nsThread::ProcessNextEvent(int,int *)
10087475

FUNC f1a4c 33 8 NS_ProcessNextEvent_P(nsIThread *,int)
100f1a6c

FUNC f1a7f d0 4 nsThread::Shutdown()
100f1b47

FUNC 1a02d4 ea 4 nsSocketTransportService::Shutdown()
101a035a

FUNC 178440 d3 8 nsIOService::SetOffline(int)
101784ea

FUNC 1ad38b 63 0 nsIOService::TrackNetworkLinkStatusForOffline()
101ad335

FUNC a5180 a0 4 nsTimerImpl::Release()
100a51d8

FUNC 17a640 ae 0 mozJSComponentLoader::CloseFastLoad()
1017a6ec

FUNC 1afc1e 56 10 mozJSComponentLoader::Observe(nsISupports *,char const *,unsigned short const *)
101afc4c

FUNC 28420 fb 10 nsObserverService::NotifyObservers(nsISupports *,char const *,unsigned short const *)
10028509

FUNC 172904 223 4 NS_ShutdownXPCOM_P
10172995

FUNC 1be3c0 44 0 ScopedXPCOMStartup::~ScopedXPCOMStartup()
101be3f5

FUNC 10f988 f88 c XRE_main
10110814

FUNC 51250 61f 8 nsLocalFile::GetParent(nsIFile * *)
100517a9

FUNC 54680 d9 4 nsLocalFile::Release()
10054693

FUNC 56940 f 0 nsCOMPtr_base::~nsCOMPtr_base()
1005694e

FUNC 1b3e4b bf 8 XRE_CreateAppData
101b3f04

Comment 20

[Olli Pettay [:smaug][[e-mail]]]

To me this looks like we're processing DDE/commandline handlers *after*
shutting down gklayout. And that is bad.

Benjamin, perhaps you know the reason for that, or
is that just a bug?

Comment 21

[Olli Pettay [:smaug][[e-mail]]]

The same reason might cause Bug 525575.

Comment 22

[Olli Pettay [:smaug][[e-mail]]]

And Jeff, thanks!
Could you please write down somewhere how to get that data out of
minidumps.

Comment 23

[Benjamin Smedberg]

I think that's just a bug. There is a nsINativeAppSupport::Quit API, but we don't appear to be using it any more. I'm going to have to troll through history a bit and figure out if/when it got removed.

Comment 24

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #22)
> And Jeff, thanks!
> Could you please write down somewhere how to get that data out of
> minidumps.

I've added some instructions to https://developer.mozilla.org/en/Debugging_a_minidump Hopefully they make sense.

Comment 25

[Benjamin Smedberg]

nsNativeAppSupportWin::Quit is called by a quit-application observer. We don't always fire quit-application observers, though, especially when we're in the initial restart sequence.

It's also really strange that we have this nsINativeAppSupport::Stop and ::Quit API which we hardly ever use.

Comment 26

[Benjamin Smedberg]

It's certainly possible to imagine that we posted a "process a remote command line" event, then started the shutdown sequence that led to this crash, and I'm not sure there's a lot we can do about that from the startup sequence.

Comment 27

[Olli Pettay [:smaug][[e-mail]]]

(In reply to comment #25)
> nsNativeAppSupportWin::Quit is called by a quit-application observer. We don't
> always fire quit-application observers, though, especially when we're in the
> initial restart sequence.
Well, then nsNativeAppSupportWin needs to observe some other notification.

Comment 28

[Olli Pettay [:smaug][[e-mail]]]

Attachment: "xpcom-will-shutdown"

Something like this could work.
Benjamin, is this even close to something that you could accept?

Comment 29

[Benjamin Smedberg]

yes

Comment 30

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 413093 [details] [diff] [review]
"xpcom-will-shutdown"

I tried this manually.
Without the patch trying to open a new window in an xpcom-shutdown observer leads to a crash (not necessarily in nsEventListenerManager::AddEventListenerByType).
The patch fixes the crash.

Comment 31

[Benjamin Smedberg]

Comment on attachment 413093 [details] [diff] [review]
"xpcom-will-shutdown"

Please update https://wiki.mozilla.org/XPCOM_Shutdown when this lands.

Comment 32

[Johnny Stenback (:jst)]

We should fix this for 3.6 even if we haven't seen this crash there yet. I'd bet we will. I also think this will fix bug 525575.

Comment 33

[Johnny Stenback (:jst)]

Attachment: Same thing for 1.9.2

Comment 34

[Johnny Stenback (:jst)]

Fixed on trunk and 1.9.2.

http://hg.mozilla.org/mozilla-central/rev/239c5a22fdaf
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/5a515e5b7c6a

Comment 35

[:Gavin Sharp [email: [e-mail]]]

Looks like https://developer.mozilla.org/en/Observer_Notifications and https://wiki.mozilla.org/XPCOM_Shutdown could still use updating.

Comment 36

[Olli Pettay [:smaug][[e-mail]]]

Jst, seems like you added a printf to 1.9.2. I'll remove that.

Comment 37

[Olli Pettay [:smaug][[e-mail]]]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/608fbb4d4ab1

Comment 38

[Johnny Stenback (:jst)]

Oh, duh! Yeah, that was never meant to be checked in, that was just me verifying that my merged patch still worked. Thanks for fixing!