Last Comment Bug 682444 - Thunderbird crashes at exit (shutdown) _beginthread, related to DanID/PKCS11
: Thunderbird crashes at exit (shutdown) _beginthread, related to DanID/PKCS11
Status: NEW
[psm-smartcard][gs]
: crash, regression
Product: Thunderbird
Classification: Client Software
Component: Security (show other bugs)
: 5.0
: x86_64 Windows 7
: -- critical with 4 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
:
Mentors:
http://getsatisfaction.com/mozilla_me...
: 767954 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-26 15:40 PDT by Henrik Larsson
Modified: 2016-07-20 22:58 PDT (History)
15 users (show)
vseerror: needinfo? (mozilla)
vseerror: needinfo? (mikkel)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Stack trace (67.09 KB, text/plain)
2011-10-18 10:42 PDT, Henrik Larsson
no flags Details
Stack trace 002 (56.13 KB, text/plain)
2011-11-04 15:29 PDT, Henrik Larsson
no flags Details
Stack trace 003 (56.71 KB, text/plain)
2011-11-04 15:29 PDT, Henrik Larsson
no flags Details
Stack trace 004 (61.59 KB, text/plain)
2011-11-04 15:30 PDT, Henrik Larsson
no flags Details

Description Henrik Larsson 2011-08-26 15:40:47 PDT
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110811165603

Steps to reproduce:

Open Thunderbird
Wait, typicall the longer you wait the more possible it is to experience the issue
Close Thunderbird

This happened in 5.0 as well.


Actual results:

After close, Thunderbird crash and show the crash reporter


Expected results:

Thunderbird shouldn't crash on exit
Comment 1 Henrik Larsson 2011-08-26 15:45:23 PDT
bp-57b8558a-6e19-4cd9-8714-7dec42110826
Comment 2 Henrik Larsson 2011-08-26 15:49:11 PDT
I have disabled all plug-ins, still crashes
I have removed all extensions, still crashes (i reinstalled a danish spell extension afterwards)

Looking at the crash report, a lot of other users have this issue with Thunderbird 5.0 and 6, but not earlier versions.
Comment 3 Henrik Larsson 2011-08-26 16:08:06 PDT
One more:
bp-ce05cc0c-4b15-4ec2-a598-ae9b82110826
Comment 4 Wayne Mery (:wsmwk, NI for questions) 2011-08-26 19:34:20 PDT
regression, beginning version 5


0		@0x6a1d81c9	
1	msvcrt.dll	_beginthread	
2	msvcrt.dll	_beginthread	
3	kernel32.dll	BaseThreadInitThunk	
4	ntdll.dll	__RtlUserThreadStart	

(In reply to Henrik Larsson from comment #2)
> Looking at the crash report, a lot of other users have this issue with
> Thunderbird 5.0 and 6, but not earlier versions.
actually, the average person crashes has submitted 5-10 crashes (and as many as ~20)
Comment 6 Henrik Larsson 2011-08-31 15:30:58 PDT
And it is still crashing in 6.0.1
bp-e3f67564-e26a-407f-9e14-857082110831
Comment 7 Henrik Larsson 2011-09-08 12:23:27 PDT
And it is still crashing in 6.0.2
bp-ead6dfb6-0073-4574-ae1c-4ecb62110908
Comment 8 Joshua Cranmer [:jcranmer] 2011-09-08 12:34:20 PDT
(In reply to Wayne Mery (:wsmwk) from comment #4)
> regression, beginning version 5
> 
> 
> 0		@0x6a1d81c9	
> 1	msvcrt.dll	_beginthread	
> 2	msvcrt.dll	_beginthread	
> 3	kernel32.dll	BaseThreadInitThunk	
> 4	ntdll.dll	__RtlUserThreadStart	

That sounds like something is passing a bad pointer into a thread initialization?
Comment 9 Henrik Larsson 2011-09-08 13:16:50 PDT
I have noticed that I can reproduce the crash 100% each time, if I go to "Help -> About Thunderbird" before shutdown.

Could it be related to the update check function?
Comment 11 Henrik Larsson 2011-10-01 10:41:46 PDT
Please me what information I should collect to get this issue resolved. This problem has been "Unconfirmed" for over a month.

By looking at the crash stats, several people has this issue.
Comment 12 Ludovic Hirlimann [:Usul] 2011-10-03 02:47:40 PDT
Henrik does starting in -safe-mode helps ? (see http://support.mozillamessaging.com/en-US/kb/Safe-Mode)
Comment 13 David :Bienvenu 2011-10-03 07:19:12 PDT
That's not a useful stack trace, in the sense that it doesn't give us any idea what code is causing the crash. Safe-mode is definitely the next thing to try.
Comment 14 Henrik Larsson 2011-10-18 10:01:52 PDT
This is the same crash in safe-mode:
bp-da217d39-d831-4c97-8f1d-9273f2111018
Comment 15 Henrik Larsson 2011-10-18 10:03:49 PDT
Way to reproduce consistent and every time (also in safe-mode):
- Open Thunderbird
- Go to "Help -> About Thunderbird"
- Wait for the update process to finish (even if no updates are available)
- Close Thunderbird
Comment 16 Henrik Larsson 2011-10-18 10:42:17 PDT
Created attachment 567796 [details]
Stack trace

I have tried to create a stack trace using WinDbg. I have no idea on what I have been doing, I have just been following a guide.
Comment 17 Henrik Larsson 2011-11-03 00:12:00 PDT
Is the submitted stack trace any good? Please me what information I should collect to get this issue resolved.
Comment 18 Ludovic Hirlimann [:Usul] 2011-11-03 05:32:56 PDT
(In reply to Henrik Larsson from comment #17)
> Is the submitted stack trace any good? Please me what information I should
> collect to get this issue resolved.

It looks good after a quick check bienvenu am I not mistaken , we crash in svg code right ?
Comment 19 David :Bienvenu 2011-11-03 06:28:34 PDT
(In reply to Ludovic Hirlimann [:Usul] from comment #18)
> (In reply to Henrik Larsson from comment #17)
> > Is the submitted stack trace any good? Please me what information I should
> > collect to get this issue resolved.
> 
> It looks good after a quick check bienvenu am I not mistaken , we crash in
> svg code right ?

If I had to guess, I would say that SVG is not unregistering its listeners before unloading/shutting down. I'm not seeing this crash, however.
Comment 20 David :Bienvenu 2011-11-03 06:31:11 PDT
It's also possible that the last part of the stack trace is confused and it's not SVG at all - but it's definitely an issue with a listener, perhaps a ref-counting issue. But reproducing this is the first thing we need to be able to do.
Comment 21 David :Bienvenu 2011-11-04 11:09:14 PDT
Daniel offered to look at this stack - it's possible that the cycle collector is calling into random code...
Comment 22 Daniel Holbert [:dholbert] 2011-11-04 11:22:38 PDT
(In reply to David :Bienvenu from comment #21)
> it's possible that the cycle collector is calling into random code...

That's what I suspected/hoped, by analogy to other mysterious crashes in SVG code, but that doesn't seem to be the case, from looking at the stack.  Very little cycle-collector code, and the first SVG call on the stack is a sensible one I think.  I don't immediately see where ~nsCOMPtr_base calls AddRef, but it seems reasonable that it might temporarily AddRef before destroying, to stabilize the refcount, or something.

Note that nsSVGMutationObserver::AddRef is a no-op (it's just "return 1"), so it could only be crashing if the nsSVGMutationObserver in question is a null/bogus/deleted pointer.

> If I had to guess, I would say that SVG is not unregistering its listeners

Looks like it could be something like that, yeah.
Comment 23 Daniel Holbert [:dholbert] 2011-11-04 13:39:49 PDT
Actually, on closer inspection, there is some evidence of stack corruption.  When we enter nsObserverList::NotifyObservers, things turn crazy.  Consider |aTopic| and |someData| in the following two lines:
> xul!nsObserverList::NotifyObservers(class nsISupports * aSubject = 0x09a4eab0, char * aTopic = 0x00768044 "HQ???", wchar_t * someData = 0x59a83fb4 "?????")+0x44
> xul!nsObserverService::NotifyObservers(class nsISupports * aSubject = 0x00768044, char * aTopic = 0x59a83fb4 "xpcom-shutdown", wchar_t * someData = 0x00000000 "")+0x50

The nsObserverService call has sensible values for aTopic and someData, and it always passes its values down untouched AFAICT.  However, in this case it looks like nsObserverList ended up with different (junk) values for those arguments.

A few other observations:
 - nsObserverService::NotifyObservers seems to be calling into a JS implementation of an observer.  (its Observe call descends into glue code and then JS impl code)

 - The nsSVGMutationObserver's nsCOMPtr here is getting destructed inside of ~xpcObjectHelper, which only owns two nsCOMPtrs, only one of which holds something that nsSVGMutationObserver derives from -- "nsCOMPtr<nsISupports> mCanonicalStrong".  I don't really know anything about xpcObjectHelper, but it seems to be a way to wrap C++ objects for interaction with script, and it's very odd/unexpected we'd do that for a nsSVGMutationObserver...  So I wonder if this ends up being a case where the xpcObjectHelper ends up just pointing to a random location in memory, which in this case just happened to look like a nsSVGMutationObserver.
Comment 24 Daniel Holbert [:dholbert] 2011-11-04 13:41:45 PDT
Henrik: if you could produce & attach any more stack traces like the one you've attached, that'd be helpful.  In particular, I'm curious whether this _always_ ends up in SVG code, or if it just ends up somewhere random. (per the end of my previous comment)
Comment 25 Henrik Larsson 2011-11-04 15:29:24 PDT
Created attachment 572102 [details]
Stack trace 002
Comment 26 Henrik Larsson 2011-11-04 15:29:57 PDT
Created attachment 572103 [details]
Stack trace 003
Comment 27 Henrik Larsson 2011-11-04 15:30:37 PDT
Created attachment 572104 [details]
Stack trace 004
Comment 28 Henrik Larsson 2011-11-04 15:32:48 PDT
I have added some more stack traces, produced by using this guide:
https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

Please let me know if any of the commands should be changed or added in WinDbg. As explained before I don't know what I'm doing, I just follow the guide.
Comment 29 David :Bienvenu 2011-11-04 15:36:53 PDT
Henrik, what windows do you have open before you shut down? Do you do a file exit from the 3 pane UI with the about window open?
Comment 30 Henrik Larsson 2011-11-04 15:39:46 PDT
This is how I reproduce:
- Open Thunderbird
- Help -> About Thunderbird
- Close Help -> About Thunderbird Windows (red cross)
- Close Thunderbird (red cross)
Comment 31 Daniel Holbert [:dholbert] 2011-11-04 15:45:24 PDT
Thanks for the additional stack traces! They appear to confirm that this isn't SVG-specific, as I suspected in Comment 24. (none of the new stacktraces have any SVG in the stack)

The last one (attachment 572104 [details]) is the most similar to the original stack trace -- it's got the same mysteriously-bogus-aTopic/someData values in nsObserverList::NotifyObservers, and the stack is the same up to the NativeInterface2JSObject() call (but it differs inside of that).

The other two new stack traces (attachment 572102 [details], attachment 572103 [details]) are simpler-looking, though -- in those ones, aTopic / someData are preserved intact (rather than having bogus values), and there's no JS observer implementation.
Comment 32 Henrik Larsson 2011-11-04 16:06:27 PDT
The plot thickens.

I always wondered why the comments for these crashes looked like a lot of them was from Danish people.

By looking at the stack traces I found the following modules loaded:
ModLoad: 00000000`69240000 00000000`69387000   C:\Program Files\DanID\NemID\NemID_PKCS11.dll
ModLoad: 00000000`6fbc0000 00000000`6fbc7000   C:\Program Files\DanID\NemID\mingwm10.dll
ModLoad: 00000000`6a1c0000 00000000`6a463000   C:\Program Files\DanID\NemID\QtCore4.dll
ModLoad: 00000000`65100000 00000000`65bcb000   C:\Program Files\DanID\NemID\QtGui4.dll
ModLoad: 00000000`6ff00000 00000000`7005d000   C:\Program Files\DanID\NemID\QtNetwork4.dll

These are a part of a danish citizen certificate program. These modules is loaded as a "Security Device" in Thunderbird, because I wanted to use my certificate for secure e-mail.

Apparently, running Thunderbird in "safe mode" still loads this added "Security Device".

If I uninstall the program, or unload the "Security Device" in Thunderbird, the crashes stop. So is this a problem with the way Thunderbird adds the "Security Device" or a problem with the program from NemID.

You can find the program here:
https://www.nemid.nu/support/sikker_e-mail/i_gang_med_sikker_e-mail/hent_program/windows/

And a description on how to add this as a "Security Device" in Thunderbird here:
https://www.nemid.nu/support/sikker_e-mail/i_gang_med_sikker_e-mail/opsaetning_af_e-mail-program/windows_7/mozilla_thunderbird_3/

Unfortunately both links are in Danish, so Google Translate might help ;o)
Comment 33 David :Bienvenu 2011-11-04 16:08:14 PDT
This all feels like some extension and/or binary component isn't unregistering itself.


This particular stack segment is disturbing:

0021f87c 59f3728f xul!nsObserverList::NotifyObservers(class nsISupports * aSubject = 0x0a552b30, char * aTopic = 0x0056e044 "HQLZ4QLZ", wchar_t * someData = 0x5a283fb4 "?????")+0x44 [e:\buildbot\win32_build_release\build\mozilla\xpcom\ds\nsobserverlist.cpp @ 130]
0021f898 59f2fc12 xul!nsObserverService::NotifyObservers(class nsISupports * aSubject = 0x0056e044, char * aTopic = 0x5a283fb4 "xpcom-shutdown", wchar_t * someData = 0x00000000 "")+0x50 [e:\buildbot\win32_build_release\build\mozilla\xpcom\ds\nsobserverservice.cpp @ 185]

because aTopic seems to be getting crunched during one of the notifications; it starts of xpcom-shutdown and then becomes HQLZ4QLZ
Comment 34 David :Bienvenu 2011-11-04 16:09:04 PDT
yeah, security devices aren't extensions per se...
Comment 35 David :Bienvenu 2011-11-04 16:10:20 PDT
Kai might have some insight into this.
Comment 36 David :Bienvenu 2011-11-04 16:13:07 PDT
I strongly suspect it's some problem with the security device code, and contacting the author is the first place to start.
Comment 37 Henrik Larsson 2011-11-04 16:17:00 PDT
It might be, but this problem started with Thunderbird 5.0, before this I didn't experience the crash.
Comment 38 Daniel Holbert [:dholbert] 2011-11-04 16:32:16 PDT
It's common for binary Mozilla Add-ons to experience random issues after Thunderbird/Firefox is updated, if there's a change to a XPCOM API that's used by the Add-on.  In these cases, the Add-on needs to be recompiled in order to function correctly.

(I'm not sure whether this is true of security device DLLs in particular, but it seems likely that it is.)

Note that the last URL linked in comment 32 refers to "Thunderbird 3".  A lot has changed since then -- I'd actually be surprised if binary Add-ons that use APIs from Thunderbird 3 still worked flawlessly in Thunderbird >5.
Comment 39 David :Bienvenu 2011-11-04 16:34:02 PDT
I'm curious what aspect(s) of the security device code are Thunderbird-specific. Do they have a similar download for Firefox?
Comment 40 Henrik Larsson 2011-11-04 16:52:37 PDT
The "Security Device" is needed in order to unlock the certificate issued by NemID. You can only use this certificate for e-mail signing and encryption, so the download is only for supported e-mail clients. You have to use a code card or code token to unlock the certificate for each use.

The certificate is not used for authentication in browsers. Instead a proprietary system currently running in java is used. The private key is stored at NemID servers, and you use a code card or code token to login. So in this case, you don't need to install any software.
Comment 41 David :Bienvenu 2011-11-04 16:54:57 PDT
ah, just curious if they had a newer version for Firefox. Quite a bit has changed in Thunderbird and core gecko code since TB 3, so binary components definitely need to be recompiled, and even some js components need tweaking.
Comment 42 Henrik Larsson 2011-11-04 17:06:17 PDT
The "Security Device" was obviously created in TB3, and just stayed in the config since. The "Security Device" was not version checked, like the add-ons during upgrades, so I wasn't taking any notice of this.

I will contact NemID, and ask for an updated version of their software if any.
Comment 43 Henrik Larsson 2011-11-05 05:09:55 PDT
This issue could be related to Bug 680714
Comment 44 mikkel@mzh.dk 2011-11-21 15:43:51 PST
I have the exact same problem as Henrik - and I, too, have the DanID module installed. My problems also began with TB5.

Let me know if you want me to submit any crash reports/any other info.
Comment 45 Jørgen Rasmussen 2012-04-01 11:39:22 PDT
I am experiencing the same bug. Is there any way I can help the developers in fixing this bug?
Comment 46 Wayne Mery (:wsmwk, NI for questions) 2012-04-01 11:45:15 PDT
(In reply to Henrik Larsson from comment #43)
> This issue could be related to Bug 680714

Unfortunatelly it has stagnated in NSS
Comment 47 klonos 2012-06-11 15:21:08 PDT
I always use latest nightlies (comm-central) and the x64 ones since they came out (I'm on Win7 x64).

I started having these annoying crashes (the application works just fine nevertheless) on close/restart with the few last 15.0a1 builds and continue to have these crashes till this day with the latest 16.0a1 x64. I managed to track it down to the May 20th build. I have no crash with the May 18th one (and there was no win x64 build on May 19th).

I've never reported this before till now because I figured it must be some regression that would be fixed soon (like the one where no mail folders were shown with the first 16.0a1 x64 build - it was solved with the very next build). Well, obviously it is still not solved and it got annoying enough to make me go searching for any related reported bugs. Hope I'm in the right place :)

I'm not getting the crash on a fresh (though empty - no accounts set) profile so this might be caused by an extension.

Here's the details from the event id 1000 I get in my Windows application log:

Faulting application name: thunderbird.exe, version: 16.0.0.4545, time stamp: 0x4fd5d7fe
Faulting module name: xul.dll_unloaded, version: 0.0.0.0, time stamp: 0x4fd5d770
Exception code: 0xc0000005
Fault offset: 0x000007fedd511bf0
Faulting process id: 0x15b8
Faulting application start time: 0x01cd481f5fe81439
Faulting application path: C:\Program Files\Mozilla\Thunderbird\thunderbird.exe
Faulting module path: xul.dll
Report Id: a5b07457-b412-11e1-a5c0-000df07e119d

Hope this info helps.
Comment 48 David :Bienvenu 2012-06-11 15:23:41 PDT
(In reply to klonos from comment #47)
> I always use latest nightlies (comm-central) and the x64 ones since they
> came out (I'm on Win7 x64).
> 
> I started having these annoying crashes (the application works just fine
> nevertheless) on close/restart with the few last 15.0a1 builds and continue
> to have these crashes till this day with the latest 16.0a1 x64. 
I rather think this is a different issue - do you have telemetry turned on? tools, options, advanced, general tab, submit performance info checkbox. I've found that unchecking that stops me from crashing on exit.
Comment 49 klonos 2012-06-12 02:27:05 PDT
...you know what? I'd never have thought of that. You are a life saver David! Thanx.

Is there a bug report filed for that that I can follow btw? I'm asking so to know when this gets fixed so I can then turn the telemetry back on (I want to help as much as I can by providing feedback and that does it automatically).

...once again thanx!
Comment 50 Wayne Mery (:wsmwk, NI for questions) 2012-07-02 11:00:26 PDT
*** Bug 767954 has been marked as a duplicate of this bug. ***
Comment 51 Yev 2012-07-18 14:08:02 PDT
I've had this issue since the x64 version came out, I'm not sure what is different but on the latest update to version 17 I am no longer getting crashes or windows error reporting when closing daily (No crash reports opening in background either)
Comment 52 Jørgen Rasmussen 2012-07-18 15:53:17 PDT
I have just tested Daily thunderbird-17.0a1 X64 and Daily thunderbird-17.0a1 x32 on my Windows 7.
In the x64 version I get the following message 'Certificate Manager can't locate a valid certificate that can be used to digitally sign your messages' when I try to select a certificate in account settings.

The x32 version crashed when I exit Thunderbird.
Crash ID: bp-9691e630-dacd-4a66-a5f3-4dacc2120718
Crash ID: bp-cb970f1f-45a3-4e03-89c4-f7e3a2120718
Crash ID: bp-1dd1e0b9-0f99-488c-b842-6b0c42120718
Comment 53 Yev 2012-07-20 15:11:46 PDT
well, this bug is for the x64 version crashing, I have no idea about what is wrong with your "certificates" (try doing a windows update? sometimes they release new ones), but it didn't sound like your x64 version crashed on version 17 either.

You should probably post about the x86 version crashing in another bug though.
Comment 54 Jørgen Rasmussen 2012-07-20 15:28:28 PDT
Hi Yev

As you can see from the original poster Henrik Larsson's comment 32, the crashes are caused by a PKCS11.dll file from DanID. The crash ID is _beginthread.
Is your Thunderbird crashing with the same Crash ID?
Comment 55 Yev 2012-07-21 11:34:44 PDT
Oh my apologies, didn't realize it was narrowed down to the pkcs11.dll file.

However, I can't confirm if that's what was causing it to crash because ever since I upgraded to version 17.0a1 it stopped crashing on exit. I just wanted to point that out because on every nightly with 16.0a1 it would crash 100% of the time. At some point version 16.0a1 stopped showing a message "Daily has crashed etc..." but the background process "WerFault" would run as soon as it was exited (Windows Error Reporting).

Anyways since version 17.0a1 there are no crashes and "WerFault" doesn't open up at all
Comment 56 Wayne Mery (:wsmwk, NI for questions) 2015-11-26 01:25:55 PST
Is the crash gone for you in newer versions?
And if not, please post crash ID. 
Thanks
Comment 57 Jørgen Rasmussen 2015-11-26 04:50:59 PST
The company behind the Danish certificates has updated the Security Device (PKCS11.dl) at least twice since this bug was reported. With the first update the Crash Reporter didn't catch the crash anymore, but the crash was recorded by Windows Event Viewer. With the latest version (1.3.1.0) the problem has gone as far as I can see.
Comment 58 Pavel Spálený 2016-07-20 22:58:16 PDT
I have the same. It look, that this is not fixed. Everytime I send crash report via Crash reporter.

Only I know, that thunderbord must run for some time. If I open it and close immediately, it's ok.

Note You need to log in before you can comment on or make changes to this bug.