The default bug view has changed. See this FAQ.

Thunderbird crashes at exit (shutdown) _beginthread, related to DanID/PKCS11

NEW
Unassigned
(Needinfo from 2 people)

Status

Thunderbird
Security
--
critical
6 years ago
8 months ago

People

(Reporter: Henrik Larsson, Unassigned, NeedInfo)

Tracking

({crash, regression})

x86_64
Windows 7
crash, regression

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [psm-smartcard][gs], crash signature, URL)

Attachments

(4 attachments)

(Reporter)

Description

6 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110811165603

Steps to reproduce:

Open Thunderbird
Wait, typicall the longer you wait the more possible it is to experience the issue
Close Thunderbird

This happened in 5.0 as well.


Actual results:

After close, Thunderbird crash and show the crash reporter


Expected results:

Thunderbird shouldn't crash on exit
(Reporter)

Updated

6 years ago
Crash Signature: 57b8558a-6e19-4cd9-8714-7dec42110826
(Reporter)

Comment 1

6 years ago
bp-57b8558a-6e19-4cd9-8714-7dec42110826
Severity: normal → major
Crash Signature: 57b8558a-6e19-4cd9-8714-7dec42110826
(Reporter)

Comment 2

6 years ago
I have disabled all plug-ins, still crashes
I have removed all extensions, still crashes (i reinstalled a danish spell extension afterwards)

Looking at the crash report, a lot of other users have this issue with Thunderbird 5.0 and 6, but not earlier versions.
(Reporter)

Comment 3

6 years ago
One more:
bp-ce05cc0c-4b15-4ec2-a598-ae9b82110826
regression, beginning version 5


0		@0x6a1d81c9	
1	msvcrt.dll	_beginthread	
2	msvcrt.dll	_beginthread	
3	kernel32.dll	BaseThreadInitThunk	
4	ntdll.dll	__RtlUserThreadStart	

(In reply to Henrik Larsson from comment #2)
> Looking at the crash report, a lot of other users have this issue with
> Thunderbird 5.0 and 6, but not earlier versions.
actually, the average person crashes has submitted 5-10 crashes (and as many as ~20)
Severity: major → critical
Crash Signature: [@ _beginthread]
Keywords: crash, regression
Summary: Thunderbird crashes at exit → Thunderbird crashes at exit (shutdown) _beginthread
Version: 6 → 5.0
(Reporter)

Comment 5

6 years ago
bp-3ccbfaf5-0800-4ca9-92bf-a76842110827
bp-53eeeae9-0189-4371-b768-69f1a2110827
bp-73e1407a-0bd9-4f83-8f15-c4b142110828
bp-e16c3fc7-9aa9-4774-b7a8-c681a2110829
bp-2547a446-2a40-4b49-97c9-1e53c2110830
bp-a09acc83-fcde-4eb6-a33f-608082110831
bp-291c7528-f7b4-458b-9021-54da42110831
(Reporter)

Comment 6

6 years ago
And it is still crashing in 6.0.1
bp-e3f67564-e26a-407f-9e14-857082110831
(Reporter)

Comment 7

6 years ago
And it is still crashing in 6.0.2
bp-ead6dfb6-0073-4574-ae1c-4ecb62110908
(In reply to Wayne Mery (:wsmwk) from comment #4)
> regression, beginning version 5
> 
> 
> 0		@0x6a1d81c9	
> 1	msvcrt.dll	_beginthread	
> 2	msvcrt.dll	_beginthread	
> 3	kernel32.dll	BaseThreadInitThunk	
> 4	ntdll.dll	__RtlUserThreadStart	

That sounds like something is passing a bad pointer into a thread initialization?
(Reporter)

Comment 9

6 years ago
I have noticed that I can reproduce the crash 100% each time, if I go to "Help -> About Thunderbird" before shutdown.

Could it be related to the update check function?
(Reporter)

Comment 10

6 years ago
And it is still crashing in 7.0.1
bp-455692a7-282e-4f79-a62e-9f40a2111001
bp-226eb89b-4651-4b13-a5da-4550b2111001
bp-eec470ab-f91d-430e-b18b-9c84b2111001
(Reporter)

Comment 11

6 years ago
Please me what information I should collect to get this issue resolved. This problem has been "Unconfirmed" for over a month.

By looking at the crash stats, several people has this issue.
Henrik does starting in -safe-mode helps ? (see http://support.mozillamessaging.com/en-US/kb/Safe-Mode)
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 13

6 years ago
That's not a useful stack trace, in the sense that it doesn't give us any idea what code is causing the crash. Safe-mode is definitely the next thing to try.
(Reporter)

Comment 14

6 years ago
This is the same crash in safe-mode:
bp-da217d39-d831-4c97-8f1d-9273f2111018
(Reporter)

Comment 15

6 years ago
Way to reproduce consistent and every time (also in safe-mode):
- Open Thunderbird
- Go to "Help -> About Thunderbird"
- Wait for the update process to finish (even if no updates are available)
- Close Thunderbird
(Reporter)

Comment 16

6 years ago
Created attachment 567796 [details]
Stack trace

I have tried to create a stack trace using WinDbg. I have no idea on what I have been doing, I have just been following a guide.
(Reporter)

Comment 17

6 years ago
Is the submitted stack trace any good? Please me what information I should collect to get this issue resolved.
(In reply to Henrik Larsson from comment #17)
> Is the submitted stack trace any good? Please me what information I should
> collect to get this issue resolved.

It looks good after a quick check bienvenu am I not mistaken , we crash in svg code right ?

Comment 19

6 years ago
(In reply to Ludovic Hirlimann [:Usul] from comment #18)
> (In reply to Henrik Larsson from comment #17)
> > Is the submitted stack trace any good? Please me what information I should
> > collect to get this issue resolved.
> 
> It looks good after a quick check bienvenu am I not mistaken , we crash in
> svg code right ?

If I had to guess, I would say that SVG is not unregistering its listeners before unloading/shutting down. I'm not seeing this crash, however.

Comment 20

6 years ago
It's also possible that the last part of the stack trace is confused and it's not SVG at all - but it's definitely an issue with a listener, perhaps a ref-counting issue. But reproducing this is the first thing we need to be able to do.

Comment 21

6 years ago
Daniel offered to look at this stack - it's possible that the cycle collector is calling into random code...
(In reply to David :Bienvenu from comment #21)
> it's possible that the cycle collector is calling into random code...

That's what I suspected/hoped, by analogy to other mysterious crashes in SVG code, but that doesn't seem to be the case, from looking at the stack.  Very little cycle-collector code, and the first SVG call on the stack is a sensible one I think.  I don't immediately see where ~nsCOMPtr_base calls AddRef, but it seems reasonable that it might temporarily AddRef before destroying, to stabilize the refcount, or something.

Note that nsSVGMutationObserver::AddRef is a no-op (it's just "return 1"), so it could only be crashing if the nsSVGMutationObserver in question is a null/bogus/deleted pointer.

> If I had to guess, I would say that SVG is not unregistering its listeners

Looks like it could be something like that, yeah.
Actually, on closer inspection, there is some evidence of stack corruption.  When we enter nsObserverList::NotifyObservers, things turn crazy.  Consider |aTopic| and |someData| in the following two lines:
> xul!nsObserverList::NotifyObservers(class nsISupports * aSubject = 0x09a4eab0, char * aTopic = 0x00768044 "HQ???", wchar_t * someData = 0x59a83fb4 "?????")+0x44
> xul!nsObserverService::NotifyObservers(class nsISupports * aSubject = 0x00768044, char * aTopic = 0x59a83fb4 "xpcom-shutdown", wchar_t * someData = 0x00000000 "")+0x50

The nsObserverService call has sensible values for aTopic and someData, and it always passes its values down untouched AFAICT.  However, in this case it looks like nsObserverList ended up with different (junk) values for those arguments.

A few other observations:
 - nsObserverService::NotifyObservers seems to be calling into a JS implementation of an observer.  (its Observe call descends into glue code and then JS impl code)

 - The nsSVGMutationObserver's nsCOMPtr here is getting destructed inside of ~xpcObjectHelper, which only owns two nsCOMPtrs, only one of which holds something that nsSVGMutationObserver derives from -- "nsCOMPtr<nsISupports> mCanonicalStrong".  I don't really know anything about xpcObjectHelper, but it seems to be a way to wrap C++ objects for interaction with script, and it's very odd/unexpected we'd do that for a nsSVGMutationObserver...  So I wonder if this ends up being a case where the xpcObjectHelper ends up just pointing to a random location in memory, which in this case just happened to look like a nsSVGMutationObserver.
Henrik: if you could produce & attach any more stack traces like the one you've attached, that'd be helpful.  In particular, I'm curious whether this _always_ ends up in SVG code, or if it just ends up somewhere random. (per the end of my previous comment)
(Reporter)

Comment 25

6 years ago
Created attachment 572102 [details]
Stack trace 002
(Reporter)

Comment 26

6 years ago
Created attachment 572103 [details]
Stack trace 003
(Reporter)

Comment 27

6 years ago
Created attachment 572104 [details]
Stack trace 004
(Reporter)

Comment 28

6 years ago
I have added some more stack traces, produced by using this guide:
https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

Please let me know if any of the commands should be changed or added in WinDbg. As explained before I don't know what I'm doing, I just follow the guide.

Comment 29

6 years ago
Henrik, what windows do you have open before you shut down? Do you do a file exit from the 3 pane UI with the about window open?
(Reporter)

Comment 30

6 years ago
This is how I reproduce:
- Open Thunderbird
- Help -> About Thunderbird
- Close Help -> About Thunderbird Windows (red cross)
- Close Thunderbird (red cross)
Thanks for the additional stack traces! They appear to confirm that this isn't SVG-specific, as I suspected in Comment 24. (none of the new stacktraces have any SVG in the stack)

The last one (attachment 572104 [details]) is the most similar to the original stack trace -- it's got the same mysteriously-bogus-aTopic/someData values in nsObserverList::NotifyObservers, and the stack is the same up to the NativeInterface2JSObject() call (but it differs inside of that).

The other two new stack traces (attachment 572102 [details], attachment 572103 [details]) are simpler-looking, though -- in those ones, aTopic / someData are preserved intact (rather than having bogus values), and there's no JS observer implementation.
(Reporter)

Comment 32

6 years ago
The plot thickens.

I always wondered why the comments for these crashes looked like a lot of them was from Danish people.

By looking at the stack traces I found the following modules loaded:
ModLoad: 00000000`69240000 00000000`69387000   C:\Program Files\DanID\NemID\NemID_PKCS11.dll
ModLoad: 00000000`6fbc0000 00000000`6fbc7000   C:\Program Files\DanID\NemID\mingwm10.dll
ModLoad: 00000000`6a1c0000 00000000`6a463000   C:\Program Files\DanID\NemID\QtCore4.dll
ModLoad: 00000000`65100000 00000000`65bcb000   C:\Program Files\DanID\NemID\QtGui4.dll
ModLoad: 00000000`6ff00000 00000000`7005d000   C:\Program Files\DanID\NemID\QtNetwork4.dll

These are a part of a danish citizen certificate program. These modules is loaded as a "Security Device" in Thunderbird, because I wanted to use my certificate for secure e-mail.

Apparently, running Thunderbird in "safe mode" still loads this added "Security Device".

If I uninstall the program, or unload the "Security Device" in Thunderbird, the crashes stop. So is this a problem with the way Thunderbird adds the "Security Device" or a problem with the program from NemID.

You can find the program here:
https://www.nemid.nu/support/sikker_e-mail/i_gang_med_sikker_e-mail/hent_program/windows/

And a description on how to add this as a "Security Device" in Thunderbird here:
https://www.nemid.nu/support/sikker_e-mail/i_gang_med_sikker_e-mail/opsaetning_af_e-mail-program/windows_7/mozilla_thunderbird_3/

Unfortunately both links are in Danish, so Google Translate might help ;o)

Comment 33

6 years ago
This all feels like some extension and/or binary component isn't unregistering itself.


This particular stack segment is disturbing:

0021f87c 59f3728f xul!nsObserverList::NotifyObservers(class nsISupports * aSubject = 0x0a552b30, char * aTopic = 0x0056e044 "HQLZ4QLZ", wchar_t * someData = 0x5a283fb4 "?????")+0x44 [e:\buildbot\win32_build_release\build\mozilla\xpcom\ds\nsobserverlist.cpp @ 130]
0021f898 59f2fc12 xul!nsObserverService::NotifyObservers(class nsISupports * aSubject = 0x0056e044, char * aTopic = 0x5a283fb4 "xpcom-shutdown", wchar_t * someData = 0x00000000 "")+0x50 [e:\buildbot\win32_build_release\build\mozilla\xpcom\ds\nsobserverservice.cpp @ 185]

because aTopic seems to be getting crunched during one of the notifications; it starts of xpcom-shutdown and then becomes HQLZ4QLZ

Comment 34

6 years ago
yeah, security devices aren't extensions per se...

Comment 35

6 years ago
Kai might have some insight into this.

Comment 36

6 years ago
I strongly suspect it's some problem with the security device code, and contacting the author is the first place to start.
(Reporter)

Comment 37

6 years ago
It might be, but this problem started with Thunderbird 5.0, before this I didn't experience the crash.
It's common for binary Mozilla Add-ons to experience random issues after Thunderbird/Firefox is updated, if there's a change to a XPCOM API that's used by the Add-on.  In these cases, the Add-on needs to be recompiled in order to function correctly.

(I'm not sure whether this is true of security device DLLs in particular, but it seems likely that it is.)

Note that the last URL linked in comment 32 refers to "Thunderbird 3".  A lot has changed since then -- I'd actually be surprised if binary Add-ons that use APIs from Thunderbird 3 still worked flawlessly in Thunderbird >5.

Comment 39

6 years ago
I'm curious what aspect(s) of the security device code are Thunderbird-specific. Do they have a similar download for Firefox?
(Reporter)

Comment 40

6 years ago
The "Security Device" is needed in order to unlock the certificate issued by NemID. You can only use this certificate for e-mail signing and encryption, so the download is only for supported e-mail clients. You have to use a code card or code token to unlock the certificate for each use.

The certificate is not used for authentication in browsers. Instead a proprietary system currently running in java is used. The private key is stored at NemID servers, and you use a code card or code token to login. So in this case, you don't need to install any software.

Comment 41

6 years ago
ah, just curious if they had a newer version for Firefox. Quite a bit has changed in Thunderbird and core gecko code since TB 3, so binary components definitely need to be recompiled, and even some js components need tweaking.
(Reporter)

Comment 42

6 years ago
The "Security Device" was obviously created in TB3, and just stayed in the config since. The "Security Device" was not version checked, like the add-ons during upgrades, so I wasn't taking any notice of this.

I will contact NemID, and ask for an updated version of their software if any.
(Reporter)

Comment 43

6 years ago
This issue could be related to Bug 680714

Comment 44

5 years ago
I have the exact same problem as Henrik - and I, too, have the DanID module installed. My problems also began with TB5.

Let me know if you want me to submit any crash reports/any other info.

Updated

5 years ago
Whiteboard: [psm-smartcard]
Component: General → Security
QA Contact: general → thunderbird
Whiteboard: [psm-smartcard] → [psm-smartcard][gs]

Comment 45

5 years ago
I am experiencing the same bug. Is there any way I can help the developers in fixing this bug?
(In reply to Henrik Larsson from comment #43)
> This issue could be related to Bug 680714

Unfortunatelly it has stagnated in NSS

Comment 47

5 years ago
I always use latest nightlies (comm-central) and the x64 ones since they came out (I'm on Win7 x64).

I started having these annoying crashes (the application works just fine nevertheless) on close/restart with the few last 15.0a1 builds and continue to have these crashes till this day with the latest 16.0a1 x64. I managed to track it down to the May 20th build. I have no crash with the May 18th one (and there was no win x64 build on May 19th).

I've never reported this before till now because I figured it must be some regression that would be fixed soon (like the one where no mail folders were shown with the first 16.0a1 x64 build - it was solved with the very next build). Well, obviously it is still not solved and it got annoying enough to make me go searching for any related reported bugs. Hope I'm in the right place :)

I'm not getting the crash on a fresh (though empty - no accounts set) profile so this might be caused by an extension.

Here's the details from the event id 1000 I get in my Windows application log:

Faulting application name: thunderbird.exe, version: 16.0.0.4545, time stamp: 0x4fd5d7fe
Faulting module name: xul.dll_unloaded, version: 0.0.0.0, time stamp: 0x4fd5d770
Exception code: 0xc0000005
Fault offset: 0x000007fedd511bf0
Faulting process id: 0x15b8
Faulting application start time: 0x01cd481f5fe81439
Faulting application path: C:\Program Files\Mozilla\Thunderbird\thunderbird.exe
Faulting module path: xul.dll
Report Id: a5b07457-b412-11e1-a5c0-000df07e119d

Hope this info helps.

Comment 48

5 years ago
(In reply to klonos from comment #47)
> I always use latest nightlies (comm-central) and the x64 ones since they
> came out (I'm on Win7 x64).
> 
> I started having these annoying crashes (the application works just fine
> nevertheless) on close/restart with the few last 15.0a1 builds and continue
> to have these crashes till this day with the latest 16.0a1 x64. 
I rather think this is a different issue - do you have telemetry turned on? tools, options, advanced, general tab, submit performance info checkbox. I've found that unchecking that stops me from crashing on exit.

Comment 49

5 years ago
...you know what? I'd never have thought of that. You are a life saver David! Thanx.

Is there a bug report filed for that that I can follow btw? I'm asking so to know when this gets fixed so I can then turn the telemetry back on (I want to help as much as I can by providing feedback and that does it automatically).

...once again thanx!
Duplicate of this bug: 767954

Comment 51

5 years ago
I've had this issue since the x64 version came out, I'm not sure what is different but on the latest update to version 17 I am no longer getting crashes or windows error reporting when closing daily (No crash reports opening in background either)

Comment 52

5 years ago
I have just tested Daily thunderbird-17.0a1 X64 and Daily thunderbird-17.0a1 x32 on my Windows 7.
In the x64 version I get the following message 'Certificate Manager can't locate a valid certificate that can be used to digitally sign your messages' when I try to select a certificate in account settings.

The x32 version crashed when I exit Thunderbird.
Crash ID: bp-9691e630-dacd-4a66-a5f3-4dacc2120718
Crash ID: bp-cb970f1f-45a3-4e03-89c4-f7e3a2120718
Crash ID: bp-1dd1e0b9-0f99-488c-b842-6b0c42120718

Comment 53

5 years ago
well, this bug is for the x64 version crashing, I have no idea about what is wrong with your "certificates" (try doing a windows update? sometimes they release new ones), but it didn't sound like your x64 version crashed on version 17 either.

You should probably post about the x86 version crashing in another bug though.

Comment 54

5 years ago
Hi Yev

As you can see from the original poster Henrik Larsson's comment 32, the crashes are caused by a PKCS11.dll file from DanID. The crash ID is _beginthread.
Is your Thunderbird crashing with the same Crash ID?

Comment 55

5 years ago
Oh my apologies, didn't realize it was narrowed down to the pkcs11.dll file.

However, I can't confirm if that's what was causing it to crash because ever since I upgraded to version 17.0a1 it stopped crashing on exit. I just wanted to point that out because on every nightly with 16.0a1 it would crash 100% of the time. At some point version 16.0a1 stopped showing a message "Daily has crashed etc..." but the background process "WerFault" would run as soon as it was exited (Windows Error Reporting).

Anyways since version 17.0a1 there are no crashes and "WerFault" doesn't open up at all
Is the crash gone for you in newer versions?
And if not, please post crash ID. 
Thanks
Flags: needinfo?(mozilla)
Flags: needinfo?(mikkel)
Flags: needinfo?(joradan0)
Summary: Thunderbird crashes at exit (shutdown) _beginthread → Thunderbird crashes at exit (shutdown) _beginthread, related to DanID/PKCS11

Comment 57

a year ago
The company behind the Danish certificates has updated the Security Device (PKCS11.dl) at least twice since this bug was reported. With the first update the Crash Reporter didn't catch the crash anymore, but the crash was recorded by Windows Event Viewer. With the latest version (1.3.1.0) the problem has gone as far as I can see.
Flags: needinfo?(joradan0)

Comment 58

8 months ago
I have the same. It look, that this is not fixed. Everytime I send crash report via Crash reporter.

Only I know, that thunderbord must run for some time. If I open it and close immediately, it's ok.
You need to log in before you can comment on or make changes to this bug.