1773392 - Startup Crash in [@ TLSAddToMap] with Windows 11.

Status: RESOLVED FIXED

(Thunderbird :: General, defect, P1)

Description

[Wayne Mery (:wsmwk)]

signature first occurs with 103.0a1 buildid 20220603100303

Crashing in Windows code, but still filing this because it is a major new signature - only happening on daily.

Crash report: https://crash-stats.mozilla.org/report/index/1fbfe5a3-740a-485d-84f8-e80730220608

Reason: EXCEPTION_ACCESS_VIOLATION_READ

Top 10 frames of crashing thread:

0 combase.dll TLSAddToMap onecore\com\combase\tls\tls.cxx:177
1 combase.dll TLSPreallocateData onecore\com\combase\tls\tls.cxx:378
2 combase.dll CObjectContext::InternalContextCallback onecore\com\combase\dcomrem\context.cxx:4267
3 combase.dll CAgileReferenceMarshaled::~CAgileReferenceMarshaled onecore\com\combase\dcomrem\giptbl.cxx:1931
4 combase.dll CAgileReferenceMarshaled::`scalar deleting destructor' 
5 combase.dll Microsoft::WRL::Details::RuntimeClassImpl<Microsoft::WRL::RuntimeClassFlags<2>, 1, 0, 0, IAgileReference, Microsoft::WRL::FtmBase>::Release onecore\external\sdk\inc\wrl\implements.h:1627
6 windows.ui.dll unsigned long Microsoft::WRL::ComPtr<struct Windows::Foundation::IAsyncOperationCompletedHandler<bool> >::InternalRelease 
7 windows.ui.dll virtual void* Microsoft::WRL::Details::DelegateArgTraits<long  
8 windows.ui.dll virtual unsigned long Microsoft::WRL::Details::RuntimeClassImpl<struct Microsoft::WRL::RuntimeClassFlags<2>, 1, 0, 0, struct Microsoft::WRL::Implements<struct Microsoft::WRL::RuntimeClassFlags<2>, struct Windows::Foundation::IEventHandler<struct IInspectable*>, class Microsoft::WRL::FtmBase> >::Release 
9 windows.ui.dll unsigned long Microsoft::WRL::ComPtr<struct IUnknown>::InternalRelease 

Comment 1

[Wayne Mery (:wsmwk)]

#2 crash for beta 102.
I don't see anything interesting in the modules list.
~70% of crashes <30 seconds uptime

bp-bfe807c4-6694-48a8-a32d-a513a0220619

Comment 2

[Wayne Mery (:wsmwk)]

m_kato, any idea what this might be? If not, who to ask?

Comment 3

[Makoto Kato [:m_kato]]

I don't know. But is anything COM object leaked since this doesn't occurs on Firefox?

Comment 4

[Wayne Mery (:wsmwk)]

Still #2 crash for v102

The origins are not 100% clear, but based on crash history it would seem that 103 crashes came first, but then 102 crashes must have started from an uplift, first only one crash in 102.0b3, none in 102.0b4, and then in a big way with 102.0b5.

• 102.0b3 uplifts
• 102.0b5 uplifts

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Makoto Kato [:m_kato] from comment #3)

I don't know. But is anything COM object leaked since this doesn't occurs on Firefox?

#1 crash for the release - at double the crash rate of our next largest crash. Plus a startup crash.

So this deserves quick attention.

Comment 7

[Wayne Mery (:wsmwk)]

reporter from bug 1777639 is specifying profile name via profile manager at startup.

Comment 8

[Ryan Lee Sipes]

Andrei, can you see to it that someone takes a look into this? (Maybe get on the horn with BenC?)

Comment 10

[Wayne Mery (:wsmwk)]

Going to be disabling updates to 102 because of this and another bug.

Comment 11

[Kai Engert (:KaiE:)]

(In reply to Wayne Mery (:wsmwk) from comment #4)

The origins are not 100% clear, but based on crash history it would seem that 103 crashes came first, but then 102 crashes must have started from an uplift, first only one crash in 102.0b3, none in 102.0b4, and then in a big way with 102.0b5.

I think we'd rather have to look at uplifts to core gecko code, the crash doesn't seem related to Thunderbird specific code.

Comment 12

[Kai Engert (:KaiE:)]

Hello all reporters of this bug:

Is my understanding correct, that this crash report doesn't stop you from using Thunderbird?
Thunderbird starts up anyway, and all its functionality works fine?

Comment 13

[duanemattern]

"Thunderbird starts up anyway, and all its functionality works fine" .... I've not exercised the full functionality, but for my normal day-to-day use, yes, that has been my experience and I am still using Thunderbird.

Comment 14

[Kai Engert (:KaiE:)]

Reporters of this bug:

Can you please make an experiment? Please set an advanced pref to disable some media key integration. (I don't understand exactly what functionality this enables/disables, it seems like some kind of control to medias played in thunderbird with system keys.)

Please open Thunderbird settings, and the "config editor". Confirm the scary warning, if you're willing to. In the tab that opens, search for the following: media.hardwaremediakeys.enabled

You should see one result line, and after that text, it should show the text "true".

Double click that line, and the text should switch to "false".

Restart Thunderbird. Does that stop the crash?

Comment 15

[Andrey Marinov]

I've been redirected from bug 1777777 here as it was considered a duplicate of bug 1773392.
No, setting media.hardwaremediakeys.enabled to false doesn't stop the bug after restart. It only delays the crash dialog to appear after the application window appears.

Comment 16

[Kai Engert (:KaiE:)]

Thanks Andrey for this test. That's an interesting result. It would be interesting to know the crash ID of your new crash, as it might be a different cause.

Are you using any special media hardware connected to your computer? Maybe some virtual reality device?

Comment 17

[Andrey Marinov]

(In reply to Kai Engert (:KaiE:) from comment #16)

Thanks Andrey for this test. That's an interesting result. It would be interesting to know the crash ID of your new crash, as it might be a different cause.

Are you using any special media hardware connected to your computer? Maybe some virtual reality device?

I have the bug on 3 different machines with 3 independent installations. On each of them I am using /profilemanager switch to start Thunderbird.
I didn't see "profilemanager" in bug 1773392 and I don't understand why bug 1777777 was redirected, here since it is a problem of 102.0 over 91.11 and nothing to do with any media…

Comment 18

[Kai Engert (:KaiE:)]

(In reply to Andrey Marinov from comment #17)

I don't understand why bug 1777777 was redirected

Based on the crash report, we can see that the cause of the crash APPEARS to be the same, so unless we have other evidence we're collecting information in one place.

Comment 19

[Andrey Marinov]

(In reply to Kai Engert (:KaiE:) from comment #18)

(In reply to Andrey Marinov from comment #17)

I don't understand why bug 1777777 was redirected

Based on the crash report, we can see that the cause of the crash APPEARS to be the same, so unless we have other evidence we're collecting information in one place.

I respect.

Comment 20

[Andrey Marinov]

(In reply to Kai Engert (:KaiE:) from comment #16)

Thanks Andrey for this test. That's an interesting result. It would be interesting to know the crash ID of your new crash, as it might be a different cause.

Are you using any special media hardware connected to your computer? Maybe some virtual reality device?

https://crash-stats.mozilla.org/report/index/c1550b3c-a34b-4ef0-9867-4dd3d0220705

Comment 21

[Andrey Marinov]

It continues with 104.0a1 (2022-07-05) (32-bit)
https://crash-stats.mozilla.org/report/index/c1eca01b-0715-428c-9024-f9bb90220705

Comment 22

[Kai Engert (:KaiE:)]

(In reply to Andrey Marinov from comment #20)

It would be interesting to know the crash ID of your new crash, as it might be a different cause.
https://crash-stats.mozilla.org/report/index/c1550b3c-a34b-4ef0-9867-4dd3d0220705

Thanks. Looks like it's still the same cause of the crash.

Comment 23

[Kai Engert (:KaiE:)]

In bug 1777639 duanemattern said:

If I continue in Troubleshoot mode I don't get a crash notification.

Could you please try to turn off "hardware acceleration"? You can find that in Thunderbird Settings, General, then scroll all the way down.
Does that fix the crash in normal mode?

Comment 24

[Kai Engert (:KaiE:)]

Are you using a non-standard Thunderbird theme?
If yes, could you please try to switch back to the default theme?

Can you please try to disable all installed extensions?

(My suggestions are inspired are based on https://support.mozilla.org/en-US/kb/troubleshoot-extensions-themes-to-fix-problems )

Comment 25

[duanemattern]

Disabling hardware acceleration made no difference for me. Still get Crash Reporter upon start after profile selected.
I don't have any extensions installed.
I use the default theme.
Restarting in troubleshooting mode does not generate a crash report, nor does clicking the two checkboxes in troubleshoot mode and restarting, however, a restart does not go through the "Choose User Profile" dialog box.
If I select Exit from the "Choose User Profile", I still get a crash report.
If I select the "Use the selected profile without asking at startup", then start Thunderbird normally, without a "-p" command line option, it starts without a crash. So it seems to be an issue handing off from the "Choose User Profile" dialog box.

The Windows Management Event Viewer records the "Crash" as an Application Error with "combase.dll"
Faulting application name: thunderbird.exe, version: 102.0.0.0, time stamp: 0x62ba4026
Faulting module name: combase.dll, version: 10.0.22000.778, time stamp: 0x61943c4d
Exception code: 0xc0000005
Fault offset: 0x0000000000034393
Faulting process id: 0x39f8
Faulting application start time: 0x01d8915d9cbe0330
Faulting application path: C:\Program Files\Mozilla Thunderbird\thunderbird.exe
Faulting module path: C:\WINDOWS\System32\combase.dll
Report Id: f91908a0-d59c-494e-8efa-cad8bf184287

In case there was a problem with combase.dll, I ran: sfc /scannow
then DISM /Online /Cleanup-Image /RestoreHealth as Admin and re-ran Thunderbird. No change, same crash.

Comment 26

[Kai Engert (:KaiE:)]

I've setup a build and debug environment on Windows 11.

I was able to get the exact location of the crash:
https://searchfox.org/mozilla-esr102/rev/80d08affb82a28712b64173c07c12b9772e5112a/widget/windows/WindowsUIUtils.cpp#339

Comment 27

[Kai Engert (:KaiE:)]

That line in Mozilla code isn't sufficient to see what's happening.

I see two more call stack levels, which go into Windows SDK headers, so I can see more details.

It's in the cleanup/unloading stage. It's calling a ComPtr destructor, file client.h, reaching function InternalRelease(). It crashes on temp->Release().

The pointer is non-null. This suggests it's a a bad pointer (e.g. corrupted memory, or already freed memory).

Comment 28

[Kai Engert (:KaiE:)]

Well actually, it's not that simple. After the call levels I described above, it goes on and calls into several additional levels of Windows specific code - as can be seen in the crashreport stack.

But at least we now have a clue what Mozilla code is related to the crash. I can try to look deeper tomorrow.

Comment 29

[Kai Engert (:KaiE:)]

Because the crash is in Mozilla core code, we'll likely need a fix uplifted to mozilla-esr102, so I've filed a separate bug to track those changes:
bug 1778444

Comment 30

[Wayne Mery (:wsmwk)]

Kai, nice digging! Thanks for jumping in.

Comment 31

[Magnus Melin [:mkmelin]]

Should this be duped to bug 1778444 then?

Comment 32

[Kai Engert (:KaiE:)]

Magnus, the bugs have different components.
This bug is for tracking whether the fix has arrived in thunderbird.

Comment 33

[Magnus Melin [:mkmelin]]

Yes, but since bug 1778444 landed, it has arrived in Thunderbird (daily). Or is there more work expected?

Comment 35

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin [:mkmelin] from comment #31)

Should this be duped to bug 1778444 then?

For ease of tracking and generating our release notes, I think it is preferable that we don't dup big issues into m-c.
When we confirm via testing or crash-stats that it is fixed in Thunderbird, then we can mark this fixed.

Comment 36

[Wayne Mery (:wsmwk)]

m-c patch landed 2022-07-07 and beta landed 2022-07-07 (but we haven't yet built from that).
Nightly Thunderbird looks promising. But it needs more days before declaring victory, and beta a week from now will be the real benchmark.

Comment 37

[Wayne Mery (:wsmwk)]

Confirmed fixed in beta - gone in Thunderbird 103.0b5. Good for a release note. And then also for 102.0.3.

Comment 38

[Rob Lemley [:rjl]]

Added to 103.0beta5 notes.

(In reply to Wayne Mery (:wsmwk) from comment #37)

Confirmed fixed in beta - gone in Thunderbird 103.0b5. Good for a release note. And then also for 102.0.3.

Firefox has this fixed in 102.1.0, so it will not get picked up for Thunderbird for 102.0.3.

Comment 39

[Rob Lemley [:rjl]]

This will be fixed in 102.0.3, the fix is on the Firefox 102.0 release branch now.