Closed Bug 1924623 Opened 25 days ago Closed 3 days ago

Startup Crash in [@ Detour_VariantClear]

Categories

(Thunderbird :: General, defect)

Thunderbird 132
Unspecified
Windows 11
defect

Tracking

(thunderbird_esr128+ affected, thunderbird133 affected)

RESOLVED FIXED
134 Branch
Tracking Status
thunderbird_esr128 + affected
thunderbird133 --- affected

People

(Reporter: wsmwk, Assigned: yannis, NeedInfo)

References

(Regression)

Details

(4 keywords)

Crash Data

Attachments

(3 files)

crashes start with 132 beta

Crash report: https://crash-stats.mozilla.org/report/index/7f7c61d0-cd88-4f82-9094-3b71f0241007

Reason:

EXCEPTION_ACCESS_VIOLATION_WRITE

Top 10 frames:

0  ?  @0x00007ffdeb4b042a
1  mso.dll  Detour_VariantClear(tagVARIANT*)
2  msctf.dll  QuickVariantClear(tagVARIANT*)
3  msctf.dll  CThreadInputMgr::_ApplyContextUrlToFocusedWindow(IInputContextPrivate*)
4  msctf.dll  CThreadInputMgr::_SetFocus(IDocumentInputManagerPrivate*, int)
5  msctf.dll  CThreadInputMgr::SetFocus(ITfDocumentMgr*)
6  xul.dll  mozilla::widget::TSFTextStore::CreateAndSetFocus(nsWindow*, mozilla::widget::...  widget/windows/TSFTextStore.cpp:5919
7  xul.dll  mozilla::widget::TSFTextStore::OnFocusChange(bool, nsWindow*, mozilla::widget...  widget/windows/TSFTextStore.cpp:5866
8  xul.dll  mozilla::widget::IMEHandler::NotifyIME(nsWindow*, mozilla::widget::IMENotific...  widget/windows/WinIMEHandler.cpp:278
9  xul.dll  mozilla::widget::TextEventDispatcher::NotifyIME(mozilla::widget::IMENotificat...  widget/TextEventDispatcher.cpp:487

This has become #2 crash for 128.4.0esr. And it starts with 132 beta and 128.4.0.

All are Windows, 64bit. There appear to be two types of crashes, startup and not-startup.

The vast majority are startup, and most of those less than 10 seconds uptime:

bp-415a6e01-09bf-44b4-a1ff-6dca50241101 is an example outlier where the reporter states "Tried to import Outlook PST file. Thunderbird crashed as soon as I selected the option to import the Outlook file."

Severity: -- → S2
Flags: needinfo?(toby)
Summary: Crash in [@ Detour_VariantClear] → Startup Crash in [@ Detour_VariantClear]
Flags: needinfo?(toby)

Too many coincidences here -- definitely caused by bug 1920643. Most likely, mso.dll needs to detour VariantClear and fails to do so because we already detoured it. Let's make the code from that patch Firefox-only.

Regressed by: 1920643

A TB user pointed this bug out and it's spiking quite a bit so I decided to take a look. For starters this is not happening in our code, it's happening deep into Microsoft code and in particular when we enter mso.dll which IIRC is part of Office. The stack winds up to this point in our code, where we call a function we pulled out of of MAPI32.DLL or MAPI32BAK.DLL. I had a look at the log for that file but nothing stands out.

There appear to be two different type of crashes as mentioned in comment 3 but the root cause seems the same, we're calling the same Microsoft function and it's crashing in the same way. So I wonder if the users affected by the startup crash have imported things from Outlook in the past, and that set some condition that triggers the crash.

Since I don't really understand the root cause of this I cannot suggest a fix, but I can suggest a workaround. Placing a structured exception handler around the call to the external library and swallowing the exception would save the users a crash, but maybe leave them with something non-functional.

Oh, I see Yannis is on this, well the problem is likely going to be fixed soon.

Detouring VariantClear resulted in a huge crash spike for Thunderbird,
so let's do that only in Firefox.

Assignee: nobody → yjuglaret
Status: NEW → ASSIGNED

The Thunderbird user from comment 3 confirmed that replacing their mozglue.dll 132.0.0.2335 by a patched version where I made WindowsOleAut32Initialization() a no-op fixed the issue for them, confirming the link to bug 1920643.

Attachment #9435377 - Attachment description: Bug 1924623 - Only detour VariantClear in Firefox. r=gstoll,#win-reviewers → Bug 1924623 - Only detour VariantClear in Firefox. r=gstoll,#win-reviewers!,#thunderbird-build-system-reviewers!
Pushed by yjuglaret@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/025c089998e3 Only detour VariantClear in Firefox. r=gstoll,win-reviewers
Status: ASSIGNED → RESOLVED
Closed: 3 days ago
Resolution: --- → FIXED
Target Milestone: --- → 134 Branch

Thanks for taking care Yannis!

The crash comments make it clear this would, mainly, happen when trying to import from Outlook. Ramona, does that reproduce on Windows?
I think the alternative other triggering factor (not mentioned there) is having enabled the windows addressbook which is disabled unless you fiddle with hidden prefs. That would explain the startup crashes.

Flags: needinfo?(ramona)

Looking at today's Nightly builds, everything looks as expected (details below). I will propose a beta uplift now. I just have to proceed as if the patch were for Firefox, and then Thunderbird will receive it too -- is that correct?

Thunderbird 128.4.1esr -- calls mozilla::WindowsOleAut32Initialization (bad):

xul!XREMain::XRE_main [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5829]:
00007ffa`eaf3c4c0 4156            push    r14
...
00007ffa`eaf3c962 ff1540fcae03    call    qword ptr [xul!_imp_?WindowsBCryptInitializationmozillaYA_NXZ (00007ffa`eea2c5a8)]
00007ffa`eaf3c968 ff1542fcae03    call    qword ptr [xul!_imp_?WindowsMsctfInitializationmozillaYA_NXZ (00007ffa`eea2c5b0)]
00007ffa`eaf3c96e ff1544fcae03    call    qword ptr [xul!_imp_?WindowsOleAut32InitializationmozillaYA_NXZ (00007ffa`eea2c5b8)]
00007ffa`eaf3c974 488d4c2424      lea     rcx,[rsp+24h]
00007ffa`eaf3c979 e862c0c7fc      call    xul!mozilla::mscom::ProcessRuntime::ProcessRuntime (00007ffa`e7bb89e0)
00007ffa`eaf3c97e c644242300      mov     byte ptr [rsp+23h],0
00007ffa`eaf3c983 488d542423      lea     rdx,[rsp+23h]
00007ffa`eaf3c988 4889f1          mov     rcx,rsi
00007ffa`eaf3c98b e8a070ffff      call    xul!XREMain::XRE_mainInit (00007ffa`eaf33a30)
...

Thunderbird Nightly -- does not call mozilla::WindowsOleAut32Initialization (good):

xul!XREMain::XRE_main [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5829]:
00007ffa`c5c5bf10 4156            push    r14
...
00007ffa`c5c5c3b2 ff151825df03    call    qword ptr [xul!_imp_?WindowsBCryptInitializationmozillaYA_NXZ (00007ffa`c9a4e8d0)]
00007ffa`c5c5c3b8 ff151a25df03    call    qword ptr [xul!_imp_?WindowsMsctfInitializationmozillaYA_NXZ (00007ffa`c9a4e8d8)]
00007ffa`c5c5c3be e8cd5816fc      call    xul!mozilla::IOInterposer::Init (00007ffa`c1dc1c90)
00007ffa`c5c5c3c3 488d4c2424      lea     rcx,[rsp+24h]
00007ffa`c5c5c3c8 e8538089fc      call    xul!mozilla::mscom::ProcessRuntime::ProcessRuntime (00007ffa`c24f4420)
00007ffa`c5c5c3cd c644242300      mov     byte ptr [rsp+23h],0
00007ffa`c5c5c3d2 488d542423      lea     rdx,[rsp+23h]
00007ffa`c5c5c3d7 4889f1          mov     rcx,rsi
00007ffa`c5c5c3da e87178ffff      call    xul!XREMain::XRE_mainInit (00007ffa`c5c53c50)
...

Firefox Nightly -- still calls mozilla::WindowsOleAut32Initialization (good):

xul!XREMain::XRE_main [/builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp @ 5829]:
00007ffb`1c93e110 4156            push    r14
...
00007ffb`1c93e36a ff1510de2c05    call    qword ptr [xul!_imp_?WindowsBCryptInitializationmozillaYA_NXZ (00007ffb`21c0c180)]
00007ffb`1c93e370 ff1512de2c05    call    qword ptr [xul!_imp_?WindowsMsctfInitializationmozillaYA_NXZ (00007ffb`21c0c188)]
00007ffb`1c93e376 ff1514de2c05    call    qword ptr [xul!_imp_?WindowsOleAut32InitializationmozillaYA_NXZ (00007ffb`21c0c190)]
00007ffb`1c93e37c e86fc88bfd      call    xul!mozilla::IOInterposer::Init (00007ffb`1a1fabf0)
00007ffb`1c93e381 488d4c2434      lea     rcx,[rsp+34h]
00007ffb`1c93e386 e8559f91fd      call    xul!mozilla::mscom::ProcessRuntime::ProcessRuntime (00007ffb`1a2582e0)
...

Detouring VariantClear resulted in a huge crash spike for Thunderbird,
so let's do that only in Firefox.

Original Revision: https://phabricator.services.mozilla.com/D227957

Attachment #9435889 - Flags: approval-mozilla-beta?

beta Uplift Approval Request

  • User impact if declined: Startup (or regular) crash for Thunderbird users.
  • Code covered by automated testing: yes
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • Steps to reproduce for manual QE testing: -
  • Risk associated with taking this patch: Low
  • Explanation of risk level: Opts Thunderbird out of instrumentation that was only present to address bug 1920643, where the concerning crash volume was coming from Firefox. A user has confirmed the link between the crash and the instrumentation. Removing the instrumentation fixed the crash for them.
  • String changes made/needed: no
  • Is Android affected?: no

(In reply to Yannis Juglaret [:yannis] from comment #10)

Looking at today's Nightly builds, everything looks as expected (details below). I will propose a beta uplift now. I just have to proceed as if the patch were for Firefox, and then Thunderbird will receive it too -- is that correct?

Correct!

Attachment #9435889 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Detouring VariantClear resulted in a huge crash spike for Thunderbird,
so let's do that only in Firefox.

Original Revision: https://phabricator.services.mozilla.com/D227957

Attachment #9436113 - Flags: approval-mozilla-esr128?

esr128 Uplift Approval Request

  • User impact if declined: Startup (or regular) crash for Thunderbird users.
  • Code covered by automated testing: yes
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • Steps to reproduce for manual QE testing: -
  • Risk associated with taking this patch: Low
  • Explanation of risk level: Opts Thunderbird out of instrumentation that was only present to address bug 1920643, where the concerning crash volume was coming from Firefox. A user has confirmed the link between the crash and the instrumentation. Removing the instrumentation fixed the crash for them.
  • String changes made/needed: no
  • Is Android affected?: no
Attachment #9436113 - Flags: approval-mozilla-esr128? → approval-mozilla-esr128+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: