2037923 - Crash in [@ shutdownhang | mozilla::scache::StartupCache::GetBuffer]

Status: RESOLVED FIXED

(Core :: XPCOM, defect, P2)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Crash report: https://crash-stats.mozilla.org/report/index/a4ff4dfa-900c-434a-83cf-24daa0260507

MOZ_CRASH Reason:

Shutdown hanging at step XPCOMWillShutdown. Something is blocking the main-thread.

main thread:

ZwWaitForAlertByThreadId	
RtlAcquireSRWLockExclusive	
mozilla::OffTheBooksMutex::Lock()	xpcom/threads/Mutex.h:64
mozilla::detail::BaseAutoLock<mozilla::Mutex&>::BaseAutoLock(mozilla::Mutex&)	xpcom/threads/Mutex.h:159
mozilla::scache::StartupCache::GetBuffer(char const*, char const**, unsigned int*)	startupcache/StartupCache.cpp:392
ReadCachedStencil(mozilla::scache::StartupCache*, nsTSubstring<char>&, JSContext*, JS::ReadOnlyDecodeOptions const&, js::frontend::InitialStencilAndDelazifications**)	js/xpconnect/loader/mozJSLoaderUtils.cpp:44
mozJSModuleLoader::GetScriptForLocation(JSContext*, ModuleLoaderInfo&, nsIFile*, bool, JS::MutableHandle<JSObject*>, char**)	js/xpconnect/loader/mozJSModuleLoader.cpp:889
mozJSModuleLoader::LoadSingleModule(mozilla::loader::SyncModuleLoader*, JSContext*, JS::loader::ModuleLoadRequest*, JS::MutableHandle<JSObject*>)	js/xpconnect/loader/mozJSModuleLoader.cpp:725
mozilla::loader::SyncModuleLoader::StartFetch(JS::loader::ModuleLoadRequest*)	js/xpconnect/loader/SyncModuleLoader.cpp:146
JS::loader::ModuleLoaderBase::StartOrRestartModuleLoad(JS::loader::ModuleLoadRequest*, JS::loader::ModuleLoaderBase::RestartRequest)	js/loader/ModuleLoaderBase.cpp:601 

Top 9 frames:

0  xul.dll  MOZ_Crash(char const*, int, char const*)  mfbt/Assertions.h:402
0  xul.dll  mozilla::(anonymous namespace)::RunWatchdog(void*)  toolkit/components/terminator/nsTerminator.cpp:238
1  nss3.dll  _PR_NativeRunThread(void*)  nsprpub/pr/src/threads/combined/pruthr.c:381
2  nss3.dll  pr_root(void*)  nsprpub/pr/src/md/windows/w95thred.c:129
3  ucrtbase.dll  thread_start<unsigned int (__cdecl*)(void*), 1>
4  kernel32.dll  BaseThreadInitThunk
5  mozglue.dll  mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mo...  toolkit/xre/dllservices/mozglue/nsWindowsDllInterceptor.h:142
5  mozglue.dll  patched_BaseThreadInitThunk(int, void*, void*)  toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:564
6  ntdll.dll  RtlUserThreadStart

Comment 1

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 20 desktop browser crashes on beta
• Top 5 desktop browser crashes on Windows on beta

:gtonietto, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 2

[Gustavo Tonietto (he/him) [:gtonietto] [:tonietto]]

Keeping it S3 for now.

Comment 3

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 20 desktop browser crashes on beta (startup)
• Top 5 desktop browser crashes on Windows on beta (startup)

For more information, please visit BugBot documentation.

Comment 4

[Pascal Chevrel (relman team) -> :pascalc]

Setting as tracking+ across all releases, this crash spiked in 152 beta and is now crashing in 152 release

Comment 5

[Donal Meehan [:dmeehan]]

:lars, Claude thinks Bug 2026686 is the potential regressor here. It may be incorrect, but please take a look:

What the patch does: wraps the StartupCache background write thread with nsAutoLowPriorityIO, which on Windows means
SetThreadPriority(THREAD_MODE_BACKGROUND_BEGIN) — background I/O priority — held while the thread is also holding mTableLock.

Why it hangs at StartupCache::GetBuffer during shutdown:

1. Shutdown saturates the disk with normal-priority I/O (sessionstore, places, cookies, dom storage, telemetry, etc.).
2. The background write thread is mid-WriteToDisk, holding mTableLock, but its I/O is now Windows background-tier — the
   scheduler can defer it for many seconds while foreground writes go through.
3. Late in shutdown, a lazy ESM getter fires → ImportESModule → SyncModuleLoader::StartFetch → ReadCachedStencil →
   StartupCache::GetBuffer → tries to acquire mTableLock → blocked.
4. The watchdog's AppShutdownConfirmed timeout fires before the throttled write releases the lock → crash with shutdownhang |
   StartupCache::GetBuffer.

Comment 6

[Lars Eggert [:lars]]

I agree with the analysis.

Comment 7

[Lars Eggert [:lars]]

Attachment: Bug 2037923 - Don't use low-priority I/O for startup cache writes during shutdown r=tcampbell,mccr8,#xpcom-reviewers

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a316c0cc866b
https://hg.mozilla.org/integration/autoland/rev/601037445071
Don't use low-priority I/O for startup cache writes during shutdown r=xpcom-reviewers,emilio

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Any concerns with me reverting https://github.com/mozilla-firefox/firefox/commit/0af8f0534bd5 for a 152 dot release, assuming that'd be a sufficient mitigation?

Comment 10

[Lars Eggert [:lars]]

Nope, that should be a safe fix.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Revert pushed for 152.0.1.

Comment 12

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/f0b29c979228
https://hg.mozilla.org/releases/mozilla-release/rev/a5b8930b576c

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/601037445071

Comment 14

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:leggert, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox153 to wontfix.

For more information, please visit BugBot documentation.

Comment 15

[Lars Eggert [:lars]]

:RyanVM, does this need an uplift, since you had a different fix for the point release?

Comment 16

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: Fixes topcrash on Windows
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: Simple fix
• String changes made/needed?: No
• Is Android affected?: no

Comment 17

[Lars Eggert [:lars]]

Attachment: Bug 2037923 - Don't use low-priority I/O for startup cache writes during shutdown

Original Revision: https://phabricator.services.mozilla.com/D307210

Comment 18

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/e7588c14c0f6
https://hg.mozilla.org/releases/mozilla-beta/rev/f833c8897ca0