1584377 - Crash in [@ shutdownhang | nsThread::Shutdown | mozilla::LazyIdleThread::ShutdownThread]

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Telemetry, defect, P1)

Description

[(No longer employed by Mozilla) Aaron Klotz]

This bug is for crash report bp-1dc8f833-02fc-459c-b147-10b520190926.

Top 10 frames of crashing thread:

0 ntdll.dll NtWaitForAlertByThreadId 
1 ntdll.dll RtlSleepConditionVariableSRW 
2 kernelbase.dll SleepConditionVariableSRW 
3 mozglue.dll mozilla::detail::ConditionVariableImpl::wait mozglue/misc/ConditionVariable_windows.cpp:50
4 xul.dll struct already_AddRefed<nsIRunnable> mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue>::GetEvent xpcom/threads/ThreadEventQueue.cpp:153
5 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1134
6 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:486
7 xul.dll nsThread::Shutdown xpcom/threads/nsThread.cpp:900
8 xul.dll mozilla::LazyIdleThread::ShutdownThread xpcom/threads/LazyIdleThread.cpp:276
9 xul.dll mozilla::LazyIdleThread::Shutdown xpcom/threads/LazyIdleThread.cpp:465

This is because the background thread is running at low priority. The main thread is blocked on it during shutdown (but not truly blocked from a kernel perspective, as the main thread continues to spin its event loop; the kernel doesn't "see" the dependency between the main thread and the background thread, so priority inheritance can't kick in).

The current method that we're using for dropping priority is by calling SetThreadPriority with THREAD_MODE_BACKGROUND_BEGIN. Unfortunately that technique also requires that THREAD_MODE_BACKGROUND_END be called on its own thread; a different thread cannot raise the background thread out of that mode using that flag.

We should investigate how the older constants such as THREAD_PRIORITY_NORMAL affect a thread in background mode. If they have no effect then untrusted modules may need to THREAD_PRIORITY_IDLE/ THREAD_PRIORITY_NORMAL instead so that the main thread can bump the background thread's priority during shutdown.

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1584377: Raise untrusted module processing thread priority to normal upon shutdown notification; r=mhowell!

To ensure that we do not block shutdown while waiting on a low-priority
untrusted modules thread:

• We switch to using THREAD_PRIORITY_IDLE and THREAD_PRIORITY_NORMAL
  instead of THREAD_MODE_BACKGROUND_BEGIN and THREAD_MODE_BACKGROUND_END.
  This is necessary because the latter flags only work when settting priority
  on the current thread; they do not permit other threads to alter them.

• We add BackgroundPriorityRegion::Clear to normalize the specified thread's
  priority.

• We invoke BackgroundPriorityRegion::Clear on the untrusted modules thread
  as soon as we learn that we will be shutting down, thus boosting the thread's
  priority to normal.

Comment 2

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea790ebd201f46a91181e282eac4f28524bb162e

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f728e62d403ddc50ba7aa6533a86df2cca2785f1

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/853ea993c4f5
Raise untrusted module processing thread priority to normal upon shutdown notification; r=mhowell

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/853ea993c4f5

Comment 6

[Florin Strugariu [:Bebe]]

== Change summary for alert #23300 (as of Tue, 01 Oct 2019 10:43:07 GMT) ==

Improvements:

24% tp5n main_startup_fileio windows10-64-shippable-qr opt e10s stylo 773,351.00 -> 585,020.33
9% tp5n nonmain_normal_fileio windows10-64-shippable-qr opt e10s stylo 521,751,962.62 -> 473,831,840.92
9% tp5n nonmain_normal_fileio windows10-64-shippable-qr opt e10s stylo 527,583,744.00 -> 481,135,167.92
7% tp5n nonmain_normal_fileio windows7-32-shippable opt e10s stylo 425,244,152.42 -> 394,528,221.83

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23300