1883040 - At high CPU load, the main thread can get stuck setting its QoS

Status: RESOLVED DUPLICATE of bug 1876306

(Core :: XPCOM, defect)

Description

[Kris Wright :KrisWright]

I've managed to reproduce an extremely specific hang scenario on our MacOS QoS work. To create this hang, the system must be operating at long-term very high load with extremely busy threads spinning at the highest priority. Due to the resource starvation, lower priority threads than the |User-interactive| level get scheduled >60 seconds to never from the time they receive work.

This is how I reproduce the hang and see the results:

• Spin up a bunch of high-priority threads running busy work (I use sqrt() on 50-100 at the |user-interactive| level)
• Wait for machine to start lagging behind, as it's trying to run all of this high-priority work
• Put a Firefox tab in the background by tabbing out
• Either wait, or just initiate a crash.
• You will see it’s never left RecvSetMainThreadQoS

In this scenario it looks like the main thread gets stuck setting itself. While this scenario is very rare and I had to make a conscious effort to produce it, I still would like to plug this hole. This is as easy as not setting an override when the intended priority is background-level, as we aren't dedicated to the timeliness of setting this priority. We only want to use the override to warm up the main thread.

Comment 1

[Kris Wright :KrisWright]

Attachment: Bug 1883040 - Don't set low-priority QoS overrides. r?nika

Ensures we don't override a high-priority thread with a low priority. The only time we really need to set an override is for low priority threads moving into a high priority.

Comment 2

[Kris Wright :KrisWright]

On further review, it appears that macos QoS.h already fulfills this role. Resolving this bug.

Comment 3

[Kris Wright :KrisWright]

Actually - reopening this, this is still an issue, my running theory was wrong.

Comment 4

[Kris Wright :KrisWright]

Attachment: Bug 1883040 - Always set QoS overrides at the highest priority.

In looking for ways to set the main thread in a timely fashion, I realized we can speed up the process at high CPU load by using the override to have the main thread set itself while only in the highest possible priority. The intention is so that it finishes the RecvSetMainThreadQosPriority task before going into the background, which based on this process should ensure we don't get stuck here:

• Spin up a bunch of high-priority threads running busy work (I use sqrt() on 50-100 at the |user-interactive| level)
• Wait for machine to start lagging behind, as it's trying to run all of this high-priority work
• Put a Firefox tab in the background by tabbing out
• Initiate a crash
• You will see it’s never left RecvSetMainThreadQoS

Comment 5

[Kris Wright :KrisWright]

After discussions with Nika via matrix, I realized I'm looking at the same issue as bug 1876306.