Follow-up on the idle-at-shutdown bucket from the previous comment (821 of 1814 sampled). I pulled 20 idle-main-thread crash dumps from the last 14 days and walked every thread on every dump (~640 thread-instances) looking for any sign that the child had received or begun processing shutdown. The sample is small, but the pattern looks consistent: - No frame matching RecvShutdown / ShutdownInternal / XPCOMShutdown / Quit / any shutdown-observer call appeared on any thread of any of these 20 dumps. - ShutdownProgress, AsyncShutdownTimeout, MozCrashReason are all null on these dumps. The only shutdown-related field is the synthetic ipc_channel_error: "ShutDownKill" that the parent's KillHard path sets. - On every sampled dump, every thread appears parked on a Windows kernel wait (ZwWaitForAlertByThreadId on the worker pool, NtRemoveIoCompletion on IPC I/O Child / AudioIPC, NtWaitForSingleObject/MultipleObjects elsewhere). On the dumps I checked the IPC I/O Child had nothing pending in its completion port. If that pattern holds more broadly, these crashes seem unlikely to be "child stuck on something." They look more like the child being fully quiescent at the moment the parent terminates it, apparently before the shutdown message has been picked off the IO completion port - and the minidump is captured by ContentParent::KillHard immediately before TerminateProcess. System demographics for the 20 sampled (sorted by uptime): uptime avail_mb total_mb cpu os 31 202 3499 2 Win11 34 569 8090 4 Win10 138 1942 7971 4 Win11 149 22487 40894 6 Win11 245 569 16239 4 Win10 469 1104 16070 12 Win11 1048 4981 12184 4 Win11 2442 421 3517 2 Win10 2445 506 3517 2 Win10 2729 602 3918 2 Win11 2860 1334 8087 2 Win10 2980 835 8087 2 Win10 11577 536 3517 2 Win10 12642 2672 10170 4 Win11 61980 844 6020 2 Win11 66390 779 5965 2 Win11 73330 19454 32452 2 Win10 81549 4157 32272 8 Win11 15/20 have <=4 CPU cores and most show under 1 GB available physical RAM. Uptime spreads from 31s to 22h; both ends are represented but low-end / memory-pressured machines seem to dominate. With n=20 this is suggestive rather than conclusive. If that demographic skew is real, the most plausible mechanisms appear to be variants of "the OS apparently doesn't give the child enough wall-clock to react inside the parent's KillHard timeout": 1. Working set trimmed: a backgrounded content process may have its private working set evicted under system pressure; when RecvShutdown arrives the kernel might need to fault code/data pages back in just to schedule the IO Child thread. 2. EcoQoS / Process State Manager throttling on Win11 background processes. 3. Modern Standby / system-resume races on laptops. 4. Below-Normal priority combined with foreground contention on 2-core systems. If any of these is the dominant cause, none of them would really be a hang in the child to fix - the issue would be that the OS is giving the child less wall-clock than the parent assumed it would have. A possible practical lever, rather than chasing these as per-signature bugs, could be to proactively retire idle background content children when the parent observes system-level memory pressure, instead of letting them go cold and become KillHard candidates later. The relevant pieces already seem to exist but apparently aren't joined up: - In-process memory-pressure observers fire inside each content process; they don't appear to drive parent-side eviction. - nsITabUnloader unloads tabs under pressure but a content child seems to stay alive until its keep-alive grace expires. The lag between unload and process termination might be the window where pages get trimmed. - ProcessPriorityManager adjusts priority/visibility, and the parent does lift process priority and main-thread QoS just before sending RecvShutdown (ContentParent.cpp:3659 and 1694-1698) - but neither path terminates a child to relieve system pressure, and the kill-timeout itself is not gated on observed pressure. One candidate next step would be exploring whether a system-wide pressure signal (Windows MemoryResourceNotification, equivalents elsewhere) could feed into the parent-side decision to prefer terminating empty/idle content children earlier. If that worked, it would presumably reduce both the cold-page footprint during the session and the number of children the parent has to wake at quit - which appears to be what produces the bulk of this 821-crash bucket. But this is speculative; it would need to be proofed by data once we land something like this. Tentative recommendation for this meta: treat the idle bucket as probably non-actionable per-signature noise.
Bug 1279293 Comment 220 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Follow-up on the idle-at-shutdown bucket from the previous comment (821 of 1814 sampled). I pulled 20 idle-main-thread crash dumps from the last 14 days and walked every thread on every dump (~640 thread-instances) looking for any sign that the child had received or begun processing shutdown. The sample is small, but the pattern looks consistent: - No frame matching RecvShutdown / ShutdownInternal / XPCOMShutdown / Quit / any shutdown-observer call appeared on any thread of any of these 20 dumps. - ShutdownProgress, AsyncShutdownTimeout, MozCrashReason are all null on these dumps. The only shutdown-related field is the synthetic ipc_channel_error: "ShutDownKill" that the parent's KillHard path sets. - On every sampled dump, every thread appears parked on a Windows kernel wait (ZwWaitForAlertByThreadId on the worker pool, NtRemoveIoCompletion on IPC I/O Child / AudioIPC, NtWaitForSingleObject/MultipleObjects elsewhere). On the dumps I checked the IPC I/O Child had nothing pending in its completion port. If that pattern holds more broadly, these crashes seem unlikely to be "child stuck on something." They look more like the child being fully quiescent at the moment the parent terminates it, apparently before the shutdown message has been picked off the IO completion port - and the minidump is captured by ContentParent::KillHard immediately before TerminateProcess. System demographics for the 20 sampled (sorted by uptime): ``` uptime avail_mb total_mb cpu os 31 202 3499 2 Win11 34 569 8090 4 Win10 138 1942 7971 4 Win11 149 22487 40894 6 Win11 245 569 16239 4 Win10 469 1104 16070 12 Win11 1048 4981 12184 4 Win11 2442 421 3517 2 Win10 2445 506 3517 2 Win10 2729 602 3918 2 Win11 2860 1334 8087 2 Win10 2980 835 8087 2 Win10 11577 536 3517 2 Win10 12642 2672 10170 4 Win11 61980 844 6020 2 Win11 66390 779 5965 2 Win11 73330 19454 32452 2 Win10 81549 4157 32272 8 Win11 ``` 15/20 have <=4 CPU cores and most show under 1 GB available physical RAM. Uptime spreads from 31s to 22h; both ends are represented but low-end / memory-pressured machines seem to dominate. With n=20 this is suggestive rather than conclusive. If that demographic skew is real, the most plausible mechanisms appear to be variants of "the OS apparently doesn't give the child enough wall-clock to react inside the parent's KillHard timeout": 1. Working set trimmed: a backgrounded content process may have its private working set evicted under system pressure; when RecvShutdown arrives the kernel might need to fault code/data pages back in just to schedule the IO Child thread. 2. EcoQoS / Process State Manager throttling on Win11 background processes. 3. Modern Standby / system-resume races on laptops. 4. Below-Normal priority combined with foreground contention on 2-core systems. If any of these is the dominant cause, none of them would really be a hang in the child to fix - the issue would be that the OS is giving the child less wall-clock than the parent assumed it would have. A possible practical lever, rather than chasing these as per-signature bugs, could be to proactively retire idle background content children when the parent observes system-level memory pressure, instead of letting them go cold and become KillHard candidates later. The relevant pieces already seem to exist but apparently aren't joined up: - In-process memory-pressure observers fire inside each content process; they don't appear to drive parent-side eviction. - nsITabUnloader unloads tabs under pressure but a content child seems to stay alive until its keep-alive grace expires. The lag between unload and process termination might be the window where pages get trimmed. - ProcessPriorityManager adjusts priority/visibility, and the parent does lift process priority and main-thread QoS just before sending RecvShutdown (ContentParent.cpp:3659 and 1694-1698) - but neither path terminates a child to relieve system pressure, and the kill-timeout itself is not gated on observed pressure. One candidate next step would be exploring whether a system-wide pressure signal (Windows MemoryResourceNotification, equivalents elsewhere) could feed into the parent-side decision to prefer terminating empty/idle content children earlier. If that worked, it would presumably reduce both the cold-page footprint during the session and the number of children the parent has to wake at quit - which appears to be what produces the bulk of this 821-crash bucket. But this is speculative; it would need to be proofed by data once we land something like this. Tentative recommendation for this meta: treat the idle bucket as probably non-actionable per-signature noise.