The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shutdown but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shutdown. To do that, we should look at what the main thread is doing (the logs contain stacks for the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what's going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
Bug 1831236 Comment 32 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shutdown but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shutdown in time. To do that, we should look at what the main thread is doing (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what's going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread is doing (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what's going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process is doing (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what's going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process is doing (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what's going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching memory addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process was doing when we forced the crash (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what was going on in the main thread of the child process. Here are the results of doing that: - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (13 failures): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (7 failures): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching memory addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
The intermittent failures that we observe with this signature were introduced by bug 1793525. They occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process was doing when we forced the crash (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what was going on in the main thread of the child process. Here are the results of doing that (test categorization is not exact, please check the linked jobs directly if you need something more accurate): - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures in mochitest-browser-chrome/mochitest-plain/mochitest-a11y/mochitest-webgpu): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (12 failures in mochitest-browser-chrome/mochitest-plain/mochitest-media, 1 failure in xpcshell test): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures in mochitest-browser-media-wmfme): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures in mochitest-browser-media-wmfme): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (4 failures in mochitest-browser-chrome-msix, 3 failures in mochitest-media): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures in mochitest-browser-chrome/mochitest-plain): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures in mochitest-plain): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure in mochitest-browser-chrome): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure in mochitest-browser-chrome-swr): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure in xpcshell-spi-nw): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching memory addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.
I think that the intermittent failures that we observe with this signature were introduced by bug 1793525. They would occur when the main process kindly asked a child process to shut down but the child process is taking too long to do that. Thanks to bug 1793525, when that happens, we now get a dump of the child process, so we can have an idea of why it did not shut down in time. To do that, we should look at what the main thread of the child process was doing when we forced the crash (the logs contain stacks for the threads of the child process), and see if that explains why it is not processing events. If we are seeing false positives where the child process was legitimately taking a lot of time for shutdown, we could consider increasing [the constants that define how much time we wait](https://searchfox.org/mozilla-central/source/ipc/chromium/src/chrome/common/process_watcher_win.cc#16-32). NI [:jld] for his opinion on about that. Now, I looked at the child process dumps from comment 25 and comment 30, and grouped them based on common patterns of what was going on in the main thread of the child process. Here are the results of doing that (test categorization is not exact, please check the linked jobs directly if you need something more accurate): - `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` (24 failures in mochitest-browser-chrome/mochitest-plain/mochitest-a11y/mochitest-webgpu): [25-11](https://treeherder.mozilla.org/logviewer?job_id=431482470&repo=autoland&lineNumber=3381), [25-12](https://treeherder.mozilla.org/logviewer?job_id=431482994&repo=autoland&lineNumber=2857), [25-22](https://treeherder.mozilla.org/logviewer?job_id=431339066&repo=autoland&lineNumber=3381), [25-25](https://treeherder.mozilla.org/logviewer?job_id=431329545&repo=mozilla-central&lineNumber=2489), [25-29](https://treeherder.mozilla.org/logviewer?job_id=431242556&repo=autoland&lineNumber=2905), [25-30](https://treeherder.mozilla.org/logviewer?job_id=431242587&repo=autoland&lineNumber=3388), [25-4](https://treeherder.mozilla.org/logviewer?job_id=431613816&repo=mozilla-central&lineNumber=3388), [25-6](https://treeherder.mozilla.org/logviewer?job_id=431577996&repo=autoland&lineNumber=3388), [25-7](https://treeherder.mozilla.org/logviewer?job_id=431577869&repo=autoland&lineNumber=2821), [25-8](https://treeherder.mozilla.org/logviewer?job_id=431556171&repo=mozilla-central&lineNumber=2821), [30-1](https://treeherder.mozilla.org/logviewer?job_id=432506137&repo=autoland&lineNumber=3352), [30-14](https://treeherder.mozilla.org/logviewer?job_id=432284008&repo=autoland&lineNumber=3397), [30-15](https://treeherder.mozilla.org/logviewer?job_id=432282441&repo=autoland&lineNumber=4037), [30-18](https://treeherder.mozilla.org/logviewer?job_id=432250269&repo=autoland&lineNumber=3467), [30-2](https://treeherder.mozilla.org/logviewer?job_id=432502540&repo=autoland&lineNumber=3682), [30-21](https://treeherder.mozilla.org/logviewer?job_id=432171397&repo=mozilla-central&lineNumber=4028), [30-25](https://treeherder.mozilla.org/logviewer?job_id=431980757&repo=autoland&lineNumber=3379), [30-27](https://treeherder.mozilla.org/logviewer?job_id=431952144&repo=mozilla-central&lineNumber=2945), [30-28](https://treeherder.mozilla.org/logviewer?job_id=431900533&repo=mozilla-central&lineNumber=2872), [30-31](https://treeherder.mozilla.org/logviewer?job_id=431869508&repo=autoland&lineNumber=3392), [30-33](https://treeherder.mozilla.org/logviewer?job_id=431864261&repo=autoland&lineNumber=3047), [30-34](https://treeherder.mozilla.org/logviewer?job_id=431865012&repo=autoland&lineNumber=2940), [30-4](https://treeherder.mozilla.org/logviewer?job_id=432429173&repo=mozilla-central&lineNumber=4042), [30-5](https://treeherder.mozilla.org/logviewer?job_id=432424636&repo=autoland&lineNumber=4052); - `xul.dll!mozilla::ShutdownXPCOM` (12 failures in mochitest-browser-chrome/mochitest-plain/mochitest-media, 1 failure in xpcshell test): [25-0](https://treeherder.mozilla.org/logviewer?job_id=431733734&repo=mozilla-central&lineNumber=5370), [25-1](https://treeherder.mozilla.org/logviewer?job_id=431698556&repo=autoland&lineNumber=5343), [25-14](https://treeherder.mozilla.org/logviewer?job_id=431459630&repo=autoland&lineNumber=4574), [25-15](https://treeherder.mozilla.org/logviewer?job_id=431450976&repo=mozilla-beta&lineNumber=3073), [25-19](https://treeherder.mozilla.org/logviewer?job_id=431401286&repo=mozilla-central&lineNumber=5265), [25-20](https://treeherder.mozilla.org/logviewer?job_id=431351379&repo=mozilla-beta&lineNumber=5184), [25-31](https://treeherder.mozilla.org/logviewer?job_id=431242584&repo=autoland&lineNumber=2945), [30-10](https://treeherder.mozilla.org/logviewer?job_id=432329174&repo=try&lineNumber=40105), [30-13](https://treeherder.mozilla.org/logviewer?job_id=432293394&repo=mozilla-beta&lineNumber=114369), [30-16](https://treeherder.mozilla.org/logviewer?job_id=432263845&repo=mozilla-beta&lineNumber=2818), [30-19](https://treeherder.mozilla.org/logviewer?job_id=432232725&repo=try&lineNumber=82359), [30-22](https://treeherder.mozilla.org/logviewer?job_id=432063402&repo=mozilla-central&lineNumber=3987), [30-6](https://treeherder.mozilla.org/logviewer?job_id=432420700&repo=mozilla-esr115&lineNumber=2276); - `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown` (11 failures in mochitest-browser-media-wmfme): [25-18](https://treeherder.mozilla.org/logviewer?job_id=431406290&repo=mozilla-central&lineNumber=2988), [25-2](https://treeherder.mozilla.org/logviewer?job_id=431702541&repo=mozilla-central&lineNumber=2989), [25-23](https://treeherder.mozilla.org/logviewer?job_id=431336982&repo=mozilla-central&lineNumber=2981), [25-28](https://treeherder.mozilla.org/logviewer?job_id=431270158&repo=mozilla-central&lineNumber=2982), [25-3](https://treeherder.mozilla.org/logviewer?job_id=431624776&repo=mozilla-central&lineNumber=2981), [30-0](https://treeherder.mozilla.org/logviewer?job_id=432559069&repo=mozilla-central&lineNumber=3633), [30-20](https://treeherder.mozilla.org/logviewer?job_id=432230450&repo=mozilla-central&lineNumber=3617), [30-23](https://treeherder.mozilla.org/logviewer?job_id=432063386&repo=mozilla-central&lineNumber=2982), [30-29](https://treeherder.mozilla.org/logviewer?job_id=431909503&repo=mozilla-central&lineNumber=2993), [30-3](https://treeherder.mozilla.org/logviewer?job_id=432483641&repo=mozilla-central&lineNumber=3636), [30-9](https://treeherder.mozilla.org/logviewer?job_id=432368136&repo=mozilla-central&lineNumber=3632); - `xul.dll!mozilla::WMFDecoderModule::Init` (8 failures in mochitest-browser-media-wmfme): [25-10](https://treeherder.mozilla.org/logviewer?job_id=431506480&repo=mozilla-central&lineNumber=3547), [25-16](https://treeherder.mozilla.org/logviewer?job_id=431449479&repo=autoland&lineNumber=4160), [25-26](https://treeherder.mozilla.org/logviewer?job_id=431286163&repo=autoland&lineNumber=4523), [25-32](https://treeherder.mozilla.org/logviewer?job_id=431178491&repo=mozilla-central&lineNumber=3545), [25-5](https://treeherder.mozilla.org/logviewer?job_id=431579915&repo=autoland&lineNumber=4156), [30-11](https://treeherder.mozilla.org/logviewer?job_id=432304166&repo=autoland&lineNumber=3963), [30-30](https://treeherder.mozilla.org/logviewer?job_id=431905493&repo=mozilla-esr115&lineNumber=2254), [30-32](https://treeherder.mozilla.org/logviewer?job_id=431869664&repo=autoland&lineNumber=4160); - `xul.dll!CrashReporter::UnsetRemoteExceptionHandler` (4 failures in mochitest-browser-chrome-msix, 3 failures in mochitest-media): [25-13](https://treeherder.mozilla.org/logviewer?job_id=431482432&repo=autoland&lineNumber=2946), [25-24](https://treeherder.mozilla.org/logviewer?job_id=431335043&repo=mozilla-central&lineNumber=2956), [25-33](https://treeherder.mozilla.org/logviewer?job_id=431142222&repo=try&lineNumber=3377), [30-12](https://treeherder.mozilla.org/logviewer?job_id=432302139&repo=autoland&lineNumber=3602), [30-24](https://treeherder.mozilla.org/logviewer?job_id=431991503&repo=mozilla-beta&lineNumber=3644), [30-26](https://treeherder.mozilla.org/logviewer?job_id=431979061&repo=mozilla-beta&lineNumber=3414), [30-7](https://treeherder.mozilla.org/logviewer?job_id=432397771&repo=try&lineNumber=3999); - `xul.dll!mozilla::Telemetry::Init` (3 failures in mochitest-browser-chrome/mochitest-plain): [25-21](https://treeherder.mozilla.org/logviewer?job_id=431351358&repo=mozilla-beta&lineNumber=3074), [25-35](https://treeherder.mozilla.org/logviewer?job_id=431068729&repo=mozilla-beta&lineNumber=3410), [30-17](https://treeherder.mozilla.org/logviewer?job_id=432264839&repo=mozilla-beta&lineNumber=3770); - `xul.dll!nsComponentManagerImpl::Init` (2 failures in mochitest-plain): [25-17](https://treeherder.mozilla.org/logviewer?job_id=431441523&repo=mozilla-beta&lineNumber=3078), [25-27](https://treeherder.mozilla.org/logviewer?job_id=431279029&repo=try&lineNumber=2811); - `xul.dll!mozilla::ChildProfilerController::Init` (1 failure in mochitest-browser-chrome): [25-34](https://treeherder.mozilla.org/logviewer?job_id=431140452&repo=try&lineNumber=3375); - `xul.dll!mozilla::ipc::PUtilityProcessChild::OnChannelClose` (1 failure in mochitest-browser-chrome-swr): [30-8](https://treeherder.mozilla.org/logviewer?job_id=432375131&repo=try&lineNumber=4048); - `xul.dll!mozilla::net::SocketProcessChild::Init` (1 failure in xpcshell-spi-nw): [25-9](https://treeherder.mozilla.org/logviewer?job_id=431540725&repo=autoland&lineNumber=2390). The failures in `xul.dll!mozilla::ipc::UtilityProcessImpl::LoadLibraryOrCrash` should indeed be considered the same as bug 1851889. Here, we crash while the child process is walking its own stack and printing it to stdout before crashing, as part of `MOZ_ReportCrash`. It is expected that this can take some time, as it requires matching memory addresses to code lines using PDB files, and `xul.pdb` is huge. If the child process was given enough time, these failures would end up in bug 1851889. Also, fixing the issue there would make the ones we have here disappear as well. The other failures would require more analysis. Also, profiling these tests could give a more precise idea of what is really taking time for the failures that seem quite generic like `xul.dll!mozilla::ShutdownXPCOM` and `xul.dll!mozilla::layers::CompositorThreadHolder::Shutdown`. Here we only know what the process was doing when we reached the timeout, and we could reach wrong conclusions if we only look at that.