Bugzilla

(In reply to Nika Layzell [:nika] (ni? for response) from comment #0)
> This was noticed by :nalexander (https://matrix.to/#/!ykdkAGURCpjeYhwLFB:mozilla.org/$7IsoT4jUdN7iNMLbnELd06M70g3AiaC1J1yy03UX1n0?via=mozilla.org&via=matrix.org&via=mattn.ca) in a try push which creates a large number of `firefox --backgroundtask` instances (https://treeherder.mozilla.org/logviewer?job_id=377708202&repo=try&lineNumber=10531). Apparently these instances of Firefox will start, run a small amount of JS, and then shut down almost immediately.
> 
> The issue here appears to be that we don't wait for the background thread to actually finish initializing before proceeding to the next shutdown phase, which can lead to issues. When the service is initialized, it starts a background thread and dispatches a startup task to it which creates the actor managed by `PBackground` (https://searchfox.org/mozilla-central/rev/88792eff309001778cb2431f2a0ed92f8f3c258a/dom/workers/remoteworkers/RemoteWorkerService.cpp#97-101, https://searchfox.org/mozilla-central/rev/88792eff309001778cb2431f2a0ed92f8f3c258a/dom/workers/remoteworkers/RemoteWorkerService.cpp#117). If this occurs after the `xpcom-shutdown-threads` phase, this will assert as PBackground will already be shut down, meaning new actors cannot be started (https://searchfox.org/mozilla-central/rev/88792eff309001778cb2431f2a0ed92f8f3c258a/ipc/glue/BackgroundImpl.cpp#458).
> 
> The service is set up to observe `xpcom-shutdown`, which it will dispatch an event to the background thread from to ask it to shut down (https://searchfox.org/mozilla-central/rev/88792eff309001778cb2431f2a0ed92f8f3c258a/dom/workers/remoteworkers/RemoteWorkerService.cpp#165-169). It however does not wait for the shutdown task to be run, so there is no guarantee that the background thread ever finished initializing before it returns, potentially leading to this bug. The only thing which synchronizes with the thread shutting down after this point to ensure it's dead before the process exits is the forced-shutdown of the thread in `nsThreadManager::ShutdownNonMainThreads()`.
> 
> An easy fix here might be to remove the second dispatch from the target thread back to the main thread, and to instead call `mThread->Shutdown();` immediately after dispatching the event to it from the main thread, which will spin a nested event loop waiting for the thread to exit after processing all posted events, and will also wait for it to finish initializing as a side-effect.

FWIW, I hacked the easy fix in and my previously unhappy try builds are no longer crashing.  See all the green TV jobs [here](https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=H83b5QbwRYSVCGzDIwvz3Q.0&revision=3a1ac6077fb02daa1ea3d2e828ba2c59646b8ab1) and [the trivial patch](https://hg.mozilla.org/try/rev/4f742d3b98aeb3ce5423e90f80889493bd552d87).

(In reply to Nika Layzell [:nika] (ni? for response) from comment #0)

> An easy fix here might be to remove the second dispatch from the target thread back to the main thread, and to instead call `mThread->Shutdown();` immediately after dispatching the event to it from the main thread, which will spin a nested event loop waiting for the thread to exit after processing all posted events, and will also wait for it to finish initializing as a side-effect.

FWIW, I hacked the easy fix in and my previously unhappy try builds are no longer crashing.  See all the green TV jobs [here](https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=H83b5QbwRYSVCGzDIwvz3Q.0&revision=3a1ac6077fb02daa1ea3d2e828ba2c59646b8ab1) and [the trivial patch](https://hg.mozilla.org/try/rev/4f742d3b98aeb3ce5423e90f80889493bd552d87).

Thanks so much for this easy-to-test suggestion, Nika!