1924886 - Make web servers and livegrep codesearch be run/monitored/automatically restarted by systemd

Status: NEW

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

Currently our various web-servers are just run by web-server-run.sh without use of any kind of mechanism to automatically terminate and restart the processes. A caveat is that router.py knows how to start/restart the codesearch daemon.

I'd looked into this a while back and systemd seems to be the most pragmatic solution especially since it supports a heartbeat mechanism, but I don't have strong feelings. (I know some people have strong feelings about systemd.) There was a limitation for WSL in the past, but systemd is now available/reliable on WSL(2).

This has worked quite well since the processes haven't fallen over for quite some time, but:

• When router.py daemonizes the codesearch daemon, it hooks up its stdout/stderr to /dev/null and in general I think we would actually like to have that output, especially as we potentially look into performance in bug 1794004.
  • This mattered where, for a while, we were experiencing mysterious problems when spinning up the codesearch instances and not being able to see what the codesearch daemon was saying was quite frustrating.
• In bug 1924881 I noted observing the first known OOM termination of the pipeline-server where it ran against the ulimit we give it. Because the pipeline-server is fundamentally very dynamic, it makes sense for us to make sure we can restart it as-needed.