1441181 - (bmo-cloud-jobqueue) Implement new process model for running multiple email jobqueue daemons

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, enhancement)

Description

[:glob ✱]

Implement new process model for running multiple email jobqueue daemons, as per ckolos's recommendations.

Comment 1

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 1 - Remove old subprocess code

This PR removes the subprocess code that was added because of memory leaks.
It's intended to be replaced at the end of the patch series with Process::SizeLimit::Core.

Comment 2

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 2 - Fix jobqueue header and @INC

The header in the jobqueue code is weird; this patch makes it more like more recent changes.

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 3 - Clear memory after each job, not before

While working on this I noticed memory wasn't being freed in a predictable way. This is because the memory is cleared before each job, instead of after.

Comment 4

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 4 - Re-implement subprocess code with IO::Async

This re-implements the sub process code in terms of IO::Async. It doesn't yet handle memory problems but I'm going to add that in my next period of coding.

Comment 5

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 5 - Add logging

This is not strictly needed, but it wires up the TheSchwartz logging code into our logging framework.

Comment 6

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 6 - Add concurrency

Comment 7

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 7 - Fix bug in catch_signal()

Comment 8

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 5 - Add logging

Comment 9

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 8 - Add jobqueue command to entrypoint

Comment 10

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 9 - use pdeathsig to prevent workers running without supervision

Comment 11

[Dylan Hardison [:dylan] (he/him)]

Attachment: Step 6 - Add concurrency