804363 - [grizwald] Implement hybrid jobs

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Add-on Validation, defect)

Description

[Matt Basta [:basta]]

Hybrid jobs should accept both a task list and a procfile. Hybrid jobs may provide two repo URLs and two commit IDs: one for daemon jobs, the other for task jobs.

- Daemon and task workers initialize their respective repos.
- Daemons from procfile initialized.
- Task workers wait until daemons are active through countdown latch in Redis
- Task workers execute against the task list. The IPs/ports are passed to the griz.py script as a JSON-encoded list in the format that a daemon job would pass as its output key.
- Upon task completion (detected by the daemon workers when INCOMPLETE is zero or has been deleted), the daemon workers should tear down.
- Task workers should tear down after the task queues are emptied and final reduction has taken place.

Some considerations:

- Limits should be imposed on the number of workers that can initialize a job by using EXISTS and then DECR on a key. If the value comes back negative or null, the job is ignored (other workers have already taken all the work). The key should be removed on cleanup.
  - Currently, all idle workers will download a repo. This will cause heartache if we only want some of the workers to get a copy.
- If a daemon terminates with a non-zero status code, the task queue is emptied, the other daemons should terminate, and the job should be cancelled.
- If a reduction throws an exception, the job should be cancelled in the same manner.

Comment 1

[Matt Basta [:basta]]

This is largely no longer needed, and the WDSCCF is no longer really a thing.