795417 - Persistent job list

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Mark Côté [:mcote]]

If a worker dies or autophone is shut down, current and pending jobs are forgotten.

They should be saved to disk and flushed on change to ensure they stay up to date. We will also need to remember the number of attempts on the current job and discard if it gets too high, in case it causes multiple crashes or other errors.

Comment 1

[Mark Côté [:mcote]]

Attachment: Class for a persistent job list

This is a work in progress and hasn't been tested.

Comment 2

[Mark Côté [:mcote]]

This will require a few structural changes to get right.

Firstly we'll further decouple the tcp server and the AutoPhone class by having the server push onto a command queue and then sitting on a response queue or the like rather than calling into route_cmd() directly.  Then we'll have the workers write their statuses to that queue as well.

After this change, I'll modify AutoPhone to write jobs to device-specific persistent queues rather than directly to the worker processes as they come in.  The workers will send requests for jobs to the command queue, which will cause autophone to write only the first queued job to the worker.  This will also solve the problem of non-job messages (like disable) being queued after all pending jobs.  Finally, the worker will send a job completed message back to AutoPhone, which will cause the job to be erased from the queue.

I think this will also lay further groundwork for mozpool integration.

Comment 3

[Mark Côté [:mcote]]

Attachment: Persistent job queues and a few little changes and cleanups

Avoided a lot of structural changes by just using a sqlite db, which is multiprocess-safe.

As described in the patch, I also removed the reboot-on-start behaviour (and the associated --no-reboot option), added a "reboot" command, and added support for "all" instead of a particular device ID for device commands.

Comment 4

[Bob Clary [:bc] (inactive)]

Comment on attachment 727783 [details] [diff] [review]
Persistent job queues and a few little changes and cleanups

r+ sweet

We can deal with the removal of previous partial results separately.

Comment 5

[Mark Côté [:mcote]]

Yup, previously filed as bug 850438.  Will work on that next.

https://github.com/mozilla/autophone/commit/63933d50f835c0b7c2f87d4de089ec94fc027b49