1076761 - Prioritise log parsing of failed jobs over successful ones

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder, defect, P4)

Description

[Ed Morley [:emorley]]

Broken out from bug 1074927.

We need to:
1) Double check that we are correctly giving higher priority to log parsing tasks for jobs that are failed, vs ones that were successful.
2) Add a cut-off time (or something similar), such that even a successful job won't have to wait longer than N minutes before being parsed (to prevent a backlog from suppressing log parsing of successful jobs indefinitely). This will help with Joel's use-case in bug 1075641.

Comment 1

[Ed Morley [:emorley]]

It turns out the log_parser, log_parser_hp and log_parser_fail queues are equal priority, and supposedly dealt with in a round robin manner.

It would help with backlogs if we prioritised log_parser_hp and log_parser_fail over log_parser.

Perhaps we should rebalance some of the queues once the default queue has been split up. We could also try adding some of the high priority queues to more than one node type (eg log_parser_hp and high_priority could be done by any of {etl,processor} etc).

Comment 2

[Ed Morley [:emorley]]

We already kind of prioritise logs for failed jobs, in the sense that the round-robin behaviour means they get handled ahead of the passing ones 50% of the time. As for prioritising further, I don't think it's worth it.