1076763 - Consider prioritising log parsing for some repository groups over others

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P4)

Description

[Ed Morley [:emorley]]

Broken out from bug 1074927.

Certain repositories are more affected by a backlog of log parsing jobs more than others. eg: a push to Try can wait an extra 20 mins for log parsing if needs be (ideally it won't have to, but in worst case) - whereas a delay of that length will cause problems on repositories where changes stack on top of each other.

If we do this, I would suggest making "development" and "release stabilisation" repositories top priority, and type "try" lowest.

We'll also need to implement the cut-off time idea from bug 1076761 comment 0, to ensure that the jobs don't get indefinitely postponed in extreme load conditions.

Comment 1

[Ed Morley [:emorley]]

With the additional workers & also bug 1076769, the log parsing times should be less of an issue, so this can wait a bit longer (it will help with the longer term scaling as we add more repos/data sources).

Comment 2

[Ed Morley [:emorley]]

With the de-emphasis of log parsing inside treeherder (vs non-buildbot submitters submitting already-parsed logs), I don't think we'll need/want to do this, since bug 1076761 is the quicker win, should we need it.