1074927 - [Meta] Issues with log parsing times

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

In order for sheriffs to be able to maintain the trees that they sheriff (particularly those with high levels of commits per hour), they need to identify the cause of failed jobs quickly - since the longer a bad change is left in the repo, the more piled on breakage results as people land on top of it.

At the moment I'm seeing quite a few failed jobs on major trees such as mozilla-inbound that have a log parsing status of pending. Now these jobs have only finished recently (eg <15mins ago), but it would be great if we could still improve the speed with which they get parsed.

Some ideas:
* Increase number of workers on production.
* Double check we are correctly de-prioritising successful jobs.
* Prioritise certain categories of repository over others (eg prioritise "development" and "release stabilisation" repos over Try, since Try doesn't have the pile-on issue so can wait 15 mins longer etc).
* Come up with a way to on-demand increase prioritisation for a particular job - eg: if someone selects the job & the job details panels requests its artefacts - raise it to the top of the queue. Plus make the job details panel automatically refresh when the log has been parsed (only if it wasn't already parsed). 
* Profile the parser and try to make it faster.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I'm seeing a similar issue that may have a different root cause. I have a try job running at https://tbpl.mozilla.org/?tree=Try&rev=848b5ba710d4 and https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=848b5ba710d4 - on the tbpl link, The B2G ICS Emulator Debug X2 job is done and orange. I even starred it. On the treeherder page it still looks like it's pending. I did a shift-reload on treeherder and it's still pending. Neither the raw nor parsed log is available for it.

Comment 3

[Ed Morley [:emorley]]

Making this bug generic & going to file issues for each of the ideas in comment 0.

Comment 4

[Ed Morley [:emorley]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #1)
> I'm seeing a similar issue that may have a different root cause. I have a
> try job running at https://tbpl.mozilla.org/?tree=Try&rev=848b5ba710d4 and
> https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=848b5ba710d4 - on
> the tbpl link, The B2G ICS Emulator Debug X2 job is done and orange. I even
> starred it. On the treeherder page it still looks like it's pending. I did a
> shift-reload on treeherder and it's still pending. Neither the raw nor
> parsed log is available for it.

This should be helped by the deps of bug 1075799.

Comment 5

[Ed Morley [:emorley]]

Marking this meta as a P2, since the deps in need of fixing first are already P1s.

Comment 6

[Ed Morley [:emorley]]

We no longer have significant issues with log parsing times and most of the potential ideas for further improvements are already file, as deps of this bug. As such, I think we can close this meta bug out.