1294544 - Treeherder should limit the task time for log parsing so we don't get a backlog on long logs

Status: NEW

(Tree Management :: Treeherder, defect, P3)

Description

[Cameron Dawson [:camd]]

We are hitting some cases where logs are 2GB, which can take 16 minutes to parse.  We need to limit the time we allow a task to run

Comment 1

[Ed Morley [:emorley]]

In addition to the instances in bug 1294548 and bug 1343831, we just had another today.

23:28 <emorley> gbrown: this try run is creating 400MB logs which momentarily caused log parser backlogs: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1b2c3fe5bbe2cb539e9513067d333891ed4e511c
23:29 <gbrown> emorley: wow, spectacular failure. sorry. cancelled.
23:30 <emorley> gbrown: np, treeherder should handle this case better (doing so in bug 1294544, though requires changing the way we record log parser failures so we can improve the UX and explain that log parsing was skipped deliberately)
23:31 <emorley> it's just that in the meantime, 500 jobs x 200-400MB logs is quite a bit of parsing time :-)

Comment 2

[Ed Morley [:emorley]]

Oops comment 1 was meant for bug 1295997.

This bug is still useful for these situations, particularly in high-compression levels cases. See bug 1295997 comment 5.

Comment 3

[Ed Morley [:emorley]]

I think this is lower priority than it's size-limit related counterpart, bug 1295997.