1328453 - calculate-durations DataError: (1264, "Out of range value for column 'average_duration' at row 1")

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P1)

Description

[Ed Morley [:emorley]]

https://rpm.newrelic.com/accounts/677903/applications/14179757/filterable_errors#/show/ff71fed0-d1f7-11e6-9c7c-b82a72d22a14_0_5979/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart&_k=gdwj3t

File "/app/treeherder/model/derived/jobs.py", line 407, in calculate_durations
django.db.utils:DataError: (1264, "Out of range value for column 'average_duration' at row 1")

Which is here:
https://github.com/mozilla/treeherder/blob/de4c189c15e5115c6e8a3f51c59932268a95782b/treeherder/model/derived/jobs.py#L404-L407

Comment 1

[William Lachance (:wlach)]

This is almost certainly somehow my fault. :)

Comment 2

[William Lachance (:wlach)]

On closer examination, I think this could happen if the end time for a job was less than the start time. This should never happen, but apparently does somehow. I think we can guard against it easily, but probably best to wait for the rest of bug 1318474 to land first (so we don't have merge conflicts).

Comment 3

[GitHub Autolander Bot]

Attachment: [treeherder] KWierso:jobtimes > mozilla:master

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 8843112 [details] [review]
[treeherder] KWierso:jobtimes > mozilla:master

If skipping jobs that finish before they start is needed, would this be what we need?

Comment 5

[William Lachance (:wlach)]

Comment on attachment 8843112 [details] [review]
[treeherder] KWierso:jobtimes > mozilla:master

Yes, this is the right idea, though I think we can make things more efficient by skipping these jobs at the query level: http://stackoverflow.com/a/12380982 Could you try that? Ideally we would also add a unit test for this case in tests/model/test_calculate_durations.py. If you don't have time to do this, I can probably take care of it, let me know.

On reflection I'm not sure if we should try to fix this at the data ingestion level, since (I think) the end before start weirdness is something coming from the jobs themselves.

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

If you can take it, that'd be great.

Comment 7

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1328453 > mozilla:master

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8843444 [details] [review]
[treeherder] wlach:1328453 > mozilla:master

Turns out I had accidentally deleted the calculate durations tests. Re-added them, along with a fix for this issue.

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8843444 [details] [review]
[treeherder] wlach:1328453 > mozilla:master

Many thanks for fixing this!

Comment 10

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/fe5742a236a3bcd96f16fd68cd32b010b628403b
Bug 1328453 - Re-add accidentally deleted unit test for calculating durations

https://github.com/mozilla/treeherder/commit/1a19c1601ab513e996c486a9bca0e5a580f1ce5e
Bug 1328453 - Only use completed jobs for calculating average job duration

Completed jobs = those with an end time > start_time