1105800 - Treeherder missing job completion data for Try pushes

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

I pushed off a bunch of uplift simulation Try pushes last night, and Treeherder shows them as all running still. TBPL and buildapi correctly show them as completed.
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=48296352efb7
https://tbpl.mozilla.org/?tree=Try&rev=48296352efb7

Comment 1

[Mauro Doglio [:mdoglio]]

Yeah there's a data ingestion issue on try completed jobs. I'm working on it.

Comment 2

[Mauro Doglio [:mdoglio]]

The web service is timing out when loading the try job data, I'm gonna increase the timeout as a short term solution.

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/78ecb0900c95daab5a4db17e65b1d8640b1cd62c
Bug 1105800 - increase the web service timeout limit

Comment 6

[Phil Ringnalda (:philor)]

Despite everything being fine on other trees on treeherder, and everything being fine on try on tbpl (now that the revision with 100k jobs is gone from builds-4hr), treeherder+try still isn't showing finished jobs.

Comment 8

[Mauro Doglio [:mdoglio]]

Completed jobs are visible now on try. The changes described in comment 2 + the builds4hr file being back to a normal size solved the problem apparently.

Comment 9

[Phil Ringnalda (:philor)]

Because I didn't understand this, and for the benefit of people with try pushes in the affected range:

treeherder doesn't know how to retry ingesting the stuff that was in the affected range, now long gone from builds-4hr, so for that stuff from Wednesday evening to Thursday afternoon that treeherder still shows as running, you just need to look on tbpl to see your results.