1162868 - push to try, only showing 4 build jobs when buildapi shows many more

Status: RESOLVED DUPLICATE of bug 1191934

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=08745d039078 shows linux32 opt/debug and linux64 opt/debug running.

this link:
https://secure.pub.build.mozilla.org/buildapi/self-serve/try/rev/08745d039078

shows 6 pending jobs (windows, osx), and 4 running (the above mentioned linux).

This is misleading as we should be showing pending jobs in the UI.

Comment 1

[Ed Morley [:emorley]]

Unfortunately the jobs in question are now showing as completed, and so are no longer in builds-pending.js for me to check whether Treeherder was missing them, or whether they were just not appearing in the datasources made available to us.

Looking at the stored artefacts for say the WinXP build:
https://treeherder.mozilla.org/api/project/try/artifact/?job_id=7346100&type=json&name__in=buildapi_running,buildapi_pending,buildapi_complete

I see we have captured the builds-running.js part (and obviously the completed one), but not for builds-pending.js

The job details panel shows:
Requested: Fri May 8, 9:05:56
Started: Fri May 8, 9:58:47 

Which rules out the job being moved from pending->running so quickly that we wouldn't expect it to appear in builds-pending.js.

Checking New Relic, there is a timeout for the builds-running.js task around that time (ie the request to the releng server hosting the file timed out), but not for the builds-pending.js task (which is the one that matters here):
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors?tw[end]=1431076354&tw[start]=1431069290

Moving onto the celery worker logs for the builds-pending.js task:
https://emorley.pastebin.mozilla.org/8832871

It shows that for whatever reason the pending jobs were not finding the resultset repeatedly. (And we can't import the jobs without the resultset). I'm not sure if the logging is just wrong, or we're caching something we shouldn't or what.

I'll dig in more.

If anything like this happens again, could you add to this bug and also save a copy of https://secure.pub.build.mozilla.org/builddata/buildjson/builds-pending.js at the time, so we can take a look? :-)