1145998 - Some jobs on my try push never got finished, according to Treeherder but not to buildapi

Status: RESOLVED DUPLICATE of bug 1162526

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[(no longer active)]

See this try push from yesterday: <https://treeherder.mozilla.org/#/jobs?repo=try&revision=4776b6bbfede>.

Comment 1

[Ed Morley [:emorley]]

tl;dr inconsistencies in the data we get given in builds-4hr / we could pick more reliable properties.

Comment 2

[Ed Morley [:emorley]]

Or the infra issues we've been having, take your pick :-)

Comment 3

[Ed Morley [:emorley]]

Although there are no dupe jobs, so I'll reopen this to remind me to dig into it more when I get back from PTO & have caught up (realistically 7th April given public holidays too).

Comment 4

[(no longer active)]

The jobs are still pending, so hopefully they'll remain so until you're back!  ;-)

Comment 5

[Mike Hommey [:glandium]]

Same for me on
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b162c1cce259
and
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=247fe1dfb150

Comment 6

[Chris Manchester (limited bugmail, email directly)]

My perception is this happens somewhat frequently (https://treeherder.mozilla.org/#/jobs?repo=try&revision=e22656e8151c most recently, a number of jobs on that push).

Comment 7

[Ed Morley [:emorley]]

Sorry it's taken a while to double back to this, it needed a bit time/headspace to dig into it.

Ehsan's:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4776b6bbfede&filter-resultStatus=running

https://treeherder.mozilla.org/api/project/try/jobs/5813383/
"job_guid": "1f640a0deac98d8b9df8b27e5f2ddddde7c402c7"
https://treeherder.mozilla.org/api/project/try/jobs/5813347/
"job_guid": "38fe1d832bf476c91b7615295c8f62300fd01dea"
https://treeherder.mozilla.org/api/project/try/jobs/5813385/
"job_guid": "e7de8ec3cb34a868dda4103e34020bcbde15d0cc"
https://treeherder.mozilla.org/api/project/try/jobs/5813349/
"job_guid": "8d478dcacafa49e86104ee331f448958f363830f"
https://treeherder.mozilla.org/api/project/try/jobs/5813371/
"job_guid": "f4f5a308acc8f74f6b3a5f588bdcfa24fac2c59a"
https://treeherder.mozilla.org/api/project/try/jobs/5813377/
"job_guid": "715da47f34c7ec99ac20ad51d58df39996d569ab"

Chris':
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e22656e8151c&filter-resultStatus=running

https://treeherder.mozilla.org/api/project/try/jobs/7173925/
"job_guid": "f4c4129c184e2521ba41fef4b32c58b8d8f3638b"
https://treeherder.mozilla.org/api/project/try/jobs/7173700/
"job_guid": "af0bc79228731acc0f2a825b39245685db6bba7c"
https://treeherder.mozilla.org/api/project/try/jobs/7173698/
"job_guid": "73790344eb6bb580c8fde3d5ad1427e32276eee0"
https://treeherder.mozilla.org/api/project/try/jobs/7173911/
"job_guid": "9dd9e7fa7bdc077969447809d9c4a04efbc719f8"
https://treeherder.mozilla.org/api/project/try/jobs/7170692/
"job_guid": "22291fb5ab16ca6bd3bc33d6afc64830c4c874c4"

Execute:
> SELECT id, job_guid, from_unixtime(loaded_timestamp) as date, processed_state, worker_id FROM try_objectstore_1.objectstore 
WHERE job_guid IN
("1f640a0deac98d8b9df8b27e5f2ddddde7c402c7", "38fe1d832bf476c91b7615295c8f62300fd01dea",
"e7de8ec3cb34a868dda4103e34020bcbde15d0cc", "8d478dcacafa49e86104ee331f448958f363830f",
"f4f5a308acc8f74f6b3a5f588bdcfa24fac2c59a", "715da47f34c7ec99ac20ad51d58df39996d569ab",
"f4c4129c184e2521ba41fef4b32c58b8d8f3638b", "af0bc79228731acc0f2a825b39245685db6bba7c",
"73790344eb6bb580c8fde3d5ad1427e32276eee0", "9dd9e7fa7bdc077969447809d9c4a04efbc719f8",
"22291fb5ab16ca6bd3bc33d6afc64830c4c874c4")

+ ------- + ------------- + --------- + -------------------- + -------------- +
| id      | job_guid      | date      | processed_state      | worker_id      |
+ ------- + ------------- + --------- + -------------------- + -------------- +
| 8115506 | 1f640a0deac98d8b9df8b27e5f2ddddde7c402c7 | 2015-03-20 18:40:18 | loading              | 106595457      |
| 9182900 | 22291fb5ab16ca6bd3bc33d6afc64830c4c874c4 | 2015-05-01 23:51:13 | loading              | 550756198      |
| 8115639 | 38fe1d832bf476c91b7615295c8f62300fd01dea | 2015-03-20 18:43:17 | loading              | 106597818      |
| 8115642 | 715da47f34c7ec99ac20ad51d58df39996d569ab | 2015-03-20 18:43:17 | loading              | 106597818      |
| 8115640 | 8d478dcacafa49e86104ee331f448958f363830f | 2015-03-20 18:43:17 | loading              | 106597818      |
| 8115505 | e7de8ec3cb34a868dda4103e34020bcbde15d0cc | 2015-03-20 18:40:18 | loading              | 106595457      |
| 8115641 | f4f5a308acc8f74f6b3a5f588bdcfa24fac2c59a | 2015-03-20 18:43:17 | loading              | 106597818      |
+ ------- + ------------- + --------- + -------------------- + -------------- +
7 rows

So for 7 of the 11 jobs, we're seeing bug 1125476.

For one more job (https://treeherder.mozilla.org/#/jobs?repo=try&revision=e22656e8151c&filter-searchStr=Windows%208%20x64%20debug%20Build%20%28B%29), there is a ghost/duplicate completed job (ie we've not correctly associated the running and completed jobs, which will be due to bug 1093743, which has a patch awaiting review).

This leaves three that are stuck running but have no completed counterpart. I believe these may be due to the ongoing builds-4hr inconsistencies, for which releng landed a patch yesterday (bug 942616).

All of comment 0's running jobs were due to bug 1125476, so duping there.

The sooner we're rid of builds-4hr/builds-running/builds-pending and using Taskcluster/... the better! (Many of these issues stem from the hoops we have to jump through due to data available to us being suboptimal; TBPL worked around this by just not bothering to deal with pending/running server-side, which was effective, but limiting in many other ways).

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #7)
> So for 7 of the 11 jobs, we're seeing bug 1125476.

We've recently found and fixed a significant subset of this bug - bug 1162526.

> For one more job
> (https://treeherder.mozilla.org/#/jobs?repo=try&revision=e22656e8151c&filter-
> searchStr=Windows%208%20x64%20debug%20Build%20%28B%29), there is a
> ghost/duplicate completed job (ie we've not correctly associated the running
> and completed jobs, which will be due to bug 1093743, which has a patch
> awaiting review).

This has now landed.

The situation should be much improved from now on - sorry for the issues until now! I'll also file a bug for manually updating the DB to rescue the jobs stuck in the 'loading' state prior to the fix for bug 1162526 landing.