1159279 - Buildbot job did not make it into any buildjson file

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Something happened on the "buildbot status --> buildjson file" generation.
TH shows the job as still running [1]
self-serve believes it has completed [2] (search for "Linux mozilla-aurora build")
If you open the self-serve status [3], you will see it as completed.
If you open the x86-64 job [4], you will see that the data of both have similar structure (no missing keys or values).
We even see talos jobs that were triggered by it.
I can even see the job completed on buildbot [5]

Somehow, the data did not make it into buildjson file and that is why mozci and TH fail to query more information about it.

I have also checked the 26th and 27th files even though I know that the job is not to be there.

Could we please query the status db?

I know we're moving to TC and we probably don't want to fix this, however, I believe we should at least investigate it to find the root cause.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=61d78164c214&filter-searchStr=Linux%20mozilla-aurora%20build
[2] https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-aurora/rev/61d78164c214
[3] https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-aurora/build/69366760
[4] https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-aurora/build/69366768
[5] http://buildbot-master71.bb.releng.use1.mozilla.com:8001/builders/Linux%20mozilla-aurora%20build/builds/150

Contents of [3]
{
  "build_id": 69366760,
  "buildnumber": 150,
  "status": 0,
  "starttime": 1429958921,
  "branch": "releases/mozilla-aurora",
  "buildername": "Linux mozilla-aurora build",
  "requests": [
    {
      "complete": 1,
      "buildername": "Linux mozilla-aurora build",
      "claimed_at": 1429966574,
      "reason": "scheduler",
      "properties": {
        "buildid": "20150425033517",
        "scheduler": "mozilla-aurora-firefox",
        "builduid": "464a0a6586084421ba5b43493003d42c"
      },
      "complete_at": 1429967139,
      "priority": 0,
      "submittime": 1429958118,
      "branch": "releases/mozilla-aurora",
      "request_id": 68131370,
      "revision": "61d78164c21478d3d6464d645b6be8ea398599cb"
    }
  ],
  "endtime": 1429967139,
  "claimed_by_name": "buildbot-master71.bb.releng.use1.mozilla.com:/builds/buildbot/build1/master",
  "revision": "61d78164c21478d3d6464d645b6be8ea398599cb"
}

Comment 1

[Ed Morley [:emorley]]

Thank you for filing this - might explain a few of the data oddities we've seen in treeherder too.

Comment 2

[Chris AtLee [:catlee]]

I think this is bug 942616 again. Looking at the statusdb directly, I see this job:
select * from builds, builders where builds.builder_id = builders.id and builders.name = 'mozilla-aurora-linux' and builds.buildnumber = 150 order by starttime desc;                        
+----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+--------+----------------------+-----------+----------------+
| id       | buildnumber | builder_id | slave_id | master_id | starttime           | endtime             | result | reason    | source_id | lost | id     | name                 | master_id | category       |
+----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+-----------+-----------+------+--------+----------------------+-----------+----------------+
| 64654622 |         150 |     359584 |     6936 |       175 | 2015-04-25 10:48:41 | 2015-04-25 13:05:36 |      0 | scheduler |   9157297 |    0 | 359584 | mozilla-aurora-linux |       175 | mozilla-aurora |

This build is in builds-2015-04-25 with very minimal information:
    {
      "builder_id": 359584, 
      "buildnumber": 150, 
      "endtime": 1429967136, 
      "id": 64654622, 
      "master_id": 175, 
      "properties": {}, 
      "reason": "scheduler", 
      "request_ids": [], 
      "requesttime": null, 
      "result": 0, 
      "slave_id": 6936, 
      "starttime": 1429958921
    },

Comment 3

[Armen [:armenzg]]

catlee thank you very much for looking.

If we were going to stay with buildbot it would make sense to go ahead and find the issue but there's no good reason for that.