750611 - self-serve not triggering PGO builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Phil Ringnalda (:philor)]

https://build.mozilla.org/buildapi/self-serve/jobs/4394 and https://secure.pub.build.mozilla.org/buildapi/self-serve/jobs/4392 are requests to trigger PGO builds on the rev before and the rev which I think caused the current tree-closing Windows PGO bustage.

Neither shows any sign of actually triggering builds, and when I asked about that in #build, I was told that someone mentioned that being a problem yesterday.

Now that it's sitting between me and potentially reopening inbound, it seems like more of a problem.

Comment 1

[Phil Ringnalda (:philor)]

https://build.mozilla.org/buildapi/self-serve/jobs/4405 and https://build.mozilla.org/buildapi/self-serve/jobs/4406 are well on their way to making me think that new_nightly_at_revision doesn't work either.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

From the selfserve-agent log:

2012-04-30 22:29:02,763 New PGO build by [e-mail] of mozilla-inbound 32e001c1351b
2012-04-30 22:29:02,764 ,
Traceback (most recent call last):
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 160, in receive_message
    retval = action_func(message_data, message)
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 447, in do_new_pgobuild_at_revision
    "%% %s pgo-build" % branch)
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 373, in _create_build_for_revision
    repo_path = self._get_repo_path(branch)
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 64, in _get_repo_path
    return self.branch_map[branch].repo_path
KeyError: u'mozilla-inbound'
2012-04-30 22:34:41,583 Received {u'action': u'rebuild_build', u'body': {u'priority': 0, u'bid': 11046245, u'when': 1335850481.787976, u'request_id': 4395}, u'who': u'[e-mail]'}

Same error for the nightly, and for several other attempts going back to 2012-04-26 06:20:53.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

So the buildapi checkout in buildbot-master10:/builds/buildbot/selfserve-agent is at rev 7fe7135f3633 (Dec 21), which is an issue because we have a config file written for the change in http://hg.mozilla.org/build/buildapi/diff/432963d90f6e/buildapi/scripts/selfserve-agent.py but not the actual code.

I've updated to rev 432963d90f6e and restarted the agent. All the pending tasks must have still been in the message queue, because we started in short order:

2012-05-01 01:03:38,642 New nightly by [e-mail] of oak 27d8078fe517
2012-05-01 01:03:50,098 New PGO build by [e-mail] of mozilla-inbound eb4b76b2840f
2012-05-01 01:04:03,132 New nightly by [e-mail] of mozilla-inbound 2949b3533041
2012-05-01 01:04:15,476 New PGO build by [e-mail] of mozilla-inbound 5b2cef6c4c34
2012-05-01 01:04:27,661 New build by [e-mail] of mozilla-inbound pgo
2012-05-01 01:04:28,014 New PGO build by [e-mail] of mozilla-inbound 043266d76bb3
2012-05-01 01:04:39,982 New PGO build by [e-mail] of mozilla-inbound f99cf2f41355
2012-05-01 01:04:52,707 New PGO build by [e-mail] of mozilla-inbound f99cf2f41355
2012-05-01 01:05:05,465 New nightly by [e-mail] of ux 4ee845908e43
2012-05-01 01:05:15,869 New nightly by [e-mail] of elm d105c5b9bbe9
2012-05-01 01:05:26,929 New nightly by [e-mail] of ux 4ee845908e43
2012-05-01 01:05:39,455 New PGO build by [e-mail] of mozilla-inbound ac1504ff8740
2012-05-01 01:05:51,920 New PGO build by [e-mail] of mozilla-inbound ac1504ff8740
2012-05-01 01:06:04,359 New PGO build by [e-mail] of mozilla-inbound c0822f99d850
2012-05-01 01:06:15,690 New PGO build by [e-mail] of mozilla-inbound 32e001c1351b
2012-05-01 01:06:28,774 New nightly by [e-mail] of mozilla-inbound 32e001c1351b
2012-05-01 01:06:42,026 New nightly by [e-mail] of mozilla-inbound c0822f99d850

I sincerly hope merging will save us.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The original request times were:
2012-04-26 06:20:53,086 New nightly by [e-mail] of oak 27d8078fe517
2012-04-27 01:34:05,987 New PGO build by [e-mail] of mozilla-inbound eb4b76b2840f
2012-04-27 14:53:43,679 New nightly by [e-mail] of mozilla-inbound 2949b3533041
2012-04-30 01:32:32,358 New PGO build by [e-mail] of mozilla-inbound 5b2cef6c4c34
2012-04-30 03:59:51,212 New build by [e-mail] of mozilla-inbound pgo
2012-04-30 03:59:56,272 New PGO build by [e-mail] of mozilla-inbound 043266d76bb3
2012-04-30 05:23:06,968 New PGO build by [e-mail] of mozilla-inbound f99cf2f41355
2012-04-30 05:52:59,531 New PGO build by [e-mail] of mozilla-inbound f99cf2f41355
2012-04-30 12:22:19,922 New nightly by [e-mail] of ux 4ee845908e43
2012-04-30 18:57:47,458 New nightly by [e-mail] of elm d105c5b9bbe9
2012-04-30 20:17:42,995 New nightly by [e-mail] of ux 4ee845908e43
2012-04-30 21:17:44,697 New PGO build by [e-mail] of mozilla-inbound ac1504ff8740
2012-04-30 22:02:04,262 New PGO build by [e-mail] of mozilla-inbound ac1504ff8740
2012-04-30 22:22:24,323 New PGO build by [e-mail] of mozilla-inbound c0822f99d850
2012-04-30 22:29:02,763 New PGO build by [e-mail] of mozilla-inbound 32e001c1351b
2012-04-30 22:56:46,831 New nightly by [e-mail] of mozilla-inbound 32e001c1351b
2012-04-30 22:58:01,667 New nightly by [e-mail] of mozilla-inbound c0822f99d850

Comment 5

[Justin Wood (:Callek)]

To new CC's... Just an FYI that some nightlies/pgo's you triggered via self-serve in the last week were not actually triggered until just now... See previous comments

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've cancelled the following

> 2012-04-26 06:20:53,086 New nightly by [e-mail] of oak
> 27d8078fe517
> 2012-04-27 14:53:43,679 New nightly by [e-mail] of
> mozilla-inbound 2949b3533041   (bogus rev!)
> 2012-04-30 12:22:19,922 New nightly by [e-mail] of ux 4ee845908e43

The last was duplicated, so we got two sets of nightlies.

Comment 7

[Ed Morley [:emorley]]

Ahhh so I wasn't going crazy yesterday. 

I asked in #build whether there were any known issues with the buildapi, since my PGO retriggers were not appearing. However, when I looked back later, some PGO builds had eventually appeared, so I thought all was well - but they must have just been periodic builds on tip (which was where I was triggering).