837951 - self serve causing "wrong branch" builders to be started

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[u429623]

aki kicked a self server job on b2g18 that ended up scheduling b2g18_v1_0_1 jobs (which failed, since the changeset doesn't exist on that branch)

Data from selfserve ui:
    [e-mail]	new_nightly_at_revision

    {u'priority': 0, u'branch': u'mozilla-b2g18', u'revision': u'450fd3b8496b'}

            2013-02-04 12:36:23	2013-02-04 12:36:50	

    {u'body': {u'msg': u'Ok', u'errors': False}, u'request_id': 126475}

Which led to running jobs showing:
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_unagi_eng_nightly 	2013-02-04 14:32:27 	2013-02-04 14:32:28 	1:34:12 	buildbot-master35:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_ics_armv7a_gecko nightly 	2013-02-04 12:36:27 	2013-02-04 12:46:58 	3:19:42 	buildbot-master35:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_linux32_gecko nightly 	2013-02-04 12:36:27 	2013-02-04 12:45:03 	3:21:37 	buildbot-master49:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_linux32_gecko_localizer nightly 	2013-02-04 12:36:27 	2013-02-04 12:43:28 	3:23:12 	buildbot-master49:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_otoro_nightly 	2013-02-04 12:36:27 	2013-02-04 12:41:33 	3:25:07 	buildbot-master49:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_unagi_nightly 	2013-02-04 12:36:27 	2013-02-04 12:41:53 	3:24:47 	buildbot-master49:8001
    mozilla-b2g18	450fd3b8496b 	b2g_mozilla-b2g18_v1_0_1_win32_gecko nightly 	2013-02-04 12:36:27 	2013-02-04 12:40:13 	3:26:27 	buildbot-master13:8001

Comment 1

[u429623]

fwiw - in investigating this, I noticed that b2g18_v1_0_1 wasn't in self serve. I just checked and the enabling patch did land many days ago: http://hg.mozilla.org/build/tools/rev/7d9ff4098459

Comment 2

[Chris AtLee [:catlee]]

(In reply to Hal Wine [:hwine] from comment #1)
> fwiw - in investigating this, I noticed that b2g18_v1_0_1 wasn't in self
> serve. I just checked and the enabling patch did land many days ago:
> http://hg.mozilla.org/build/tools/rev/7d9ff4098459

strange...

https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-b2g18_v1_0_1

doesn't error out. sounds like a caching problem? anyway...

the main problem here is this code:
http://hg.mozilla.org/build/buildapi/file/a995ca23cd0a/buildapi/scripts/selfserve-agent.py#l459
is called with branch='mozilla-b2g18'

so we call _create_build_for_revision with builder_expression = "%mozilla-b2g18%nightly", which of course matches all the other mozilla-b2g18* repositories too.

Comment 3

[u429623]

well, we'll need to fix this before esr100 then!

Actually, aren't all builder names separated by '-'? so, wouldn't a better SQL query be '%mozilla-b2g18-%nightly'?

Comment 4

[John Hopkins (:jhopkins)]

mysql> SELECT DISTINCT buildername FROM buildrequests WHERE buildername LIKE '%mozilla-b2g18%nightly';
+-----------------------------------------------------------+
| buildername                                               |
+-----------------------------------------------------------+
| Linux mozilla-b2g18 nightly                               | 
| Linux x86-64 mozilla-b2g18 nightly                        | 
| OS X 10.7 mozilla-b2g18 nightly                           | 
| WINNT 5.2 mozilla-b2g18 nightly                           | 
| b2g_mozilla-b2g18_ics_armv7a_gecko nightly                | 
| b2g_mozilla-b2g18_linux32_gecko nightly                   | 
| b2g_mozilla-b2g18_linux32_gecko_localizer nightly         | 
| b2g_mozilla-b2g18_linux64_gecko nightly                   | 
| b2g_mozilla-b2g18_linux64_gecko_localizer nightly         | 
| b2g_mozilla-b2g18_macosx64_gecko nightly                  | 
| b2g_mozilla-b2g18_macosx64_gecko_localizer nightly        | 
| b2g_mozilla-b2g18_otoro_nightly                           | 
| b2g_mozilla-b2g18_unagi_nightly                           | 
| b2g_mozilla-b2g18_unagi_stable_nightly                    | 
| b2g_mozilla-b2g18_win32_gecko nightly                     | 
| b2g_mozilla-b2g18_win32_gecko_localizer nightly           | 
| b2g_mozilla-b2g18_unagi_eng_nightly                       | 
| b2g_mozilla-b2g18_v1_0_0_ics_armv7a_gecko nightly         | 
| b2g_mozilla-b2g18_v1_0_0_linux32_gecko nightly            | 
| b2g_mozilla-b2g18_v1_0_0_linux32_gecko_localizer nightly  | 
| b2g_mozilla-b2g18_v1_0_0_linux64_gecko nightly            | 
| b2g_mozilla-b2g18_v1_0_0_linux64_gecko_localizer nightly  | 
| b2g_mozilla-b2g18_v1_0_0_macosx64_gecko nightly           | 
| b2g_mozilla-b2g18_v1_0_0_macosx64_gecko_localizer nightly | 
| b2g_mozilla-b2g18_v1_0_0_unagi_nightly                    | 
| b2g_mozilla-b2g18_v1_0_0_win32_gecko nightly              | 
| b2g_mozilla-b2g18_v1_0_0_win32_gecko_localizer nightly    | 
| Linux mozilla-b2g18_v1_0_0 nightly                        | 
| Linux x86-64 mozilla-b2g18_v1_0_0 nightly                 | 
| OS X 10.7 mozilla-b2g18_v1_0_0 nightly                    | 
| WINNT 5.2 mozilla-b2g18_v1_0_0 nightly                    | 
| b2g_mozilla-b2g18_v1_0_0_otoro_nightly                    | 
| Android no-ionmonkey mozilla-b2g18 nightly                | 
| b2g_mozilla-b2g18_v1_0_1_macosx64_gecko nightly           | 
| b2g_mozilla-b2g18_v1_0_1_win32_gecko nightly              | 
| b2g_mozilla-b2g18_v1_0_1_unagi_eng_nightly                | 
| b2g_mozilla-b2g18_v1_0_1_unagi_nightly                    | 
| b2g_mozilla-b2g18_v1_0_1_otoro_nightly                    | 
| b2g_mozilla-b2g18_v1_0_1_macosx64_gecko_localizer nightly | 
| b2g_mozilla-b2g18_v1_0_1_linux64_gecko_localizer nightly  | 
| b2g_mozilla-b2g18_v1_0_1_linux64_gecko nightly            | 
| b2g_mozilla-b2g18_v1_0_1_linux32_gecko_localizer nightly  | 
| b2g_mozilla-b2g18_v1_0_1_linux32_gecko nightly            | 
| b2g_mozilla-b2g18_v1_0_1_ics_armv7a_gecko nightly         | 
| Android no-ionmonkey mozilla-b2g18_v1_0_0 nightly         | 
+-----------------------------------------------------------+

The simplest thing I can think of is to add a 'NOT LIKE' for |${buildername}_v| (note the _v suffix).  This would filter out mozilla-b2g18_v when passing in mozilla-b2g18 and filter out mozilla-b2g18_v1_0_1_v when passing in mozilla-b2g18_v1_0_1.

Perhaps the 'builder_exclusion' parameter could be leveraged for this.

Note: this solution would only work until we add a platform starting with 'v'.  I suppose we could use r'_v\d' to look for a 'v' followed by a digit.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

jhopkins, hwine: unclear if this impacts self-serv after tomorrow's v1.0.1 branch. If yes, can the suggestion in comment#4 be landed tomorrow, so we have something working in time for the branching? If no, please remove blocking bug#840830.

(open to alternate suggestions also, btw)

Comment 6

[u429623]

:jhopkins - additional info - according to aki, this is causing glitches in what partners are seeing as available in "latest" b2g links. So it's hurting us as long as mozilla-b2g18 branch is still alive & in self serve.

Comment 7

[John Hopkins (:jhopkins)]

Attachment: [buildapi] allow multiple buildername patterns to be excluded, add {buildername}_v to exclusion

Comment 8

[Justin Wood (:Callek)]

Comment on attachment 713585 [details] [diff] [review]
[buildapi] allow multiple buildername patterns to be excluded, add {buildername}_v to exclusion

Review of attachment 713585 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildapi/scripts/selfserve-agent.py
@@ +375,5 @@
>          log.info("Running %s", cmd)
>          subprocess.check_call(cmd)
>          return {"errors": False, "msg": "Ok"}
>  
> +    def _create_build_for_revision(self, who, branch, revision, priority, builder_expression, builder_exclusions=None):

the changes here look messy, and if this wasn't urgent I'd help coordinate a more please bikeshed color, but since this is urgent we can paint it this color, since the paint will work.

@@ +461,5 @@
>                      branch,
>                      revision,
>                      priority,
> +                    "%% %s pgo-build" % branch,
> +                    ["%% %s_v pgo-build" % branch, '%l10n nightly'])

don't we want _v%% here?

Comment 9

[John Hopkins (:jhopkins)]

Attachment: [buildapi] address callek's feedback

Remove 'pgo build' hunk since there was no wildcard there to begin with. Carrying forward r+

Comment 10

[John Hopkins (:jhopkins)]

Comment on attachment 713617 [details] [diff] [review]
[buildapi] address callek's feedback

Landed in http://hg.mozilla.org/build/buildapi/rev/45500847fd99 and deployed to production.  Created a mozilla-b2g18 nightly build via self-serve and hwine confirmed it looked good as expected.