846104 - Self-serve nightly wildcard matching is not strict enough (can't retrigger UX nightlies)

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

MattN triggered nightlies for the UX branch, like he always does, and it triggered every sort of nightly claiming that they were both on a UX revision and for another tree, and when he killed them, they uploaded the aborted log to the other trees, as though they really were mozilla-central and mozilla-esr and mozilla-b2g18 nightlies, e.g. ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-02-27-13-14-44-mozilla-esr17 and ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-02-27-13-14-44-mozilla-b2g18 and ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-02-27-13-14-44-mozilla-central.

Given the uncertainty about whether that would have ended up moving users from those branches onto the ux branch, we really need to shut off the ability for anyone to trigger any nightlies, immediately.

Comment 1

[Matthew N. [:MattN]]

Attachment: Screenshot of UX TBPL

To be clear, I hadn't manually triggered UX nightlies for a few months so I don't know whether this is a new issue.

OS X and Windows appeared to be unaffected on TBPL.

Comment 2

[Justin Wood (:Callek)]

fwiw looking at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-02-27-13-14-44-mozilla-b2g18/mozilla-b2g18-linux64-nightly-bm35-build1-build25.txt.gz it actually looks like it was failing to pull/clone due to the wrong revision (understandable given it was meant for UX) but does seem to have been infinitely retrying until the job was killed.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

triaged after ping in irc.


(In reply to Matthew N. [:MattN] from comment #1)
> Created attachment 719263 [details]
> Screenshot of UX TBPL
> 
> To be clear, I hadn't manually triggered UX nightlies for a few months so I
> don't know whether this is a new issue.
Multiple months... hmm... wonder if there is something UX-branch specific broken? 

MattM: can you give explicit steps on what you did to trigger the nightlies, and what you did to cancel them?



> OS X and Windows appeared to be unaffected on TBPL.

The concern is that this could be a problem for tonight's nightlies. We believe last night's nightlies were ok. 

Did anything change in our infrastructure today which could be involved?

Comment 4

[Matthew N. [:MattN]]

(In reply to John O'Duinn [:joduinn] from comment #3)
> (In reply to Matthew N. [:MattN] from comment #1)
> MattN: can you give explicit steps on what you did to trigger the nightlies,
> and what you did to cancel them?

1) Notice today's UX Nightly (on 4ec2b1c83ed0) has a bug  
2) Push a fix (https://hg.mozilla.org/projects/ux/rev/fc287e8c97a1 )
3) Waited a few minutes and then went to https://secure.pub.build.mozilla.org/buildapi/self-serve/ux/rev/fc287e8c97a1
3) Pasted fc287e8c97a1 in the "Create new nightly builds on ux revision" field and clicked Submit
4) Went back to work on other things for a while and then checked the UX TBPL tab
5) Noticed way more nightlies than I expected beside Fedora *.
6) Asked for advice in #releng at 14:46:47 PST. Philor replies at 14:56:09 PST
7) Clicked cancel on the pending and running non-UX builds at https://secure.pub.build.mozilla.org/buildapi/self-serve/ux/rev/fc287e8c97a1 

> > OS X and Windows appeared to be unaffected on TBPL.
> The concern is that this could be a problem for tonight's nightlies. We
> believe last night's nightlies were ok. 

The other concern is that this will happen again on manually triggered nightlies on any tree and that the self-serve UI for it should be removed until we know more.  This could be as simple as commenting out the HTML on the self-serve page.

Comment 5

[Aki Sasaki (not active)]

Right now my hypothesis is that our wildcard matching in self-serve is making %ux%nightly match all Linux*nightly jobs.

Not sure whether http://hg.mozilla.org/build/buildapi/rev/45500847fd99 affected this or not.

If this is true, I think this is specific to the ux branch only.

Comment 6

[Aki Sasaki (not active)]

I think this bug was introduced in http://hg.mozilla.org/build/buildapi/rev/0bb3e16ba8cb , only for the ux branch.

Comment 7

[Aki Sasaki (not active)]

I think we have to allow for [ -_] on either side of the branch name, but I'm not sure how to do that in a sql query.  I know how in regexes.

Comment 8

[Matthew N. [:MattN]]

(In reply to Aki Sasaki [:aki] from comment #5)
> Right now my hypothesis is that our wildcard matching in self-serve is
> making %ux%nightly match all Linux*nightly jobs.

That seems reasonable and aligns with the fact that I didn't notice extra non-Linux nightlies being built. I guess the priority of this can probably be lowered if someone updates the tree status on the UX repo to tell people not to trigger nightlies. I'll notify jwein and mconley directly about this too.

Comment 9

[Matthew N. [:MattN]]

(In reply to Matthew N. [:MattN] from comment #8)
> I guess the priority of this can probably be lowered…

if that change only affects self-serve and not the scheduled nightlies.

Comment 10

[Aki Sasaki (not active)]

Correct, it only affects self-serve.
Not sure if we can add treestatus to UX since it's not in the treestatus page.

Comment 11

[Aki Sasaki (not active)]

N/m, added. https://tbpl.mozilla.org/?tree=UX

Comment 12

[Matthew N. [:MattN]]

(In reply to Aki Sasaki [:aki] from comment #11)
> N/m, added. 

Thanks. I sent my email too.

Comment 13

[Matthew N. [:MattN]]

I think this also mean that if a Nightly is triggered on "Mozilla-B2g18", one will also be created on "Mozilla-B2g18-v1.0.1" because it is a substring of the former repo name?

Comment 14

[Phil Ringnalda (:philor)]

You'd think I'd remember that, since we've actually hit that very same overmatching before, but The Fear overtook me first.

Comment 15

[Aki Sasaki (not active)]

Nope, we actually explicitly look for BRANCH_v and exclude that.

I propose we put spaces around the branch name for b2g builds and back out these two self-serve patches.

Comment 16

[Aki Sasaki (not active)]

Changing base_name in mozilla/b2g_config.py to replace all underscores with spaces gave me this:

  File "/src/selfservespace/buildbotcustom/misc.py", line 1025, in generateBranchObjects
    'slavebuilddir': normalizeName('%s_dep' % pf['base_name']),
  File "/src/selfservespace/buildbotcustom/common.py", line 160, in normalizeName
    raise ValueError(msg)
ValueError: Cannot shorten "b2g mozilla-b2g18 unagi_eng_dep" to maximum length (30). Got to: b2g mozilla-b2g18 unagi_eng_dep

So easier said than done.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

I poked at this briefly, the following SQL snippets match all 49 of our nightly builders on mozilla-central (where we have b2g and desktop enabled):
   WHERE buildername LIKE '%\_mozilla-central\_%nightly'    # b2g
   WHERE buildername LIKE '% mozilla-central %nightly%'     # desktop + android

The b2g case has to handle the two forms
  b2g_<branch>_<platform> nightly             # desktop b2g
  b2g_<branch>_<platform>_nightly             # device b2g
NB: escaping of _ since that's a single char match with LIKE.

For rest, everything ends ' nightly' except the likes of'Android mozilla-central l10n nightly 1/5', which is the trailing %. XULRunner and ASAN builds push you towards 'mozilla-central % nightly', but builds like 'Android 2.2 mozilla-central nightly' only have a single space between branch and nightly.

I'm not going any further with this right now because of other priorities. To make this work self-serve-agent.py's _create_build_for_revision() may need to take a list of builder_expression instead of a single one.

Comment 18

[:Gijs (he/him)]

Attachment: self-serve nightly wildcard matching should be stricter

I *think* this is what comment 17 was getting at, but I'm totally not at home in these waters. I just want to be able to run nightly builds when necessary... :-)

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 809482 [details] [diff] [review]
self-serve nightly wildcard matching should be stricter

Overall looks good, but this should definitely get tested on a staging instance of buildapi by someone in RelEng.

>     def do_new_nightly_at_revision(self, message_data, message):
...
>-            '%' + branch + '%nightly',
>+            ['%\_' + branch + '\_%nightly', '% ' + branch + ' nightly%'],

I was expecting this line to end:   ... branch + ' %nightly%'],
ie a % prefixing nightly. Did you drop that for some reason ?

>             ['%' + branch + '_v%nightly', '%l10n nightly'])

I hadn't paid attention to this exclusion list when writing comment #17. The interaction of this with the inclusions is important because we only want to create en-US nightlies, and let them trigger the dependent l10n builds. In particular we need to make sure the right thing happens for the likes of 'Android 2.2 mozilla-central l10n nightly 1/5'.

Comment 20

[:Gijs (he/him)]

Attachment: self-serve nightly wildcard matching should be stricter

No, I just screwed up when copying from comment 17, good catch. Here's an updated patch, and I'll try to find someone in releng to test this.

Comment 21

[:Gijs (he/him)]

Comment on attachment 810497 [details] [diff] [review]
self-serve nightly wildcard matching should be stricter

Carrying over f+

Comment 22

[:Gijs (he/him)]

Chris, word in #releng is that you may be able to help test this on buildapi staging?

Comment 23

[Chris Cooper [:coop] (he/him)]

Sorry, missed the NEEDINFO and then was out on PTO.

Sadly, we don't have an official buildapi staging instance to test this out, but the instructions for setting one up seem reasonably complete. Let me talk to people in releng and see if someone already has such an env setup and can validate your change.

Comment 24

[:Gijs (he/him)]

(In reply to Chris Cooper [:coop] from comment #23)
> Sorry, missed the NEEDINFO and then was out on PTO.
> 
> Sadly, we don't have an official buildapi staging instance to test this out,
> but the instructions for setting one up seem reasonably complete. Let me
> talk to people in releng and see if someone already has such an env setup
> and can validate your change.

Ping?

Comment 25

[Chris Cooper [:coop] (he/him)]

(In reply to :Gijs Kruitbosch from comment #24) 
> Ping?

I don't know if you've had a peek at https://wiki.mozilla.org/ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI, but getting a staging env setup is not straightforward. I didn't get any takers when I polled releng last week, so I've been trying to get the env setup myself. I just made some db slices today that I should be able to use for testing, provided the rest of the env cooperates.

I also filed bug 937781 to get a proper staging environment setup for buildapi.

Comment 26

[:Gijs (he/him)]

(In reply to Chris Cooper [:coop] from comment #25)
> (In reply to :Gijs Kruitbosch from comment #24) 
> > Ping?
> 
> I don't know if you've had a peek at
> https://wiki.mozilla.org/ReleaseEngineering/BuildAPI/
> Setup_Local_Virtualenv_for_BuildAPI, but getting a staging env setup is not
> straightforward. I didn't get any takers when I polled releng last week, so
> I've been trying to get the env setup myself. I just made some db slices
> today that I should be able to use for testing, provided the rest of the env
> cooperates.

Oh, wow. That is crazy. Thank you for taking the time. Sorry if I'm naggy, but every time I've wanted to retrigger UX nightlies I go back and look at this bug and grumble. It's annoying not to be able to do anything about how/when these builds are created. :-(

> I also filed bug 937781 to get a proper staging environment setup for
> buildapi.

Awesome, I'll follow along there, too.

Comment 27

[Chris Cooper [:coop] (he/him)]

(In reply to :Gijs Kruitbosch from comment #26) 
> Oh, wow. That is crazy. Thank you for taking the time. Sorry if I'm naggy,
> but every time I've wanted to retrigger UX nightlies I go back and look at
> this bug and grumble. It's annoying not to be able to do anything about
> how/when these builds are created. :-(

No worries. You've found a real bug in the parsing and I'd love to get this fixed. It's proving harder than I imagined though.

Comment 28

[Chris Cooper [:coop] (he/him)]

Comment on attachment 810497 [details] [diff] [review]
self-serve nightly wildcard matching should be stricter

After getting rabbitmq setup locally, I can verify that the patch doesn't break buildapi, and does actually submit only new nightlies for the ux branch when used in that context.

I'd still like catlee to take a look since he has a broader understanding of buildapi and what else this patch might tickle.

Comment 29

[Chris Cooper [:coop] (he/him)]

Comment on attachment 810497 [details] [diff] [review]
self-serve nightly wildcard matching should be stricter

Review of attachment 810497 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildapi/rev/0b68a3d395b3

Comment 30

[Chris Cooper [:coop] (he/him)]

Buildapi should pick up the code automatically within the next 30min.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 810497 [details] [diff] [review]
self-serve nightly wildcard matching should be stricter

Backed out in https://hg.mozilla.org/build/buildapi/rev/e7baa3e519f6 for bug 975191.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1325371 may have made a difference here, but UX is dead now so it'll be hard to tell.