986990 - Modify test channel names to support releases on Balrog

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

There's some history in bug 407435, but the requirements have shifted a little bit for Balrog. 

Whenever we use betatest/releasetest for beta *and* release builds we'll have trouble with Balrog:
* RC builds - release builds go on beta channel for stability checking, the meaning of betatest depends on what you start with (in a way which can't be written compactly in Balrog rules)
* OS blocking - we will want to block deprecated OS in beta test-channels before release

We could do something like this:
   release       -->   release
   betatest      -->   release-localtest
   releasetest   -->   release-cdntest

   beta          -->   beta
   betatest      -->
   releasetest   -->   beta-cdntest

   esr           -->   esr
   esrtest       -->  
   releasetest   -->   esr-cdntest

betatest & esrtest aren't useful if we've automatically got the updates available via download.m.o and CDN. beta-cdntest would be used for both regular betas and the RC situation, we're going via bouncer & CDN either way.

We don't need this before beta on Balrog, in fact much more convenient after that transition. We do need it for two or more of (beta,release,esr) on Balrog though.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Another proposal could be to use releasetest-ftp for updates served via our ftp server, and add releasetest-cdn for release builds. Similar for the other build types. That way we emphasize the test nature of the channel while making it clear where the updates are coming from.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That would be fine too. 

I didn't say it directly in comment #0, but I don't want to use any variation on releasetest for the beta or esr channels.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sure. That was clear for me after reading comment 0! :) It's the same for me, given that the current usage is kinda confusing.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: update rule ids for all release configs

I went through Balrog dev and prod today and updated all of the rules to use the new-style channel names. I didn't add rules for the live release and esr channels (though one existed for release in dev already) for now. This patch updates all of the rule ids in the release config templates to reflect the changes. Here's a couple of queries that might help review:
Dev:
+---------+-------------------+-------------+------------------------------------------+---------+
| rule_id | channel           | product     | mapping                                  | comment |
+---------+-------------------+-------------+------------------------------------------+---------+
|      32 | NULL              | Firefox     | Firefox-32.0b1-build2                    | NULL    |
|      17 | NULL              | FirefoxOS   | FirefoxOS-mozilla-central-nightly-latest | NULL    |
|      12 | aurora            | NULL        | No-Update                                | NULL    |
|       7 | aurora            | Fennec      | Fennec-mozilla-aurora-nightly-latest     | NULL    |
|       2 | aurora            | Firefox     | Firefox-mozilla-aurora-nightly-latest    | NULL    |
|      14 | aurora            | Thunderbird | Thunderbird-comm-aurora-nightly-latest   | NULL    |
|      34 | beta              | Fennec      |                                          | NULL    |
|      26 | beta              | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      33 | beta              | Thunderbird | No-Update                                | NULL    |
|      25 | beta*             | NULL        | No-Update                                | NULL    |
|      36 | beta*             | Firefox     | Firefox-29.0b8-build1-schema2            | NULL    |
|      37 | beta-cdntest      | Fennec      |                                          | NULL    |
|      41 | beta-cdntest      | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      43 | beta-cdntest      | Thunderbird | No-Update                                | NULL    |
|      40 | beta-localtest    | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      21 | beta-localtest    | Thunderbird | No-Update                                | NULL    |
|       3 | elm               | Firefox     | Firefox-elm-nightly-latest               | NULL    |
|      42 | esr-cdntest       | Firefox     | No-Update                                | NULL    |
|      29 | esr-localtest     | Firefox     | No-Update                                | NULL    |
|      28 | nightly           | B2G         | B2G-mozilla-central-nightly-latest       | NULL    |
|       6 | nightly           | Fennec      | Fennec-mozilla-central-nightly-latest    | NULL    |
|       1 | nightly           | Firefox     | Firefox-mozilla-central-nightly-latest   | NULL    |
|      13 | nightly           | Thunderbird | Thunderbird-comm-central-nightly-latest  | NULL    |
|       9 | nightly*          | NULL        | No-Update                                | NULL    |
|       8 | nightly-cedar     | Fennec      | Fennec-cedar-nightly-latest              | NULL    |
|      15 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-1   | NULL    |
|      16 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-2   | NULL    |
|      18 | nightly-holly     | Firefox     | Firefox-31.0.0esr-build1                 | ueue    |
|       5 | nightly-ux        | Firefox     | Firefox-ux-nightly-latest                | NULL    |
|      35 | release           | Fennec      |                                          | NULL    |
|      31 | release           | Firefox     | Firefox-30.0-build2                      | NULL    |
|      24 | release*          | NULL        | No-Update                                | NULL    |
|      39 | release-cdntest   | Fennec      |                                          | NULL    |
|      20 | release-cdntest   | Firefox     | Firefox-30.0-build2                      | NULL    |
|      22 | release-cdntest   | Thunderbird | No-Update                                | NULL    |
|      38 | release-localtest | Fennec      |                                          | NULL    |
|      19 | release-localtest | Firefox     | Firefox-30.0-build2                      | NULL    |
|      44 | release-localtest | Thunderbird | No-Update                                | NULL    |
+---------+-------------------+-------------+------------------------------------------+---------+

Prod:
mysql> select rule_id, channel, product, mapping, substring(comment, 1, 75) as comment from rules order by channel, product;
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
| rule_id | channel           | product      | mapping                                 | comment                                                                     |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
|       2 | aurora            | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Aurora.                            |
|      41 | aurora            | B2G          | B2G-mozilla-aurora-nightly-latest       | NULL                                                                        |
|       8 | aurora            | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      10 | aurora            | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      18 | aurora            | MetroFirefox | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      13 | aurora            | Thunderbird  | Thunderbird-comm-aurora-nightly-latest  | NULL                                                                        |
|      15 | auroratest        | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      16 | auroratest        | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      32 | beta              | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      43 | beta              | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      31 | beta*             | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Beta channel                       |
|      51 | beta*             | Firefox      | Firefox-11.0b5-build1                   | watershed                                                                   |
|      52 | beta*             | Firefox      | Firefox-29.0b8-build1-schema2           | watershed                                                                   |
|      45 | beta-cdntest      | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      60 | beta-cdntest      | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      25 | beta-localtest    | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      27 | beta-localtest    | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      34 | esr               | Firefox      | No-Update                               | NULL                                                                        |
|      36 | esr               | Thunderbird  | No-Update                               | NULL                                                                        |
|      59 | esr-cdntest       | NULL         | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      58 | esr-localtest     | Firefox      | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      19 | nightly           | NULL         | No-Update                               | Global shut off rule. Bump priority of this rule to 100 to shut off all upd |
|      39 | nightly           | B2G          | B2G-mozilla-central-nightly-latest      | disabled because of busted patch making it into latest nightly              |
|       4 | nightly           | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|       3 | nightly           | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      17 | nightly           | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      14 | nightly           | Thunderbird  | Thunderbird-comm-central-nightly-latest | NULL                                                                        |
|       1 | nightly*          | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on all nightly channels.              |
|      42 | nightly-b2g30     | B2G          | B2G-mozilla-b2g30_v1_4-nightly-latest   | NULL                                                                        |
|      47 | nightly-b2g32     | B2G          | B2G-mozilla-b2g32_v2_0-nightly-latest   | NULL                                                                        |
|       6 | nightly-cedar     | Fennec       | Fennec-cedar-nightly-20131107040204     | NULL                                                                        |
|       9 | nightly-elm       | Firefox      | Firefox-elm-nightly-20131221040201      | NULL                                                                        |
|      12 | nightly-esr24     | Firefox      | Firefox-mozilla-esr24-nightly-latest    | NULL                                                                        |
|      44 | nightly-esr31     | Firefox      | Firefox-mozilla-esr31-nightly-latest    | NULL                                                                        |
|      48 | nightly-esr31     | Thunderbird  | Thunderbird-comm-esr31-nightly-latest   | NULL                                                                        |
|      20 | nightly-holly     | Firefox      | Firefox-holly-nightly-20140321040202    | NULL                                                                        |
|       5 | nightly-oak       | Firefox      | Firefox-oak-nightly-latest              | NULL                                                                        |
|       7 | nightly-ux        | Firefox      | Firefox-ux-nightly-20140612040202       | NULL                                                                        |
|      40 | nightlytest       | B2G          | B2G-mozilla-central-nightly-latest      | NULL                                                                        |
|      24 | nightlytest       | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|      22 | nightlytest       | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      23 | nightlytest       | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      33 | release           | Firefox      | No-Update                               | NULL                                                                        |
|      35 | release           | Thunderbird  | No-Update                               | NULL                                                                        |
|      57 | release-cdntest   | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      62 | release-cdntest   | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
|      56 | release-localtest | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      61 | release-localtest | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+

Comment 5

[[e-mail] (:bhearsum)]

Attachment: make final verify support new channel names

Pete, I think comment #0 should give you enough background, but the tl;dr is that we need to support *-{localtest,cdntest} as the respective internal and external testing channels (in addition to betatest/releasetest for awhile). This tweak to final verify's regex should give it support for doing that.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: support new channel names in balrog submitter

This block is used to determine whether we should produce direct-to-ftp or bouncer URLs. localtest is the new direct-to-ftp channel, so we need to include it here.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

Review of attachment 8489419 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the db info, very helpful.

::: mozilla/release-firefox-mozilla-esr31.py.template
@@ +119,5 @@
>  }
>  releaseConfig['releaseChannel']        = 'esr'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
>  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> +releaseConfig['testChannelRuleIds']    = [58,59]

59 needs product set to Firefox, it's NULL at the moment.

::: mozilla/staging_release-thunderbird-comm-esr31.py.template
@@ +131,5 @@
>      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']        = 'release'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3

The other staging configs have a release rule, would be worth adding one to keep things consistent.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

(In reply to Nick Thomas [:nthomas] from comment #7)
> Comment on attachment 8489419 [details] [diff] [review]
> update rule ids for all release configs
> 
> Review of attachment 8489419 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Thanks for the db info, very helpful.
> 
> ::: mozilla/release-firefox-mozilla-esr31.py.template
> @@ +119,5 @@
> >  }
> >  releaseConfig['releaseChannel']        = 'esr'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> >  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> > +releaseConfig['testChannelRuleIds']    = [58,59]
> 
> 59 needs product set to Firefox, it's NULL at the moment.

Done.

> ::: mozilla/staging_release-thunderbird-comm-esr31.py.template
> @@ +131,5 @@
> >      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']        = 'release'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> 
> The other staging configs have a release rule, would be worth adding one to
> keep things consistent.

Sure thing. I added this and landed the patch as https://hg.mozilla.org/build/buildbot-configs/rev/0fa6f60cfa9f

Comment 9

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

Review of attachment 8489488 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Ben,

Thanks for the overview. I can't claim to completely understand how the channels work, but looking at the table, and seeing the *localtest and *cdntest entries, I see that when there is a *localtest entry, there seems to be a corresponding *cdntest entry, and based on your explanation it seems intuitive that one would be for internal (localtest) and the other for external (cdntest) testing. Regarding the validity of the sed expression, I can confirm that it works, and plays well on both mac and linux (which I am happy about - because it keeps final verification completely mac-compatible).

The only thing I see missing is maybe a wiki page explaining the channel names, which I think could be important given the number of channel names and complexity of the setup, especially as this has been a little revised now.

Of course I don't want to block on this, but do you think you could doc something up as an extra artefact for this bug? Maybe a page which just gives a high level explanation about the new channel name strategy? Sorry for the extra work! I would write it myself, but I am one of the people that would need to read it, to understand how it works. :)

Last point was rule_id 18 seems to have an incorrect comment - probably also worth fixing.

Thanks!
Pete

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

(In reply to Pete Moore [:pete][:pmoore] from comment #9)
> Comment on attachment 8489488 [details] [diff] [review]
> make final verify support new channel names
> 
> Review of attachment 8489488 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hi Ben,
> 
> Thanks for the overview. I can't claim to completely understand how the
> channels work, but looking at the table, and seeing the *localtest and
> *cdntest entries, I see that when there is a *localtest entry, there seems
> to be a corresponding *cdntest entry, and based on your explanation it seems
> intuitive that one would be for internal (localtest) and the other for
> external (cdntest) testing. Regarding the validity of the sed expression, I
> can confirm that it works, and plays well on both mac and linux (which I am
> happy about - because it keeps final verification completely mac-compatible).
> 
> The only thing I see missing is maybe a wiki page explaining the channel
> names, which I think could be important given the number of channel names
> and complexity of the setup, especially as this has been a little revised
> now.

I created https://wiki.mozilla.org/Releases/Update_Channels just a few days ago - is that the kind of thing you're looking for?

> Last point was rule_id 18 seems to have an incorrect comment - probably also
> worth fixing.

That's a rule from the dev server, I'm not going to worry about it...

Comment 11

[[e-mail] (:bhearsum)]

Attachment: support new test channel variables in balrog pusher

Comment 12

[[e-mail] (:bhearsum)]

Attachment: support new test channel substitutions in release mail; use local test channel for update verify

Because we have different channels for different release streams, some scripts and tools (final verify, update verify config bumper, etc.) can't easily figure out which channels to use. Rather than adding hardcodes to all of those tools, changing our release configs like this lets us do the right thing in all places without hardcodes.

I did a bunch of staging tests with this (and the other) patches and everything looked fine. Specifically, I ran the updates, final verify, and ready for releasetest builders of Firefox+Thunderbird Beta+Release - everything seemed fine there (modulo some staging specific issues).

Comment 13

[[e-mail] (:bhearsum)]

Attachment: switch to new test channel variables

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Review of attachment 8490207 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with minor tweak.

::: mozilla/staging_release-thunderbird-comm-esr24.py.template
@@ +138,5 @@
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']      = 'release'
> +releaseConfig['localTestChannel']    = 'betatest'
> +releaseConfig['cdnTestChannel']      = 'releasetest'

These are empty in the production config, because updates are gone for esr24 ?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Review of attachment 8490206 [details] [diff] [review]:
-----------------------------------------------------------------

Scratch everything I said on IRC about snippet comparison, got the wrong end of the stick.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #14)
> Comment on attachment 8490207 [details] [diff] [review]
> switch to new test channel variables
> 
> Review of attachment 8490207 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with minor tweak.
> 
> ::: mozilla/staging_release-thunderbird-comm-esr24.py.template
> @@ +138,5 @@
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']      = 'release'
> > +releaseConfig['localTestChannel']    = 'betatest'
> > +releaseConfig['cdnTestChannel']      = 'releasetest'
> 
> These are empty in the production config, because updates are gone for esr24
> ?

I'll change this for staging on check-in - thanks!

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #15)
> Comment on attachment 8490206 [details] [diff] [review]
> support new test channel substitutions in release mail; use local test
> channel for update verify
> 
> Review of attachment 8490206 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Scratch everything I said on IRC about snippet comparison, got the wrong end
> of the stick.

Just to be safe I ended up doing full runs of the updates builder for mozilla-beta and mozilla-release on my dev-master -- they both went totally fine.

Comment 18

[Justin Wood (:Callek)]

In production

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Landed this, including a transplant to the production-0.8 branch so that it will get picked up with 33.0b5 today.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Landed this, including a transplant to the production branch so that it's picked up with 33.0b5 today.

Comment 21

[[e-mail] (:bhearsum)]

I had to land a follow-up to put the new variables in the actual release configs (instead of just the templates), otherwise the reconfig would've busted:
https://hg.mozilla.org/build/buildbot-configs/rev/86f1fc6aff10

Comment 22

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-18 08:14 PT

Comment 23

[[e-mail] (:bhearsum)]

Everything we need to do for Beta is done now. I'm going to leave this bug open to continue to track the migration when we put Release and ESR on Balrog.

Comment 24

[[e-mail] (:bhearsum)]

Attachment: switch release configs to new rules/channels

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 8546710 [details] [diff] [review]
switch release configs to new rules/channels

Review of attachment 8546710 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_release-fennec-mozilla-release.py.template
@@ +163,3 @@
>  releaseConfig['partialUpdates']      = {}
>  releaseConfig['bouncerServer']       = 'download.mozilla.org'
>  releaseConfig['releaseChannelRuleIds'] = []

I updated my local branch to include a release channel rule id, which now exists.

Comment 26

[[e-mail] (:bhearsum)]

Only thing left to do here is mozilla-esr31, which will happen at the same time as bug 1117959 (hopefully in the next week or two).

Comment 27

[[e-mail] (:bhearsum)]

Attachment: switch esr templates to new channels, add rule ids

The last ones!

Comment 28

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/70a8291cfe60

Comment 29

[[e-mail] (:bhearsum)]

With the ESR changes landed we're all done here!

Comment 30

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/8830cc5285d1