Modify test channel names to support releases on Balrog

RESOLVED FIXED

Status

defect
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: nthomas, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(8 attachments)

There's some history in bug 407435, but the requirements have shifted a little bit for Balrog. 

Whenever we use betatest/releasetest for beta *and* release builds we'll have trouble with Balrog:
* RC builds - release builds go on beta channel for stability checking, the meaning of betatest depends on what you start with (in a way which can't be written compactly in Balrog rules)
* OS blocking - we will want to block deprecated OS in beta test-channels before release

We could do something like this:
   release       -->   release
   betatest      -->   release-localtest
   releasetest   -->   release-cdntest

   beta          -->   beta
   betatest      -->
   releasetest   -->   beta-cdntest

   esr           -->   esr
   esrtest       -->  
   releasetest   -->   esr-cdntest

betatest & esrtest aren't useful if we've automatically got the updates available via download.m.o and CDN. beta-cdntest would be used for both regular betas and the RC situation, we're going via bouncer & CDN either way.

We don't need this before beta on Balrog, in fact much more convenient after that transition. We do need it for two or more of (beta,release,esr) on Balrog though.
Another proposal could be to use releasetest-ftp for updates served via our ftp server, and add releasetest-cdn for release builds. Similar for the other build types. That way we emphasize the test nature of the channel while making it clear where the updates are coming from.
That would be fine too. 

I didn't say it directly in comment #0, but I don't want to use any variation on releasetest for the beta or esr channels.
Sure. That was clear for me after reading comment 0! :) It's the same for me, given that the current usage is kinda confusing.
I went through Balrog dev and prod today and updated all of the rules to use the new-style channel names. I didn't add rules for the live release and esr channels (though one existed for release in dev already) for now. This patch updates all of the rule ids in the release config templates to reflect the changes. Here's a couple of queries that might help review:
Dev:
+---------+-------------------+-------------+------------------------------------------+---------+
| rule_id | channel           | product     | mapping                                  | comment |
+---------+-------------------+-------------+------------------------------------------+---------+
|      32 | NULL              | Firefox     | Firefox-32.0b1-build2                    | NULL    |
|      17 | NULL              | FirefoxOS   | FirefoxOS-mozilla-central-nightly-latest | NULL    |
|      12 | aurora            | NULL        | No-Update                                | NULL    |
|       7 | aurora            | Fennec      | Fennec-mozilla-aurora-nightly-latest     | NULL    |
|       2 | aurora            | Firefox     | Firefox-mozilla-aurora-nightly-latest    | NULL    |
|      14 | aurora            | Thunderbird | Thunderbird-comm-aurora-nightly-latest   | NULL    |
|      34 | beta              | Fennec      |                                          | NULL    |
|      26 | beta              | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      33 | beta              | Thunderbird | No-Update                                | NULL    |
|      25 | beta*             | NULL        | No-Update                                | NULL    |
|      36 | beta*             | Firefox     | Firefox-29.0b8-build1-schema2            | NULL    |
|      37 | beta-cdntest      | Fennec      |                                          | NULL    |
|      41 | beta-cdntest      | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      43 | beta-cdntest      | Thunderbird | No-Update                                | NULL    |
|      40 | beta-localtest    | Firefox     | Firefox-32.0b1-build1                    | NULL    |
|      21 | beta-localtest    | Thunderbird | No-Update                                | NULL    |
|       3 | elm               | Firefox     | Firefox-elm-nightly-latest               | NULL    |
|      42 | esr-cdntest       | Firefox     | No-Update                                | NULL    |
|      29 | esr-localtest     | Firefox     | No-Update                                | NULL    |
|      28 | nightly           | B2G         | B2G-mozilla-central-nightly-latest       | NULL    |
|       6 | nightly           | Fennec      | Fennec-mozilla-central-nightly-latest    | NULL    |
|       1 | nightly           | Firefox     | Firefox-mozilla-central-nightly-latest   | NULL    |
|      13 | nightly           | Thunderbird | Thunderbird-comm-central-nightly-latest  | NULL    |
|       9 | nightly*          | NULL        | No-Update                                | NULL    |
|       8 | nightly-cedar     | Fennec      | Fennec-cedar-nightly-latest              | NULL    |
|      15 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-1   | NULL    |
|      16 | nightly-cert-test | Firefox     | Firefox-mozilla-central-newcert-test-2   | NULL    |
|      18 | nightly-holly     | Firefox     | Firefox-31.0.0esr-build1                 | ueue    |
|       5 | nightly-ux        | Firefox     | Firefox-ux-nightly-latest                | NULL    |
|      35 | release           | Fennec      |                                          | NULL    |
|      31 | release           | Firefox     | Firefox-30.0-build2                      | NULL    |
|      24 | release*          | NULL        | No-Update                                | NULL    |
|      39 | release-cdntest   | Fennec      |                                          | NULL    |
|      20 | release-cdntest   | Firefox     | Firefox-30.0-build2                      | NULL    |
|      22 | release-cdntest   | Thunderbird | No-Update                                | NULL    |
|      38 | release-localtest | Fennec      |                                          | NULL    |
|      19 | release-localtest | Firefox     | Firefox-30.0-build2                      | NULL    |
|      44 | release-localtest | Thunderbird | No-Update                                | NULL    |
+---------+-------------------+-------------+------------------------------------------+---------+

Prod:
mysql> select rule_id, channel, product, mapping, substring(comment, 1, 75) as comment from rules order by channel, product;
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
| rule_id | channel           | product      | mapping                                 | comment                                                                     |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
|       2 | aurora            | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Aurora.                            |
|      41 | aurora            | B2G          | B2G-mozilla-aurora-nightly-latest       | NULL                                                                        |
|       8 | aurora            | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      10 | aurora            | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      18 | aurora            | MetroFirefox | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      13 | aurora            | Thunderbird  | Thunderbird-comm-aurora-nightly-latest  | NULL                                                                        |
|      15 | auroratest        | Fennec       | Fennec-mozilla-aurora-nightly-latest    | NULL                                                                        |
|      16 | auroratest        | Firefox      | Firefox-mozilla-aurora-nightly-latest   | NULL                                                                        |
|      32 | beta              | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      43 | beta              | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      31 | beta*             | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on Beta channel                       |
|      51 | beta*             | Firefox      | Firefox-11.0b5-build1                   | watershed                                                                   |
|      52 | beta*             | Firefox      | Firefox-29.0b8-build1-schema2           | watershed                                                                   |
|      45 | beta-cdntest      | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      60 | beta-cdntest      | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      25 | beta-localtest    | Firefox      | Firefox-33.0b3-build1                   | NULL                                                                        |
|      27 | beta-localtest    | Thunderbird  | Thunderbird-32.0b1-build1               | NULL                                                                        |
|      34 | esr               | Firefox      | No-Update                               | NULL                                                                        |
|      36 | esr               | Thunderbird  | No-Update                               | NULL                                                                        |
|      59 | esr-cdntest       | NULL         | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      58 | esr-localtest     | Firefox      | Firefox-31.1.0esr-build1                | NULL                                                                        |
|      19 | nightly           | NULL         | No-Update                               | Global shut off rule. Bump priority of this rule to 100 to shut off all upd |
|      39 | nightly           | B2G          | B2G-mozilla-central-nightly-latest      | disabled because of busted patch making it into latest nightly              |
|       4 | nightly           | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|       3 | nightly           | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      17 | nightly           | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      14 | nightly           | Thunderbird  | Thunderbird-comm-central-nightly-latest | NULL                                                                        |
|       1 | nightly*          | NULL         | No-Update                               | OS Blocking rule for de-supported OS' on all nightly channels.              |
|      42 | nightly-b2g30     | B2G          | B2G-mozilla-b2g30_v1_4-nightly-latest   | NULL                                                                        |
|      47 | nightly-b2g32     | B2G          | B2G-mozilla-b2g32_v2_0-nightly-latest   | NULL                                                                        |
|       6 | nightly-cedar     | Fennec       | Fennec-cedar-nightly-20131107040204     | NULL                                                                        |
|       9 | nightly-elm       | Firefox      | Firefox-elm-nightly-20131221040201      | NULL                                                                        |
|      12 | nightly-esr24     | Firefox      | Firefox-mozilla-esr24-nightly-latest    | NULL                                                                        |
|      44 | nightly-esr31     | Firefox      | Firefox-mozilla-esr31-nightly-latest    | NULL                                                                        |
|      48 | nightly-esr31     | Thunderbird  | Thunderbird-comm-esr31-nightly-latest   | NULL                                                                        |
|      20 | nightly-holly     | Firefox      | Firefox-holly-nightly-20140321040202    | NULL                                                                        |
|       5 | nightly-oak       | Firefox      | Firefox-oak-nightly-latest              | NULL                                                                        |
|       7 | nightly-ux        | Firefox      | Firefox-ux-nightly-20140612040202       | NULL                                                                        |
|      40 | nightlytest       | B2G          | B2G-mozilla-central-nightly-latest      | NULL                                                                        |
|      24 | nightlytest       | Fennec       | Fennec-mozilla-central-nightly-latest   | NULL                                                                        |
|      22 | nightlytest       | Firefox      | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      23 | nightlytest       | MetroFirefox | Firefox-mozilla-central-nightly-latest  | NULL                                                                        |
|      33 | release           | Firefox      | No-Update                               | NULL                                                                        |
|      35 | release           | Thunderbird  | No-Update                               | NULL                                                                        |
|      57 | release-cdntest   | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      62 | release-cdntest   | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
|      56 | release-localtest | Firefox      | Firefox-32.0.1-build2                   | NULL                                                                        |
|      61 | release-localtest | Thunderbird  | Thunderbird-31.1.1-build1               | NULL                                                                        |
+---------+-------------------+--------------+-----------------------------------------+-----------------------------------------------------------------------------+
Attachment #8489419 - Flags: review?(nthomas)
Pete, I think comment #0 should give you enough background, but the tl;dr is that we need to support *-{localtest,cdntest} as the respective internal and external testing channels (in addition to betatest/releasetest for awhile). This tweak to final verify's regex should give it support for doing that.
Attachment #8489488 - Flags: review?(pmoore)
This block is used to determine whether we should produce direct-to-ftp or bouncer URLs. localtest is the new direct-to-ftp channel, so we need to include it here.
Attachment #8489492 - Flags: review?(rail)

Updated

5 years ago
Attachment #8489492 - Flags: review?(rail) → review+
Assignee

Updated

5 years ago
Attachment #8489492 - Flags: checked-in+
Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

Review of attachment 8489419 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the db info, very helpful.

::: mozilla/release-firefox-mozilla-esr31.py.template
@@ +119,5 @@
>  }
>  releaseConfig['releaseChannel']        = 'esr'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
>  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> +releaseConfig['testChannelRuleIds']    = [58,59]

59 needs product set to Firefox, it's NULL at the moment.

::: mozilla/staging_release-thunderbird-comm-esr31.py.template
@@ +131,5 @@
>      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']        = 'release'
>  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3

The other staging configs have a release rule, would be worth adding one to keep things consistent.
Attachment #8489419 - Flags: review?(nthomas) → review+
Assignee

Updated

5 years ago
Blocks: balrog-beta
Comment on attachment 8489419 [details] [diff] [review]
update rule ids for all release configs

(In reply to Nick Thomas [:nthomas] from comment #7)
> Comment on attachment 8489419 [details] [diff] [review]
> update rule ids for all release configs
> 
> Review of attachment 8489419 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Thanks for the db info, very helpful.
> 
> ::: mozilla/release-firefox-mozilla-esr31.py.template
> @@ +119,5 @@
> >  }
> >  releaseConfig['releaseChannel']        = 'esr'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> >  releaseConfig['testChannels']          = ['releasetest', 'esrtest']
> > +releaseConfig['testChannelRuleIds']    = [58,59]
> 
> 59 needs product set to Firefox, it's NULL at the moment.

Done.

> ::: mozilla/staging_release-thunderbird-comm-esr31.py.template
> @@ +131,5 @@
> >      'macosx64': 'mail/config/mozconfigs/macosx-universal/release',
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']        = 'release'
> >  releaseConfig['releaseChannelRuleIds'] = [] # Still on AUS3
> 
> The other staging configs have a release rule, would be worth adding one to
> keep things consistent.

Sure thing. I added this and landed the patch as https://hg.mozilla.org/build/buildbot-configs/rev/0fa6f60cfa9f
Attachment #8489419 - Flags: checked-in+
Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

Review of attachment 8489488 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Ben,

Thanks for the overview. I can't claim to completely understand how the channels work, but looking at the table, and seeing the *localtest and *cdntest entries, I see that when there is a *localtest entry, there seems to be a corresponding *cdntest entry, and based on your explanation it seems intuitive that one would be for internal (localtest) and the other for external (cdntest) testing. Regarding the validity of the sed expression, I can confirm that it works, and plays well on both mac and linux (which I am happy about - because it keeps final verification completely mac-compatible).

The only thing I see missing is maybe a wiki page explaining the channel names, which I think could be important given the number of channel names and complexity of the setup, especially as this has been a little revised now.

Of course I don't want to block on this, but do you think you could doc something up as an extra artefact for this bug? Maybe a page which just gives a high level explanation about the new channel name strategy? Sorry for the extra work! I would write it myself, but I am one of the people that would need to read it, to understand how it works. :)

Last point was rule_id 18 seems to have an incorrect comment - probably also worth fixing.

Thanks!
Pete
Attachment #8489488 - Flags: review?(pmoore) → review+
Comment on attachment 8489488 [details] [diff] [review]
make final verify support new channel names

(In reply to Pete Moore [:pete][:pmoore] from comment #9)
> Comment on attachment 8489488 [details] [diff] [review]
> make final verify support new channel names
> 
> Review of attachment 8489488 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hi Ben,
> 
> Thanks for the overview. I can't claim to completely understand how the
> channels work, but looking at the table, and seeing the *localtest and
> *cdntest entries, I see that when there is a *localtest entry, there seems
> to be a corresponding *cdntest entry, and based on your explanation it seems
> intuitive that one would be for internal (localtest) and the other for
> external (cdntest) testing. Regarding the validity of the sed expression, I
> can confirm that it works, and plays well on both mac and linux (which I am
> happy about - because it keeps final verification completely mac-compatible).
> 
> The only thing I see missing is maybe a wiki page explaining the channel
> names, which I think could be important given the number of channel names
> and complexity of the setup, especially as this has been a little revised
> now.

I created https://wiki.mozilla.org/Releases/Update_Channels just a few days ago - is that the kind of thing you're looking for?

> Last point was rule_id 18 seems to have an incorrect comment - probably also
> worth fixing.

That's a rule from the dev server, I'm not going to worry about it...
Attachment #8489488 - Flags: checked-in+
Because we have different channels for different release streams, some scripts and tools (final verify, update verify config bumper, etc.) can't easily figure out which channels to use. Rather than adding hardcodes to all of those tools, changing our release configs like this lets us do the right thing in all places without hardcodes.

I did a bunch of staging tests with this (and the other) patches and everything looked fine. Specifically, I ran the updates, final verify, and ready for releasetest builders of Firefox+Thunderbird Beta+Release - everything seemed fine there (modulo some staging specific issues).
Attachment #8490206 - Flags: review?(nthomas)
Assignee

Updated

5 years ago
Attachment #8490206 - Attachment description: split out test channels in release configs; use local test channel for update verify → support new test channel substitutions in release mail; use local test channel for update verify
Reporter

Updated

5 years ago
Attachment #8490204 - Flags: review?(nthomas) → review+
Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Review of attachment 8490207 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with minor tweak.

::: mozilla/staging_release-thunderbird-comm-esr24.py.template
@@ +138,5 @@
>      'win32': 'mail/config/mozconfigs/win32/release',
>  }
>  releaseConfig['releaseChannel']      = 'release'
> +releaseConfig['localTestChannel']    = 'betatest'
> +releaseConfig['cdnTestChannel']      = 'releasetest'

These are empty in the production config, because updates are gone for esr24 ?
Attachment #8490207 - Flags: review?(nthomas) → review+
Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Review of attachment 8490206 [details] [diff] [review]:
-----------------------------------------------------------------

Scratch everything I said on IRC about snippet comparison, got the wrong end of the stick.
Attachment #8490206 - Flags: review?(nthomas) → review+
(In reply to Nick Thomas [:nthomas] from comment #14)
> Comment on attachment 8490207 [details] [diff] [review]
> switch to new test channel variables
> 
> Review of attachment 8490207 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with minor tweak.
> 
> ::: mozilla/staging_release-thunderbird-comm-esr24.py.template
> @@ +138,5 @@
> >      'win32': 'mail/config/mozconfigs/win32/release',
> >  }
> >  releaseConfig['releaseChannel']      = 'release'
> > +releaseConfig['localTestChannel']    = 'betatest'
> > +releaseConfig['cdnTestChannel']      = 'releasetest'
> 
> These are empty in the production config, because updates are gone for esr24
> ?

I'll change this for staging on check-in - thanks!
(In reply to Nick Thomas [:nthomas] from comment #15)
> Comment on attachment 8490206 [details] [diff] [review]
> support new test channel substitutions in release mail; use local test
> channel for update verify
> 
> Review of attachment 8490206 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Scratch everything I said on IRC about snippet comparison, got the wrong end
> of the stick.

Just to be safe I ended up doing full runs of the updates builder for mozilla-beta and mozilla-release on my dev-master -- they both went totally fine.
In production
Assignee

Updated

5 years ago
Attachment #8490204 - Flags: checked-in+
Comment on attachment 8490206 [details] [diff] [review]
support new test channel substitutions in release mail; use local test channel for update verify

Landed this, including a transplant to the production-0.8 branch so that it will get picked up with 33.0b5 today.
Attachment #8490206 - Flags: checked-in+
Comment on attachment 8490207 [details] [diff] [review]
switch to new test channel variables

Landed this, including a transplant to the production branch so that it's picked up with 33.0b5 today.
Attachment #8490207 - Flags: checked-in+
I had to land a follow-up to put the new variables in the actual release configs (instead of just the templates), otherwise the reconfig would've busted:
https://hg.mozilla.org/build/buildbot-configs/rev/86f1fc6aff10
In prod with reconfig on 2014-09-18 08:14 PT
Everything we need to do for Beta is done now. I'm going to leave this bug open to continue to track the migration when we put Release and ESR on Balrog.
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Attachment #8546710 - Flags: review?(nthomas)
Reporter

Updated

4 years ago
Attachment #8546710 - Flags: review?(nthomas) → review+
Comment on attachment 8546710 [details] [diff] [review]
switch release configs to new rules/channels

Review of attachment 8546710 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_release-fennec-mozilla-release.py.template
@@ +163,3 @@
>  releaseConfig['partialUpdates']      = {}
>  releaseConfig['bouncerServer']       = 'download.mozilla.org'
>  releaseConfig['releaseChannelRuleIds'] = []

I updated my local branch to include a release channel rule id, which now exists.
Assignee

Updated

4 years ago
Attachment #8546710 - Flags: checked-in+
Only thing left to do here is mozilla-esr31, which will happen at the same time as bug 1117959 (hopefully in the next week or two).
The last ones!
Attachment #8551905 - Flags: review?(nthomas)
Reporter

Updated

4 years ago
Attachment #8551905 - Flags: review?(nthomas) → review+
Assignee

Updated

4 years ago
Attachment #8551905 - Flags: checked-in+
With the ESR changes landed we're all done here!
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.