745536 - Initial setup for aus3.mozilla.org to serve Thunderbird builds.

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[John Hopkins (:jhopkins)]

* Rsync Thunderbird snippets from momo-build-adm-01.sj.mozillamessaging.com to aus3-staging.mozilla.org
* Upload new Thunderbird snippets to aus3-staging
  * we can do this anytime with the existing Thunderbird build automation - as long as it can reach FF AUS.
* Verify that upgrades work as expected.
* Point the Thunderbird AUS DNS at aus3-staging.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to John Hopkins (:jhopkins) from comment #0)
> * Point the Thunderbird AUS DNS at aus3-staging.

Specifically, aus2.mozillamessaging.com -> aus3.mozilla.org ? Firefox makes requests to the latter domain (or to aus2.m.o and gets redirected to aus3). This should also allow Thunderbird to set app.update.certs.* and pin the SSL cert to a couple of CAs.

Comment 2

[Mark Banner (:standard8)]

(In reply to Nick Thomas [:nthomas] from comment #1)
> (In reply to John Hopkins (:jhopkins) from comment #0)
> > * Point the Thunderbird AUS DNS at aus3-staging.
> 
> Specifically, aus2.mozillamessaging.com -> aus3.mozilla.org ? Firefox makes
> requests to the latter domain (or to aus2.m.o and gets redirected to aus3).

I believe that's the intention.

Note we also need to do this:

* Compare configs and copy over the options for not upgrading various Linux builds & other platforms according to the specific versions we're on.

> This should also allow Thunderbird to set app.update.certs.* and pin the SSL
> cert to a couple of CAs.

Yep, just been looking at that. This is bug 593571.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

From the tarball I got from jhopkins it looks like aus2.momo is running 
* AUS code from AUS2_RTM_201113030913
* a patch which disables cookies so that varnish can be used to cache requests
* a patch to add headers to blocking caching if there is a random throttle in effect
* presumably some newer config-dist.php/config.php
This is all compiled into an rpm to deploy to the machines serving requests.

The AUS2_RTM_201113030913 tag is a bit misleading, it should be AUS2_RTM_2011103... from when bug 459972 landed. That's the last feature change we made so the code is up to date.

On aus3.m.o all requests are marked with
  Cache-Control: no-store, must-revalidate, post-check=0, pre-check=0, private
so no caching is done and we don't need the two patches.

We'll need to deploy the Thunderbird part of the config-dist.php. Could I get a copy of the current version used on aus2.momo ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

jhopkins and I talked about serving nightly updates for the test builds he is planning to have set up on Firefox infra in the next few days, using aus3.m.o. This turns out to be a bit of a hard problem because it's all tangled up.

The best I can up with would be
* swap comm-central/aurora nightlies to query aus3.m.o as soon as we can deploy a sensible Thunderbird config and test; redirect requests to aus2.momo on nightly channels to aus3.m.o
* leave the existing TB infra uploading to aus-staging.sj.mozillamessaging.com and set up a sync that keeps aus3-staging up to date. Would need firewall changes, or bouncing the files through someone with access to both worlds
* figure out some way to get builds from the new infra on the nightlytest channel. Would be really great if the comm-central mozconfigs used MOZ_UPDATE_CHANNEL from the environment like the Firefox ones do, eg http://hg.mozilla.org/mozilla-central/file/default/browser/config/mozconfigs/win32/nightly - otherwise don't know how we can do this

You might ask 'why don't we upload the new builds to aus3 and the old ones to aus2.momo?' and I'd say 'how do we get two builds from the source to have different app.update.url preferences?'

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Supposing the channel issue can be fixed up, there are a few buildbot config tweaks we'd need in the new infra for the plan in comment #4. Namely 
 BRANCHES['comm-central']['aus2_base_upload_dir']
 BRANCHES['comm-central']['aus2_base_upload_dir_l10n']
pointing to 
 /opt/aus2/incoming/2/Thunderbird/comm-central-test
and if we change the mozconfigs to depend on the environement
 BRANCHES['comm-central']['update_channel'] = 'nightlytest'

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

OK, better plan:

* get thunderbird AUS config merged into aus3.m.o and deployed (me)
* do initial sync of production snippets to aus3-staging for releases (me)
* set up sync of nightly snippets to aus3-staging (me)
* do some correctness checks (me, standard8?)

In parallel:
* get builds on ff-infra running (jhopkins)
 * upload snippets for a nightlytest channel on aus3.m.o (me for buildbot patch) 
Testers will set app.update.url.override so that they query both the right host and channel, and we don't need to worry about mozconfig for comm-central changes.

Once we have confidence in aus3.m.o setup
* change host in app.update.url for Daily, possibly EarlyBird. Follow up on Beta as appropriate (Standard8)

Separately the builds on ff-infra get promoted to official, and we deprecate the nightlytest channel. (jhopkins mostly)

We also need some way of responding to users who query aus2.momo.c - all the builds out there in the wild now. We can ask IT to setup something analagous to aus03.zlb.phx.mozilla.net (which aus3.m.o is a CNAME for), using the existing aus2.momo cert, and eventually point aus2.momo at it. We'd use the same backend nodes as aus03.zlb.phx.mozilla.net (pp-distNN), we're just terminating the SSL separately because the request is to a different domain.

Comments ?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Current config.php from aus2.momo.c

This is what's running now, for comparison to the config patch I'm gonna upload next.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Add thunderbird config to aus3.m.o

The patch itself is fairly simple. Here are the changes relative to the aus2.momo.c config:

>define('SOURCE_DIR', '/opt/aus/current');
>define('OVERRIDE_DIR', '/opt/aus/current/3');

Dropped, we'll use Firefox standard locations.

>$productThrottling = array(
>     'Thunderbird' => array(
>         '3.0.12'    => 0,
>         '3.1.20'    => 100,
>         '10.0.2'    => 100,

All removed to leave an empty array. 3.0.12 never shipped, 3.1.20 is EOL and unthrottled, 10.0.2 is unthrottled.

>$throttleExceptions = array(
>    '3.0.12' => array (
>        'betatest',
>        'releasetest',
>    ),
>    '3.1.20' => array (
>        'betatest',
>        'releasetest',

Removed to match removals above. 

NB. Because there is no Firefox/Thunderbird switching here, and we now share versions, we won't be able to set independent exceptions for those Products. This would be a problem if, eg, Firefox wanted to throttle updates to 12.0, except on beta, while Thunderbird wanted to throttle 12.0 betas. That scenario seems unlikely, can anyone think of anything more plausible ?

>$productBranchVersions = array(
>    'Thunderbird' =>  array(
>        '1.5.0.*' =>  '1.5.0.x',
>        '2.0*'    =>  '2.0',
>        '3.0*'    =>  'comm-1.9.1',
>        '3.3*'	  =>  'comm-miramar',
>        '5.0*'	  =>  'comm-miramar',

These five removed, since the there are no snippets there to point to. This change means people with old builds will get updated the latest comm-central. We will block deprecated platforms but this may be semi-traumatic for the users themselves. Not that they're doing themselves any favours by using ancient builds.

>        '3.1*'	  =>  'comm-1.9.2',

T'bird drivers should decide where they want these people to end up after the 24th. If it's comm-central then we just remove this line later.

>        '*'       =>   array(
>            'nightly' => 'comm-central',
>            'nightlytest' => 'nightlytest',

Tweaked to 'comm-central-test'.

>            'aurora'  => 'comm-aurora',

Added an 'auroratest' => 'comm-aurora-test', so that we're set up if TB wants to disable EarlyBird updates for the few days after the merge from c-c (like FF does with Aurora).

>$latestRelease = array(
>    'Thunderbird' => array(
>        'beta'      => '3.1.20',
>        'release'   => '3.1.20',

Didn't bother bringing this over. It's for defunct channel switching and this config is very out of date.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Created tbirdbld user, generally cleaned up permissions.

(In reply to Nick Thomas [:nthomas] from comment #7)
> * do initial sync of production snippets to aus3-staging for releases (me)

Creating a tarball.

> * set up sync of nightly snippets to aus3-staging (me)

Done, it's bouncing through my laptop :-S.

Comment 11

[Ludovic Hirlimann [:Usul]]

Guys ping me on irc for testing as I might miss some bugmail (it would be burried in my bugmail folder).

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Add thunderbird config to aus3.m.o, with esr

Adds the esr10 branch that's in the buildbot config.

Comment 13

[Mark Banner (:standard8)]

(In reply to Nick Thomas [:nthomas] from comment #7)
> We also need some way of responding to users who query aus2.momo.c - all the
> builds out there in the wild now. We can ask IT to setup something analagous
> to aus03.zlb.phx.mozilla.net (which aus3.m.o is a CNAME for), using the
> existing aus2.momo cert, and eventually point aus2.momo at it. We'd use the
> same backend nodes as aus03.zlb.phx.mozilla.net (pp-distNN), we're just
> terminating the SSL separately because the request is to a different domain.

I'm not sure if this affects the outcome or not, but TB isn't currently tied down to a specific SSL cert for aus2 (we will do to aus3 after the migration).


(In reply to Nick Thomas [:nthomas] from comment #9)
> NB. Because there is no Firefox/Thunderbird switching here, and we now share
> versions, we won't be able to set independent exceptions for those Products.
> This would be a problem if, eg, Firefox wanted to throttle updates to 12.0,
> except on beta, while Thunderbird wanted to throttle 12.0 betas. That
> scenario seems unlikely, can anyone think of anything more plausible ?

Generally we wouldn't have the same version on different channels. So I don't think there's an issue there.

> >$productBranchVersions = array(
> >    'Thunderbird' =>  array(
> >        '1.5.0.*' =>  '1.5.0.x',
> >        '2.0*'    =>  '2.0',
> >        '3.0*'    =>  'comm-1.9.1',
> >        '3.3*'	  =>  'comm-miramar',
> >        '5.0*'	  =>  'comm-miramar',
> 
> These five removed, since the there are no snippets there to point to. This
> change means people with old builds will get updated the latest
> comm-central. We will block deprecated platforms but this may be
> semi-traumatic for the users themselves. Not that they're doing themselves
> any favours by using ancient builds.

That's just for nightlies right? 2.0.0.24 users should still get updated to 3.1 or wherever that end point points.

> >        '3.1*'	  =>  'comm-1.9.2',
> 
> T'bird drivers should decide where they want these people to end up after
> the 24th. If it's comm-central then we just remove this line later.

If they are nightly users, then they should go to comm-central. If they are beta users, then stay on the beta channel, and release likewise.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Mark Banner (:standard8) from comment #13)
> I'm not sure if this affects the outcome or not, but TB isn't currently tied
> down to a specific SSL cert for aus2 (we will do to aus3 after the
> migration).

This is just to make the domain in the cert match up with the request.

> That's just for nightlies right? 2.0.0.24 users should still get updated to
> 3.1 or wherever that end point points.

Sorry, should have mentioned that. It is just for nightlies.

> If they are nightly users, then they should go to comm-central. If they are
> beta users, then stay on the beta channel, and release likewise.

Yes, nightly again. We can do that in bug 720822.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 616505 [details] [diff] [review]
[cvs] Add thunderbird config to aus3.m.o, with esr

Review of attachment 616505 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Nick Thomas [:nthomas] from comment #9)
> Created attachment 616427 [details] [diff] [review]
> [cvs] Add thunderbird config to aus3.m.o
> 
> The patch itself is fairly simple. Here are the changes relative to the
> aus2.momo.c config:

Everything here sounds OK to me, except....

> >$throttleExceptions = array(
> >    '3.0.12' => array (
> >        'betatest',
> >        'releasetest',
> >    ),
> >    '3.1.20' => array (
> >        'betatest',
> >        'releasetest',
> 
> Removed to match removals above. 
> 
> NB. Because there is no Firefox/Thunderbird switching here, and we now share
> versions, we won't be able to set independent exceptions for those Products.
> This would be a problem if, eg, Firefox wanted to throttle updates to 12.0,
> except on beta, while Thunderbird wanted to throttle 12.0 betas. That
> scenario seems unlikely, can anyone think of anything more plausible ?

We've been doing throttling of Firefox releases when we ship them. That is, we throttle all of the old rapid release versions that point at the new one for periods of time. If Thunderbird doesn't also do this, we've got a conflict.

Comment 16

[Mark Banner (:standard8)]

(In reply to Ben Hearsum [:bhearsum] from comment #15)
> > >$throttleExceptions = array(
> > >    '3.0.12' => array (
> > >        'betatest',
> > >        'releasetest',
> > >    ),
> > >    '3.1.20' => array (
> > >        'betatest',
> > >        'releasetest',
> > 
> > Removed to match removals above. 
> > 
> > NB. Because there is no Firefox/Thunderbird switching here, and we now share
> > versions, we won't be able to set independent exceptions for those Products.
> > This would be a problem if, eg, Firefox wanted to throttle updates to 12.0,
> > except on beta, while Thunderbird wanted to throttle 12.0 betas. That
> > scenario seems unlikely, can anyone think of anything more plausible ?
> 
> We've been doing throttling of Firefox releases when we ship them. That is,
> we throttle all of the old rapid release versions that point at the new one
> for periods of time. If Thunderbird doesn't also do this, we've got a
> conflict.

If I've understood productThrottling versus throttleExceptions correctly, then we can throttle version numbers differently per-product via productThrottling.

What we can't do is have different throttle exceptions on different channels for the same version number. I think that's fine. Generally we've only ever throttled on the *test channels iirc.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Mark Banner (:standard8) from comment #16)
> If I've understood productThrottling versus throttleExceptions correctly,
> then we can throttle version numbers differently per-product via
> productThrottling.

This is my understanding too.

> What we can't do is have different throttle exceptions on different channels
> for the same version number. I think that's fine. Generally we've only ever
> throttled on the *test channels iirc.

s/throttled/not throttled/ I presume, but yes.

ping on the r? We'd like to land this asap.

Comment 18

[Mark Banner (:standard8)]

Comment on attachment 616505 [details] [diff] [review]
[cvs] Add thunderbird config to aus3.m.o, with esr

Looks good.

Comment 19

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #17)
> (In reply to Mark Banner (:standard8) from comment #16)
> > If I've understood productThrottling versus throttleExceptions correctly,
> > then we can throttle version numbers differently per-product via
> > productThrottling.
> 
> This is my understanding too.

Ah, sorry. I misread/misinterpreted things. Carry on!

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 616505 [details] [diff] [review]
[cvs] Add thunderbird config to aus3.m.o, with esr

Checking in inc/config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.203; previous revision: 1.202
done

$ cvs tag AUS2_PRODUCTION
W inc/config-dist.php : AUS2_PRODUCTION already exists on version 1.202 : NOT MOVING tag to version 1.203
$ cvs tag -F AUS2_PRODUCTION
T inc/config-dist.php

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Requests on nightlytest get thunderbird-test nightlies:
https://aus3.mozilla.org/update/1/Thunderbird/7.0a1/2011062003/WINNT_x86-msvc/en-US/nightlytest/update.xml

There might be a permissions problem uploading snippets, they're owned tbirdbld:tbirdbld with dir modes of 750 and file mode 640. They need to be a+w and a+r for the webheads to get at them. I've done a couple of chmod's but will need to monitor.

To test updates you'd want to create a new string pref called app.update.url.override in about:config, and set it to
 https://aus3.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/nightlytest/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
We should do this with both the old and new infra builds.

Comment 22

[Mark Banner (:standard8)]

If I'm understanding this right, then today's Mac build should have got updates, this is the url it is checking:

https://aus3.mozilla.org/update/3/Thunderbird/14.0a1/20120423111814/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightlytest/Darwin%2010.8.0/default/default/update.xml?force=1

but not updates yet (its been completed for over an hour).

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Standard8, that should be fixed up now. I'm getting an offer to the bits in 
 http://ftp.mozilla.org/pub/mozilla.org/thunderbird-test/nightly/2012/04/2012-04-24-03-02-02-comm-central/
(there doesn't seem to be a nightly from the 25th)

The symlinks to make Mac work were missing, and the permissions issue mentioned in comment #21 has been resolved (primary group of tbirdbld set to users).

Comment 24

[Mark Banner (:standard8)]

I can confirm that works now. I'll monitor today's builds and get Ludovic to do the same.

Comment 25

[Ludovic Hirlimann [:Usul]]

Hum https://aus3.mozilla.org/update/3/Thunderbird/14.0a1/20120424030426/WINNT_x86-msvc/en-US/nightlytest/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml?force=1 is empty I should be offered the http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012-04-25-03-00-27-comm-central/ right ?

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Querying on the nightlytest channel you'd get offered builds from the thunderbird-test/nightly/ dir. Standard8 and I agreed that the 2012-04-25 builds for that blew up, so no update until something succeeds.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Ok, lets repurpose this bug as the initial setup of aus3.m.o for Thunderbird, and use further bugs for other setup and switching.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

This took us from config merge though user setup on aus3-staging to rsyncing nightly snippets from aus2.momo's staging to aus3-staging.