750644 - Serve requests to aus2.mozillamessaging.com using same infra as aus3.mozilla.org

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We need
* SSL termination/load balancer
* copy over release snippets
* consistency checks between old/new systems
* QA signoff
* swap aus2.mozillamessaging.com DNS record to point to new load balancer

All before the equipment in sjc1 expires.

Comment 1

[Mark Banner (:standard8)]

How far have we got on this?

Trying a test on the new server (update check from 11.0.1) gives a 403 forbidden error:

https://<server>/update/3/Thunderbird/11.0.1/20120327120934/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2010.8.0/default/default/update.xml?force=1

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I was waylaid by the stage/ftp infra moving to scl3 last week, but will get on with transferring snippets/staged snippets for aus2.momo to aus3.m.o.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Looking at the snippet stores on existing and new servers, in general aus2.momo has the 3.0 branch and newer, except that there are directories on both for 2.0.0.24, 3.0a1-3, 3.0b1-4, 3.0, 3.0.1, and 3.0.2.

I've analysed those and here's the results
* we should use aus3.m.o's 2.0.0.24/ since the in-app pref is aus2.m.o
* we should use aus2.momo's 3.0*/ since that points at 3.0.11 (the EOL release for the branch). The in-app pref points at aus2.mozillamessaging.com from 3.0b3 onwards. 

We can also do a little cleanup on aus3.m.o
* 2.0.0.24/ has several Darwin buildtargets that aren't useful:
    Darwin_x86_64-gcc3
    Darwin_x86_64-gcc3-u-i386-x86_64
    Darwin_x86-gcc3-u-i386-x86_64
    Darwin_x86-gcc3-u-ppc-i386
  because all builds had Darwin_Universal-gcc3 (verified with blocklist metrics)
* the 2.0.0.23-disabled/ directory is left over from some panic

The backup before making changes is 
  aus3-staging:/opt/aus2/snippets/backup/bug750644-pre-snippet-merge.tgz

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The snippets from aus2.momo are live on aus3.m.o. There's a short script at momo-build-adm-01.sj.mozillamessaging.com:~cltbld/capture-snippets.sh which creates a tarball, which got copied to aus3-staging's snippets/backup/ and unpacked in snippets/staging/, then moved the dirs into the production location.

Next up some testing of the LB from bug 750646.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Bug 750646 has been reopened because Thunderbird 11.0 (and probably others) can't connect due not having the 'GeoTrust SSL CA' as a built-in in NSS. Installing that CA as an intermediate on the load balancer should fix that up. I was able to take 3.1.17 and update to 3.1.20 and then 12.0.1, with the suitable /etc/hosts hack (see below) because 3.1.x does have that CA.

There are some tests running to verify the content of the responses of the existing and new servers is the same. They don't test every possible version/platform/locale/channel but hit a lot of the common scenarios and are passing so far. More info in a few hours.

I think we're pretty close here, and once we've fixed the CA issue it would be great if Standard8 and Ludovic could check for things I missed. You just put 
 63.245.217.185   aus2.mozillamessaging.com
in your /etc/hosts and test updates as normal. I haven't tried Windows at all.

Comment 6

[Philippe M. Chiasson (:gozer)]

Or use aus-test.mozillamessaging.com as the update server, that points to 63.245.217.185 as well.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: content tests

These are the content tests I ran, which all passed. It checks beta/release/esr channels
* 12.0   - all platforms/locales for all betas and the release build
* 11.0.1 - all platforms/locales for the release build
* 11.0   - all platforms/locales for all the betas and the release build
* 3.1.20 - all platforms/locales for beta & release
* 3.0.11 - all platforms/locales for beta & release
plus tb_top_locales which is en-US, de, and ru locales for all platforms and versions.

The certs also got fixed up and Thunderbird 11.0 updates to 12.0 on the new server now.

Comment 8

[Ludovic Hirlimann [:Usul]]

So I tried on win7 3.0.8_fr -> 3.0.11 -> 3.1.14 -> 3.1.20 -> 12.0.1. Wondering why I got the .14 (I remember we had a hack, but mostly for the mac around that period.)

Mac Dk got the same 3.1.14.

Besides that it all went ok.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> * swap aus2.mozillamessaging.com DNS record to point to new load balancer

At this point I think we are clear to go ahead and switch DNS to 63.245.217.185.

Comment 10

[Philippe M. Chiasson (:gozer)]

DNS updated.