885477 - switch Nightly users to Balrog

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Where "Nightly" means mozilla-central nightlies and associated branches (ux, profiling, etc.). This explicitly does not include aurora or other branches further down the line.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This will be served at aus4.mozilla.org instead of aus3.m.o, but not clear yet if we should be doing a 302 on aus3 or just changing app.update.url in the product itself.

Comment 2

[[e-mail] (:bhearsum)]

Nick and I talked about this today. We decided that we'll update the pref in mozilla-central to make this change rather than use rewrites on the server side. In the extreme case that something goes horribly wrong and we need to revert, this means we'll need to use redirection on the server side to send people back to aus3.mozilla.org (because they'll be pointed at aus4.mozilla.org, and we can't change DNS because of certificates).

We want to do this ASAP, which means we'll be doing some simple load testing once the production nodes are up. After that's done we'll make the switch in mozilla-central and keep an eye on things. If this is done before the next uplift to Aurora (September 16th), we need to make sure that it is backed out during the Aurora uplift, as we don't think we'll be ready for Aurora users right away.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

catlee suggested that if we would like to scale gently rather than all/nothing we could use a redirect for some percentage of requests. ie still change the pref in m-c, but aus4 could redirect back to aus3.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: switch to aus4; update certificate information

I'm planning to land this on Monday and then kick a new set of nightlies (to make sure I can test it right away, and also get a couple of nightlies in before the Summit). I think I've updated the certificate information correctly (based on the current live certificate on aus4.mozilla.org and the backup in bug 919746).

However, I'm confused about why there is quotes in the Thawte issuer - I don't see them in the aus3.mozilla.org Thawte certificate (from https://bugzilla.mozilla.org/show_bug.cgi?id=914285#c4).

Also note that unlike Equifax, DigiCert's intermediary certificate has no OU.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 810579 [details] [diff] [review]
switch to aus4; update certificate information

Rob, Gavin said you're probably a better reviewer for this.

Comment 7

[Robert Strong (they/them - no direct email)]

Just tested the patch which is quicker than checking the values and it failed with

Timestamp: 9/26/2013 11:20:32 AM
Error: Expected certificate attribute 'issuerName' value incorrect, expected: 'O=DigiCert Inc,C=US', got: 'CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 106

So, the issuerName for the Digicert certificate is
"CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"

I also verified that after changing the value it passes.

Comment 8

[Robert Strong (they/them - no direct email)]

The Thawte certificate failed with

Timestamp: 9/26/2013 11:30:56 AM
Error: Expected certificate attribute 'issuerName' value incorrect, expected: 'CN=Thawte SSL CA,O=Thawte, Inc.,C=US', got: 'CN=Thawte SSL CA,O="Thawte, Inc.",C=US'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 106

So, the issuerName for the Thawte certificate is
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"

I also verified that after changing the value it passes.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: fix certificate info

Thanks so much for the testing Rob.

Comment 10

[Robert Strong (they/them - no direct email)]

Comment on attachment 810701 [details] [diff] [review]
fix certificate info

Looks good and thanks!

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 810701 [details] [diff] [review]
fix certificate info

Landed to mozilla-central: https://hg.mozilla.org/mozilla-central/rev/5a49762ee832

I've triggered a new set of nightlies, too.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: switch android builds to aus4

Almost forgot about Android! I gave these another quick test today by pushing this patch to try: https://tbpl.mozilla.org/?tree=Try&rev=0179302e726e&pusher=[e-mail]

Comment 13

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 812099 [details] [diff] [review]
switch android builds to aus4

That's the only use of "aus2.mozilla.org" in mobile/android
Makes sense that the UpdaterService would be the only code using it.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 812099 [details] [diff] [review]
switch android builds to aus4

Landed and triggered new Android nightlies: https://hg.mozilla.org/mozilla-central/rev/192461a9dfe8

Comment 15

[[e-mail] (:bhearsum)]

Everything is going well here so far. The latest set of nightlies for all platforms has finished and those updates are applying fine. The bigger test will be tomorrow when everyone that got this set of nightlies talks with aus4.mozilla.org.

Comment 16

[[e-mail] (:bhearsum)]

Kairo suggested that we use the build system helpers we have to make sure this doesn't get uplifted. I'm going to do that in bug 922261.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: don't ride the trains

Rob, Kairo suggested that we make sure this doesn't ride the trains. I think it makes sense to.

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 812234 [details] [diff] [review]
don't ride the trains

whoops, i messed up one of the issuers.

Comment 19

[Robert Strong (they/them - no direct email)]

What is the concern with this riding the trains especially with there being 4 weeks until the next merge (10-28)?

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #19)
> What is the concern with this riding the trains especially with there being
> 4 weeks until the next merge (10-28)?

We haven't made a decision about when this is going to hit Aurora yet. It may jump sooner than the next merge, ride the train, or be held back (if we hit major issues). I'd rather be conservative about this until we make up our mind.

Comment 21

[[e-mail] (:bhearsum)]

Attachment: fix issuer again

Comment 22

[Robert Strong (they/them - no direct email)]

Comment on attachment 812248 [details] [diff] [review]
fix issuer again

I'm ok with this but please I'd prefer if you held off on landing this patch until it becomes clear that the change to aus4 shouldn't ride the trains.

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #22)
> Comment on attachment 812248 [details] [diff] [review]
> fix issuer again
> 
> I'm ok with this but please I'd prefer if you held off on landing this patch
> until it becomes clear that the change to aus4 shouldn't ride the trains.

That doesn't really address what I'm trying to though...I'm trying to guard against us forgetting to land something like this (or back out the change during the merge) if we make that decision. That's why I'd like to default to aus3.

Comment 24

[Robert Strong (they/them - no direct email)]

Ok and go ahead and land it... it is just the opposite approach we usually take unless we are sure that we don't want something to ride the trains. You might also want to put this on the release drivers radar by using the tracking flag. I'll also add a calendar reminder for myself so I can remember to ping this bug in case nothing comes up so the patch can be backed out.

Comment 26

[[e-mail] (:bhearsum)]

Thanks Rob. I'll land this tomorrow. I guess tracking-firefox27 is appropriate because that's what Nightly is currently?

Comment 27

[Robert Strong (they/them - no direct email)]

Yes

Comment 28

[[e-mail] (:bhearsum)]

https://hg.mozilla.org/mozilla-central/rev/cc4a3f3f899e

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD' patch leave out Android by design ?

Comment 30

[Robert Kaiser]

(In reply to Nick Thomas [:nthomas] from comment #29)
> Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> patch leave out Android by design ?

This should apply to Android as well, I'd think. Ben?

Comment 31

[[e-mail] (:bhearsum)]

(In reply to Robert Kaiser (:[e-mail]) from comment #30)
> (In reply to Nick Thomas [:nthomas] from comment #29)
> > Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> > patch leave out Android by design ?
> 
> This should apply to Android as well, I'd think. Ben?

Maybe -- I'm planning to talk to Nick and others this week about letting Balrog ride the trains to Aurora with 27. If we're doing that I'll need a different patch (one that lets Nightly and Aurora go through, but protects beta/release).

Comment 32

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #31)
> (In reply to Robert Kaiser (:[e-mail]) from comment #30)
> > (In reply to Nick Thomas [:nthomas] from comment #29)
> > > Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> > > patch leave out Android by design ?
> > 
> > This should apply to Android as well, I'd think. Ben?
> 
> Maybe -- I'm planning to talk to Nick and others this week about letting
> Balrog ride the trains to Aurora with 27. If we're doing that I'll need a
> different patch (one that lets Nightly and Aurora go through, but protects
> beta/release).

Speaking of Aurora, I filed bug 927364 to track switching it to Balrog.