Closed Bug 885477 Opened 7 years ago Closed 7 years ago

switch Nightly users to Balrog

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox27+ fixed)

RESOLVED FIXED
Tracking Status
firefox27 + fixed

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

(Whiteboard: [balrog])

Attachments

(3 files, 2 obsolete files)

Where "Nightly" means mozilla-central nightlies and associated branches (ux, profiling, etc.). This explicitly does not include aurora or other branches further down the line.
This will be served at aus4.mozilla.org instead of aus3.m.o, but not clear yet if we should be doing a 302 on aus3 or just changing app.update.url in the product itself.
Product: mozilla.org → Release Engineering
Nick and I talked about this today. We decided that we'll update the pref in mozilla-central to make this change rather than use rewrites on the server side. In the extreme case that something goes horribly wrong and we need to revert, this means we'll need to use redirection on the server side to send people back to aus3.mozilla.org (because they'll be pointed at aus4.mozilla.org, and we can't change DNS because of certificates).

We want to do this ASAP, which means we'll be doing some simple load testing once the production nodes are up. After that's done we'll make the switch in mozilla-central and keep an eye on things. If this is done before the next uplift to Aurora (September 16th), we need to make sure that it is backed out during the Aurora uplift, as we don't think we'll be ready for Aurora users right away.
OS: Linux → All
Hardware: x86_64 → All
catlee suggested that if we would like to scale gently rather than all/nothing we could use a redirect for some percentage of requests. ie still change the pref in m-c, but aus4 could redirect back to aus3.
I'm planning to land this on Monday and then kick a new set of nightlies (to make sure I can test it right away, and also get a couple of nightlies in before the Summit). I think I've updated the certificate information correctly (based on the current live certificate on aus4.mozilla.org and the backup in bug 919746).

However, I'm confused about why there is quotes in the Thawte issuer - I don't see them in the aus3.mozilla.org Thawte certificate (from https://bugzilla.mozilla.org/show_bug.cgi?id=914285#c4).

Also note that unlike Equifax, DigiCert's intermediary certificate has no OU.
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Attachment #810579 - Flags: review?(gavin.sharp)
Duplicate of this bug: 913971
Comment on attachment 810579 [details] [diff] [review]
switch to aus4; update certificate information

Rob, Gavin said you're probably a better reviewer for this.
Attachment #810579 - Flags: review?(gavin.sharp) → review?(robert.bugzilla)
Just tested the patch which is quicker than checking the values and it failed with

Timestamp: 9/26/2013 11:20:32 AM
Error: Expected certificate attribute 'issuerName' value incorrect, expected: 'O=DigiCert Inc,C=US', got: 'CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 106

So, the issuerName for the Digicert certificate is
"CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"

I also verified that after changing the value it passes.
The Thawte certificate failed with

Timestamp: 9/26/2013 11:30:56 AM
Error: Expected certificate attribute 'issuerName' value incorrect, expected: 'CN=Thawte SSL CA,O=Thawte, Inc.,C=US', got: 'CN=Thawte SSL CA,O="Thawte, Inc.",C=US'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 106

So, the issuerName for the Thawte certificate is
"CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US"

I also verified that after changing the value it passes.
Thanks so much for the testing Rob.
Attachment #810579 - Attachment is obsolete: true
Attachment #810579 - Flags: review?(robert.bugzilla)
Attachment #810701 - Flags: review?
Comment on attachment 810701 [details] [diff] [review]
fix certificate info

Looks good and thanks!
Attachment #810701 - Flags: review? → review+
Comment on attachment 810701 [details] [diff] [review]
fix certificate info

Landed to mozilla-central: https://hg.mozilla.org/mozilla-central/rev/5a49762ee832

I've triggered a new set of nightlies, too.
Attachment #810701 - Flags: checked-in+
Almost forgot about Android! I gave these another quick test today by pushing this patch to try: https://tbpl.mozilla.org/?tree=Try&rev=0179302e726e&pusher=bhearsum@mozilla.com
Attachment #812099 - Flags: review?(mark.finkle)
Comment on attachment 812099 [details] [diff] [review]
switch android builds to aus4

That's the only use of "aus2.mozilla.org" in mobile/android
Makes sense that the UpdaterService would be the only code using it.
Attachment #812099 - Flags: review?(mark.finkle) → review+
Comment on attachment 812099 [details] [diff] [review]
switch android builds to aus4

Landed and triggered new Android nightlies: https://hg.mozilla.org/mozilla-central/rev/192461a9dfe8
Attachment #812099 - Flags: checked-in+
Everything is going well here so far. The latest set of nightlies for all platforms has finished and those updates are applying fine. The bigger test will be tomorrow when everyone that got this set of nightlies talks with aus4.mozilla.org.
Kairo suggested that we use the build system helpers we have to make sure this doesn't get uplifted. I'm going to do that in bug 922261.
Attached patch don't ride the trains (obsolete) — Splinter Review
Rob, Kairo suggested that we make sure this doesn't ride the trains. I think it makes sense to.
Attachment #812234 - Flags: review?(robert.bugzilla)
Comment on attachment 812234 [details] [diff] [review]
don't ride the trains

whoops, i messed up one of the issuers.
Attachment #812234 - Attachment is obsolete: true
Attachment #812234 - Flags: review?(robert.bugzilla)
What is the concern with this riding the trains especially with there being 4 weeks until the next merge (10-28)?
(In reply to Robert Strong [:rstrong] (do not email) from comment #19)
> What is the concern with this riding the trains especially with there being
> 4 weeks until the next merge (10-28)?

We haven't made a decision about when this is going to hit Aurora yet. It may jump sooner than the next merge, ride the train, or be held back (if we hit major issues). I'd rather be conservative about this until we make up our mind.
Attached patch fix issuer againSplinter Review
Attachment #812248 - Flags: review?(robert.bugzilla)
Comment on attachment 812248 [details] [diff] [review]
fix issuer again

I'm ok with this but please I'd prefer if you held off on landing this patch until it becomes clear that the change to aus4 shouldn't ride the trains.
Attachment #812248 - Flags: review?(robert.bugzilla) → review+
(In reply to Robert Strong [:rstrong] (do not email) from comment #22)
> Comment on attachment 812248 [details] [diff] [review]
> fix issuer again
> 
> I'm ok with this but please I'd prefer if you held off on landing this patch
> until it becomes clear that the change to aus4 shouldn't ride the trains.

That doesn't really address what I'm trying to though...I'm trying to guard against us forgetting to land something like this (or back out the change during the merge) if we make that decision. That's why I'd like to default to aus3.
Ok and go ahead and land it... it is just the opposite approach we usually take unless we are sure that we don't want something to ride the trains. You might also want to put this on the release drivers radar by using the tracking flag. I'll also add a calendar reminder for myself so I can remember to ping this bug in case nothing comes up so the patch can be backed out.
Duplicate of this bug: 922261
Thanks Rob. I'll land this tomorrow. I guess tracking-firefox27 is appropriate because that's what Nightly is currently?
Attachment #812248 - Flags: checked-in+
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD' patch leave out Android by design ?
(In reply to Nick Thomas [:nthomas] from comment #29)
> Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> patch leave out Android by design ?

This should apply to Android as well, I'd think. Ben?
Flags: needinfo?(bhearsum)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #30)
> (In reply to Nick Thomas [:nthomas] from comment #29)
> > Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> > patch leave out Android by design ?
> 
> This should apply to Android as well, I'd think. Ben?

Maybe -- I'm planning to talk to Nick and others this week about letting Balrog ride the trains to Aurora with 27. If we're doing that I'll need a different patch (one that lets Nightly and Aurora go through, but protects beta/release).
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum [:bhearsum] from comment #31)
> (In reply to Robert Kaiser (:kairo@mozilla.com) from comment #30)
> > (In reply to Nick Thomas [:nthomas] from comment #29)
> > > Great to see this all done. One quick query, did the 'ifdef NIGHTLY_BUILD'
> > > patch leave out Android by design ?
> > 
> > This should apply to Android as well, I'd think. Ben?
> 
> Maybe -- I'm planning to talk to Nick and others this week about letting
> Balrog ride the trains to Aurora with 27. If we're doing that I'll need a
> different patch (one that lets Nightly and Aurora go through, but protects
> beta/release).

Speaking of Aurora, I filed bug 927364 to track switching it to Balrog.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.