Closed Bug 1171290 Opened 7 years ago Closed 7 years ago

CNAME request: pontoon.mozilla.org -> mozilla-pontoon.herokuapp.com

Categories

(Infrastructure & Operations :: SSL Certificates, task)

task
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: osmose, Assigned: fox2mike)

References

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1308] )

No description provided.
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1308]
Doh! Flubbed the enter key.

We'd like to switch the pontoon.mozilla.org subdomain to point to mozilla-pontoon.herokuapp.com instead of the current Pontoon service. We're planning to move the database from the current service over at the same time, so we'd like to coordinate a specific time to handle this. We also would like to disable a cronjob running on the current Pontoon service as part of this migration.

The steps we'd like to happen (old-pontoon refers to the current service at pontoon.mozilla.org, heroku-pontoon refers to the Heroku-hosted version at mozilla-pontoon.herokuapp.com):

1. pontoon.mozilla.org switched to hardhat.mozilla.net during the migration.

2. Cronjob running the `./manage.py update_projects` and `./manage.py commit_projects` commands on old-pontoon is disabled (mathjazz can provide details if there's confusion about which cronjob, I'm unsure what the current setup is for old-pontoon).

3. I migrate data from current Pontoon to new Heroku-hosted instance.

4. pontoon.mozilla.org switched to point to mozilla-pontoon.herokuapp.com. At some point the SSL certificate for the domain will need to be uploaded to Heroku; I can handle thatif given the necessary files, or lonnen can give access to whoever needs it to upload it themselves to the app in Heroku.

5. mathjazz and I test the service, and if anything is wrong and cannot be fixed promptly, the domain is switched back to point and old-pontoon, the cronjob is re-enabled, and we reschedule the migration. If everything checks out, we're done!

After this transfer we'll file a bug for decommissioning old-pontoon.

Since mathjazz will be on PTO next week, ideally we'd like to handle this tomorrow, but that's completely unreasonable of us to expect, so the next best date for us would be June 18th (I'll be at a local conference the 15th-17th).
Summary: Switch pontoon.mozilla.org domain over to mozilla-pontoon.herokuapp.com → CNAME request: pontoon.mozilla.org -> mozilla-pontoon.herokuapp.com
(In reply to Michael Kelly [:mkelly,:Osmose] from comment #1)
> 4. pontoon.mozilla.org switched to point to mozilla-pontoon.herokuapp.com.
> At some point the SSL certificate for the domain will need to be uploaded to
> Heroku; I can handle thatif given the necessary files, or lonnen can give
> access to whoever needs it to upload it themselves to the app in Heroku.

Setting needinfo? myself to make sure we get the SSL certificate ready. Note that it will *only* support 'pontoon.mozilla.org'.
Flags: needinfo?(rsoderberg)
June 18th it is.
On second thoughts, let's do this after Whistler please. Too many people traveling and on PTO etc to do it on the 18th. Thanks!
Assignee: server-ops-webops → smani
(In reply to Shyam Mani [:fox2mike] from comment #4)
> On second thoughts, let's do this after Whistler please. Too many people
> traveling and on PTO etc to do it on the 18th. Thanks!

Fine by me!
So, Michael and I would like to do this transition this week.

How about Wednesday, July 1?
Flags: needinfo?(smani)
Sure thing. What time works for you guys?
Flags: needinfo?(smani)
10 AM PDT works for me and Michael. You?
Flags: needinfo?(smani)
As agreed on IRC, let's do it at 1300 PDT.
Flags: needinfo?(smani)
From the webops side, here are the things we did (for future reference)

1) Setup SSL on heroku
   * Included setting up the SSL addon
   * Setup the Heroku Toolbelt (for CLI)
   * Ran heroku certs:add mozilla/ssl-certs/pontoon.mozilla.org.crt mozilla/ssl-certs/pontoon.mozilla.org.key --app mozilla-pontoon
   * Verified with heroku certs --app mozilla-pontoon 
   * Ran heroku domains:add pontoon.mozilla.org --app mozilla-pontoon
   * Verified with heroku domains --app mozilla-pontoon 

2) Redirect Zeus to hardhat. 

3) Wait for devs to move the app. 

4) Change Mozilla DNS to point to the newly setup heroku SSL endpoint
   * Ran invtool CNAME delete --pk <redacted>
   * Ran invtool CNAME create --fqdn pontoon.mozilla.org --ttl 60 --target oita-4124.herokussl.com --private --public --description "Migrating pontoon.mozilla.org to Heroku" --comment "Bug 1171290"

If there are any issues, reverting this is as easy as :

1) Delete pontoon.mozilla.org CNAME from inventory
2) Add pontoon.mozilla.org CNAME to inventory pointing at pontoon-zlb.vips.scl3.mozilla.com with a TTL of 60. 

All good here, closing :)
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(rsoderberg)
Resolution: --- → FIXED
@mathjazz, @mkelly: 

A gentle reminder that, once you've confirmed all is good with Heroku and are happy with it, it'd be great if you gave us a sign that it's safe to decomm the old VMs, configs, etc. =)
C. Liang:

Yeah, nagios keeps telling me about 100% packet loss every two hours. :-D

I see ~150 new submissions so far, and we didn't get a single complaint! The makes me confident in saying that if something unexpected comes up, we should be able to fix it without switching back to the old server.

@mkelly?

Before we decommission the dev server, we should probably point http://pontoon-dev.allizom.org/ towards https://mozilla-pontoon-staging.herokuapp.com/
(In reply to Matjaz Horvat [:mathjazz] from comment #12)
> C. Liang:
> 
> Yeah, nagios keeps telling me about 100% packet loss every two hours. :-D
> 
> I see ~150 new submissions so far, and we didn't get a single complaint! The
> makes me confident in saying that if something unexpected comes up, we
> should be able to fix it without switching back to the old server.
> 
> @mkelly?

This is promising but I'd feel better waiting a few more days. I've already got a reminder set for next week to file a decommission bug, does that sound good?
 
> Before we decommission the dev server, we should probably point
> http://pontoon-dev.allizom.org/ towards
> https://mozilla-pontoon-staging.herokuapp.com/

As per bug 1162805, we can't really do this without exposing the wildcard cert to too many people. An alternative I've been considering would be to use mozilla.io, as we have access to the cert for it. I can file a bug about that today.

In either case I think we can call this bug verified, eh?
Status: RESOLVED → VERIFIED
What's the bug # for resolving the pontoon nagios alerts issue?
See Also: → 1206112
You need to log in before you can comment on or make changes to this bug.