1241852 - [careers] Change DNS entries for careers.m.o and careers.a.o (Move to AWS)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[Giorgos Logiotatidis [:giorgos]]

Hi Ops,

We have a new home for Careers website. Please change DNS to

 - careers.mozilla.org CNAME careers-mozilla-org-36000613.us-west-2.elb.amazonaws.com 

 - careers.allizom.org CNAME allizom-org-234627437.us-west-2.elb.amazonaws.com

Please verify by visiting

 https://careers.mozilla.org/static/revision.txt

and

 https://careers.allizom.org/static/revision.txt

both must return da69f52fdef4cec250988efc8dab39bd5fcf6bf0

Thanks!

Comment 1

[Ryan Watson [:w0ts0n]]

Pushed the change, waiting for propagation.

Comment 2

[Ryan Watson [:w0ts0n]]

Pushed allizom first, the change applied and worked. 
Then pushed prod, we saw nagios alerts and the lookup didn't work, so I swiftly changed it back. 
:giorgos can you reach out to me on IRC when you have a second.

Comment 3

[Ludovic Hirlimann [:Usul]]

or our pingdom checks is failing :
Go to URL http://careers.mozilla.org/admin
HTTP status code should be 401

it's retruning 200 is that check still valid ? or should I change it ?

Comment 4

[Ryan Watson [:w0ts0n]]

For bug info. We updated the DNS again and it looks like it took this time.

Comment 5

[Ryan Watson [:w0ts0n]]

Nagios is still reporting down. ICMP pings to the nodes in AWS are not guaranteed to work so I disabled the ping check by putting in a dummy ping check for that service. the https-websites is still valid.

giorgos I assume you having monitoring in place for careers now?

Sending        phx1.pp
Transmitting file data .
Committed revision 114215.

Comment 6

[Ludovic Hirlimann [:Usul]]

[ludo@Oulanl mozilla]$ svn diff
Index: services.pp
===================================================================
--- services.pp	(revision 114215)
+++ services.pp	(working copy)
@@ -4800,7 +4800,6 @@
             normal_check_interval => 5,
             hostgroups => $::fqdn ? {
                 'nagios1.private.phx1.mozilla.com' => [
-                    'python-cluster-http-websites',
                 ],
                 default => [
                 ]
@@ -4812,7 +4811,6 @@
             normal_check_interval => 5,
             hostgroups => $::fqdn ? {
                 'nagios1.private.phx1.mozilla.com' => [
-                    'python-cluster-https-websites',
                 ],
                 default => [
                 ]
[ludo@Oulanl mozilla]$ svn commit -m"fixing 1241852"
Sending        services.pp
Transmitting file data .done
Committing transaction...
Committed revision 114216.
[ludo@Oulanl mozilla]$

Comment 7

[Ben (:bensternthal)]

I appreciate wanting to get this done but in the future no changes like this on a Friday, especially because this is end of day "Giorgos Time".

Comment 8

[Ryan C [:ryanc] (UTC-4)]

Had this come in,

12:41:55 <nagios-phx1> careers.mozilla.org:HTTPS - SSL Cert expiration is CRITICAL: CRITICAL - Socket timeout after 10 seconds (http://m.mozilla.org/HTTPS+-+SSL+Cert+expiration)

Fixed in sysadmins r114261,

13:11:31 <@nagios-scl3> careers.mozilla.org:HTTPS - SSL Cert expiration is OK - OK - Certificate 'careers.mozilla.org' will expire on 2017-11-09 04:00 -0800/PST. Last Checked: 2016-01-23 13:08:39 PST

Comment 9

[Giorgos Logiotatidis [:giorgos]]

(In reply to Ben (:bensternthal) from comment #7)
> I appreciate wanting to get this done but in the future no changes like this
> on a Friday, especially because this is end of day "Giorgos Time".

ftr we worked both stage and prod together with w0ts0n and verified that it worked OK during european time. AFAIK there's work to be done to update the nagios alerts to play well with AWS. 

That said I'll avoid filing similar bugs in the future on Fridays. Thanks Ben!

Comment 10

[Giorgos Logiotatidis [:giorgos]]

(In reply to Ryan Watson [:w0ts0n] from comment #5)
> giorgos I assume you having monitoring in place for careers now?

I've a NR Application alert policy configured to http get both prod and stage and email me and ping vectorvictor bot on IRC

https://rpm.newrelic.com/accounts/263620/application_alert_policies?search[q]=careers.mozilla.org

Comment 11

[Ryan Watson [:w0ts0n]]

As per bug: 1242454 this still needs some monitoring work done, but this bug can be closed.