1032299 - requirement to move the /22 aggregate block for PHX1 from fw1.phx1 to border[12].phx1

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Dave Curado :dcurado]

Currently, the /22 IP supernet that is used for the PHX1 data center is 
configured on fw1.phx1.  This is an odd choice, as it suggests to the network that fw1.phx1 uses all of that /22.  
That's not true.  core[12].phx1 and fw1.phx1 use a small subnet from within
that /22.
We would like to move the /22 supernet aggregate route from fw1.phx1
to border[12].phx1.  We believe we can do this without any down time, 
but recommend we do this work at some off hour, just in case.

steps involved:
 - determine the subnets used by fw1.phx1 and core[12].phx1
 - ensure that those routes are in place on border[12].phx1
 - ensure that all other more specific subnets are in place
   on border[12].phx1 (the subnets used by svc1.phx1 and 
   corp.phx1)
 - create the aggregate route statement on both border[12].phx1
 - delete the aggregate route statement from fw1.phx1
 - ensure that the aggregate route is still being announced to 
   the Internet
 - the "band aid" static routes on fw1.phx1 and core[12].phx1 for 
   the svc.phx1 BU and corp.phx1 BU can now be removed

Comment 1

[Shyam Mani [:fox2mike]]

Approved by the CAB on 2 July to be carried out on 5 July between 1800 and 2100 PDT. Notice to be sent via Whistlepig.

Comment 2

[Dave Curado :dcurado]

Well, that didn't go well.
We got the subnets advertised via ospf from fw1.phx1, but moving the aggregate route for
the /22 for PHX1 did not go well.
For some reason, moving that aggregate route to the border routers caused the route
to no longer be reachable from the global Internet.  
I'm still trying to determine why, as that should not have been the case.
e.g. checking what we were advertising to our upstream providers showed that we
were still advertising the /22 supernet for PHX1 to all of them.

I'll put together a plan where I can test this in PHX1 without causing an outage,
and try again.

Comment 3

[david garvey:dgarvey]

Here are the alerts I received. We have nagios, pingdom and catchpoint alerts.

Nagios:

<nagios-phx1> Sat 19:00:30 PDT [1197] detectportal.firefox.com is DOWN :CRITICAL - Socket timeout after 10 seconds

<nagios-scl3> Sat 19:01:06 PDT [5481] nagios1.private.euw1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%
 Sat 19:05:16 PDT [1211] ns1.private.phx1.mozilla.com:Recursive DNS is CRITICAL: CRITICAL - Plugin timed out while executing system call (http://m.mozilla.org/Recursive+DNS)

<nagios-phx1> Sat 19:04:13 PDT [1207] signer2.addons.phx1.mozilla.com:signer is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. (http://m.mozilla.org/signer)

<@nagios-corp-phx1> Sat 19:05:32 PDT [3672] ns1.private.corp.phx1.mozilla.com:Recursive DNS is CRITICAL: CRITICAL - Plugin timed out while executing system call (http://m.mozilla.org/Recursive+DNS)

<nagios-phx1> Sat 19:05:45 PDT [1212] mozillaignite.org:HTTP - / is CRITICAL: CRITICAL - Socket timeout after 10 seconds (http://m.mozilla.org/HTTP+-+/

<@nagios-phx1> Sat 19:05:55 PDT [1213] signer1.addons.phx1.mozilla.com:signer is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. (http://m.mozilla.org/signer)

<@nagios-corp-phx1> Sat 19:05:59 PDT [3673] ns2.private.corp.phx1.mozilla.com:Recursive DNS is CRITICAL: CRITICAL - Plugin timed out while executing system call (http://m.mozilla.org/Recursive+DNS)

<nagios-phx1> Sat 19:07:20 PDT [1219] bugzilla.zlb.phx.mozilla.net:http - bugzilla.m.o is CRITICAL: Name or service not known (http://m.mozilla.org/http+-+bugzilla.m.o)

<nagios-phx1> Sat 19:08:59 PDT [1223] wildcard.bugzilla.mozilla.org:HTTPS - SSL Cert expiration is CRITICAL: CRITICAL - Socket timeout after 10 seconds (http://m.mozilla.org/HTTPS+-+SSL+Cert+expiration)

<nagios-pek1> Sat 19:12:37 PDT [6330] addons-zlb.vips.pek1.mozilla.com:https - addons.mozilla.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 630 bytes in 4.931 second response time (http://m.mozilla.org/https+-+addons.mozilla.org)

<nagios-phx1> Sat 19:12:49 PDT [1231] proxy2.dmz.phx1.mozilla.com:HTTP Proxy - www.mozilla.org is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. (http://m.mozilla.org/HTTP+Proxy+-+www.mozilla.org)




external sites down Pingdom monitoring:

www.getpersonas.com

mozilla.service-now.com

snmp.mozilla.com

mail.mozilla.com



External sites down Catchpoint. WorldWide monitoring. alert was connection unknown.

mail.mozilla.com

addons.mozilla.org

www.getpersonas.com

input.mozilla.com

reps.mozilla.org

wiki.mozilla.com

support.mozilla.org

etherpad.mozilla.org

us.start.mozilla.com

Comment 4

[Dave Curado :dcurado]

Thank you -- that is helpful in debugging this.
The information in comment #3 indicates that the entire /22 went unreachable.
e.g. converting the above list of hosts into IP addresses shows that the hosts were
spread over all 3 BUs in PHX1.

Comment 5

[Laura Thomson :laura]

:dcurado, I also got Nimsoft alerts for the following sites not mentioned above during the window:
crash-stats.mozilla.com ("Timeout while connecting")
treestatus.mozilla.org ("Failed to connect to treestatus.mozilla.org port 443: Network is
unreachable")

I don't think it's hooked up to any of the external monitors but based on related nagios alerts I'm fairly sure crash-reports.mozilla.com  was also unreachable. 

Hope this helps.

Comment 6

[Lonnen :lonnen]

The outage should be better communicated to the infra teams. There's nothing about it in status.m.o, nothing on the infra teams mailing lists.

Comment 7

[Dave Curado :dcurado]

I didn't know that was a requirement.
We had approval to do this work via the Change Advisory Board.
We did the work hand in hand with the MOC so they knew what was going on.
No one has ever told me that there were further requirements that need to filled.

I will bring up your concerns with the MOC manager and the manager of NetOps so that
if the processes we follow need to be updated, they will be.

Thanks.

Comment 8

[Lonnen :lonnen]

Thank you!

Comment 9

[Dave Curado :dcurado]

So, here we go again!
We still need to move this routing announcements off of fw1.phx1 and on to the border routers
in PHX1.  Since this has been a difficult change to get done, it seems like a good idea to 
explain it a bit more.
Currently, fw1.phx1, the firewall that is used in the shared infrastructure part of the
data center is advertising to the world, "come to me to get to anything in the phx1
datacenter!"  -- this of course, is not true; other BUs in the data center (SVC/MKT and CORP)
use some of that IP address space.  It works, because as inbound packets make their way
across the Internet and into PHX1, they hit the border routers first, and the border routers
route the packets to the right place.  But what happens if fw1.phx1 fails?  (and as I am
fond of saying, *everything* breaks, eventually...)  If that happens, then all of PHX1
goes off the air, even though most of the infrastructure is fine!  
That's the problem we're trying to fix.

By having the *border routers* say to the world, "come to me to get to anything in PHX1",
a) it is the truth
b) we'd have to lose both border1.phx1 and border2.phx1 for the routing announcement to
   stop.  If both those routers fail, then PHX1 is going off the air anyway. 

I have done several tests to find a way to make this change without causing an outage.
I'm relatively confident that we have a way to do so. 

Maintenance window time: 2 hours
Impact: Hopefully none, but yes, we could end up taking PHX1 off the air for 
        several minutes.  If we see that failure, we can back out of the changes.
        (One problem is that when we have a change failure, we really would
        benefit from being able to figure out the problem, rather than just 
        backing out the change.  We don't have a network.dev or a network.stage
        to test this stuff out on.  We're working on putting a lab together, but
        even then it will be a challenge to accurately reproduce the production
        environment.)

Comment 10

[Shyam Mani [:fox2mike]]

We discussed this in the CAB today and this is important enough that we need to do this ASAP and not wait for the next TCW. Having said that, we need to notify external partners because this affects Marketplace. 

Needinfo'ing Jason, so we can find a time slot that works for them.

Needinfo'ing Hal, so we can get releng sign off (since this might affect TBPL) 

Once we have more info, we will pick a time and go through with this change.

Comment 11

[Jason Thomas [:jason]]

As per bug 1023481 Comment 8 we need a few business days to notify our partners.

Could we schedule this maintenance for July 26th 3AM - 5AM PDT?

a? clouserw
a? kward

Comment 12

[Dave Curado :dcurado]

Well, we could use that date/time, but here's the thing:
there's a lot of clean up work we need to do on this network.
We've defined 200+ action items we need to do.
We've done 68 of them, we have 205 to go.
10 days is a long time to wait, and doing this work at tough hours (3am)
is probably not sustainable.  

Maybe we need to map out a set of change windows, stretching a ways into
the future?  I'd welcome everyone's input on this.

Comment 13

[Michal Purzynski [:michal`] (use NEEDINFO)]

Is the list just for the PHX1 or the MozNet as a whole? Can you attach it to the bug as some kind of a context information to know where we are and where are we heading?

Just my 5c.

Comment 14

[Dave Curado :dcurado]

The list covers the entire network.
Yes, we can share it.  
I'm not sure it makes sense to include it into this bug.
Maybe a bug of its own?

Comment 15

[Wil Clouser [:clouserw]]

(In reply to Jason Thomas [:jason] from comment #11)
> As per bug 1023481 Comment 8 we need a few business days to notify our
> partners.
> 
> Could we schedule this maintenance for July 26th 3AM - 5AM PDT?
> 
> a? clouserw
> a? kward

That's fine with me.

(In reply to Dave Curado :dcurado from comment #12)
> Well, we could use that date/time, but here's the thing:
> there's a lot of clean up work we need to do on this network.
> We've defined 200+ action items we need to do.
> We've done 68 of them, we have 205 to go.
> 10 days is a long time to wait, and doing this work at tough hours (3am)
> is probably not sustainable.  
> 
> Maybe we need to map out a set of change windows, stretching a ways into
> the future?  I'd welcome everyone's input on this.

What date/time do you want to do it?  The 18th or 19th?

Comment 16

[hwine]

Dave,

Is it fair to summarize the potential impact as:
 - there will be a 2 hour work window
 - during that 2 hours, worst case is one 30 minute internet outage

I thought that is what I heard in CAB -- do I have that right?

Comment 17

[Michal Purzynski [:michal`] (use NEEDINFO)]

Agreed, Dave - a bug on its own will be so much better. I'm shutting in this bug then.

Comment 18

[Dave Curado :dcurado]

Hi Wil -- The 19th or 20th, maybe 7am PST?  Or even some time during next week,
again at like 6 or 7 am PST?  (I will be on PST next week, so...)

Comment 19

[Dave Curado :dcurado]

Hi Hal,
Yes, that's a fair summary.
Thanks!

Comment 20

[Dave Curado :dcurado]

Hi Michal -- OK, let me get with James and make sure I put the list of action items
into a format that he thinks makes sense for everyone, and then I'll get it done.

Thanks!

Comment 21

[Wil Clouser [:clouserw]]

19th or 20th at 7am would work best I think (I'm thinking of Europe/Asia here).  Unless the time is tonight, I don't think there is a time that you'll get objections too - I just need to know when. :)

Comment 22

[Dave Curado :dcurado]

OK, let's say the 20th, at 7am PST.
Thank you very much!
Dave

Comment 23

[Lonnen :lonnen]

+1

Comment 24

[Dave Curado :dcurado]

This work was successfully completed.