1201139 - 50% packet loss en route to blog.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: DC Other, task)

Description

[Shyam Mani [:fox2mike]]

At about 0924 PDT today, Webops got a series of alerts of servers not being reachable by New Relic. We also got a note from Dynect that bedrock-prod.zlb.phx.mozilla.net was unreachable/down from various locations.

At around the same time, various WPEngine blogs were alerting on Nagios...which seemed odd.

So I jumped on nagios1.private.phx1 and did some basic traceroutes :

blog.mozilla.org, 50% loss

https://www.dropbox.com/s/fabnudtiwia2m5x/Screenshot%202015-09-02%2009.48.43.png?dl=0

collector.newrelic.com, same route

https://www.dropbox.com/s/ucj64o5qxg2xod6/Screenshot%202015-09-02%2009.52.50.png?dl=0

Google DNS, clean 

https://www.dropbox.com/s/2ngo9w7ikmyemc7/Screenshot%202015-09-02%2009.59.20.png?dl=0

So obviously a portion of our telia route is in trouble, which relates to what we're seeing from monitoring.

Comment 1

[Dave Curado :dcurado]

If the problem is with our Telia connection in PHX1, we can just shut them down for a while.
Which is exactly what we just did.

Comment 2

[James Barnell [:jbarnell]]

Change title for clarity as it was not directly on our link.  Simply on the path.

Comment 3

[Dave Curado :dcurado]

This issue was resolved by shutting down our connection to Telia while they got whatever issues they were having corrected.  I turned our connection to them back on later in the evening.