801475 - Short term Persona outage from London

Status: RESOLVED WORKSFORME

(Cloud Services :: Server: Identity, defect)

Description

[Shane Tomlinson [:stomlinson]]

I just tried to sign in to Mozillians using Persona and saw a short term outage. The request on Mozillians for include.js timed out. I browsed directly to login.persona.org and received an HTTP timeout. I opened a bash shell and pinged login.persona.org. The response was:

"PING persona.org (63.245.209.241): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
..."

While I was writing this email, I refreshed the login.persona.org page and it appeared. I went back to the shell and pinged again, this time with a different response:

"PING persona.org (63.245.217.134): 56 data bytes
92 bytes from phx1-persona.services.mozilla.com (63.245.217.134): Destination Port Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  78 5400 7ddb   0 0000  2e  01 32da 192.168.1.88 63.245.217.134"

Different servers.

Now, if I ping 63.245.209.241 directly, I see:
"
PING 63.245.209.241 (63.245.209.241): 56 data bytes
92 bytes from scl2-persona.services.mozilla.com (63.245.209.241): Destination Port Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 948e   0 0000  2e  01 2434 192.168.1.88 63.245.209.241
"

Would this have been caused by a blip in the routing? Was SLC2 inaccessible for a few?

Comment 1

[Gene Wood [:gene]]

> Would this have been caused by a blip in the routing? Was SLC2 inaccessible for a few?

No, this would be the DNS load balancing service (Dynect) resolving the DNS name to Phoenix. This isn't necessarily caused by an outage in SCL2 since the datacenters are active-active and load balanced with DNS. As far as was persona inaccessible for some period of time, I'm not sure but will take a look into it today.

Comment 2

[Gene Wood [:gene]]

I didn't find an indication that persona was unavailable at that time from our watchmouse global monitoring. Please feel free to report if this happens again.