701049 - AMO returning "Service Unavailable" -- Zeus errors

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[tofumatt [:tofumatt]]

Attachment: Screenshot of error

clouserw mentioned there were some Zeus errors on AMO yesterday. Today, whilst trying to log in I was getting REALLY long-running requests that would eventually time out and show the error paged attached. After I managed to log in, I'd get it on various pages around the site.

My account on the site is under my email "[e-mail]" if that's in the logs anyplace and will help. This has been happening for a good third of the request I've made to AMO prod this afternoon.

Comment 1

[Jeremy Orem [:oremj]]

Username isn't recorded in the logs. Can you give us the exact time frame?

Comment 2

[tofumatt [:tofumatt]]

Started today (Nov.9) around 12:30pm Atlantic Time until at least 1:16:04 PM AST. (We're -4 GMT.)

I had a fair amount of log in problems at ~12:35pm.

Comment 3

[Wil Clouser [:clouserw]]

There have been multiple reports this morning in #amo, no specific time frames.  You can see some traffic dips from ganglia: https://addons-dev.allizom.org/z/services/graphite/addons

Looks like when we get above 2500 concurrent sessions the graphs start to get shaky.

Comment 4

[Wil Clouser [:clouserw]]

There is a pretty steady stream of these errors on AMO prod:

OperationalError: (2003, "Can't connect to MySQL server on 'db-amo-ro' (110)")

Comment 5

[tofumatt [:tofumatt]]

We're also getting tracebacks on Mozillians prod to the tune of:

OperationalError: (2003, "Can't connect to MySQL server on 'generic-rw-zeus' (111)")

Comment 7

[Matt Brandt [:mbrandt]]

(In reply to Matthew Riley MacPherson [:tofumatt] from comment #5)
> We're also getting tracebacks on Mozillians prod to the tune of:
> 
> OperationalError: (2003, "Can't connect to MySQL server on 'generic-rw-zeus'
> (111)")

The bug 701049 has the Mozillians traceback information attached to it.

Comment 9

[Corey Shields [:cshields]]

quick update:

we had Zeus devs looking at our cluster in phx1 for most of yesterday.  They discovered some slowness and issues in their code that might cause slowness, but I don't think they have discovered a root cause yet.

In the meantime, we are working on procuring new servers to test zeus on that would be hosted outside of the blade environment and have more bandwidth to them.  Unfortunately some components are on backorder (~3 weeks) due to the flooding overseas, so this 'solution' is a ways off.

Comment 10

[Jeremy Orem [:oremj]]

I don't think these severe problems are still happening.

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Post-fix screenshot

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED, as best I can tell.