Closed Bug 690033 Opened 14 years ago Closed 14 years ago

It's taking 16s to verify zeus can talk to AMO

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: clouserw, Assigned: oremj)

Details

(Whiteboard: Waiting for Zeus support.)

On https://addons.mozilla.org/services/monitor we do a bunch of tests to make sure AMO is structurally sound. oremj recently modified the page to show how long each test is taking and it looks like the zeus tests are taking around 16s right now to complete. Nagios hits this page once per minute (I think?) on each webhead, so that's 30 hits per zeus box per minute. The zeus check opens an authenticated connection to zeus and runs the getGlobalCacheInfo() call.[1] I think this got substantially worse when we moved to phx and got more zeus boxes, but I'm not sure how long that call takes per box. I don't have a grand idea on how to solve this, so the bug is more about starting a discussion on options. [1] The call is merely to see if the authentication is correct, we don't actually look at the payload beyond success or failure. If there is a better way to do this or a cheaper call to make, let me know.
The Zeus admin interface has been very slow since we moved over AMO and we've been working with tech support for about a month now trying to resolve the issues. I assume the issues are related and resolving our issue will resolve yours.
Assignee: server-ops → jeremy.orem+bugs
Whiteboard: Waiting for Zeus support.
We had to disable HERA in the monitor because of this. Any updates from Zeus support?
They were on our servers today for about 4 hours and weren't able to pinpoint the problem. They are going to continue investigating.
Could you ask sometime if there is a more correct way of telling if authentication has worked?
Asked.
"Currently, this is a known issue and we do have a feature request set with Development right now but there is no ETA at this time."
Weak sauce. Also, they didn't reply-all. Double weak sauce.
Yeah, they usually don't. Sometimes we get lucky and they do.
It's not that useful to keep this bug open. We have a ticket open with them still and they are actively working on the issue.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
How will we (AMO team) know when it's resolved? Right now the checks are commented out in our source and this is the only bug we can point to
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.