733832 - Significant reduction in overall AMO log data since switching to the netscaler solution

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Previously, the addons.mozilla.org site was hosted by a portion of the pp-zlb* zeus cluster.  Looking at the logs for yesterday vs today indicates that no other known zeus cluster was handling any significant amount of traffic for AMO.  That means that it is highly unlikely that we were dealing with an inflation of log data due to duplicate requests made by one zeus cluster to another zeus cluster.

The overall volume of log data decreased by about 20% after the switchover.  This resulted in ADI and download metrics for individual add-ons dropping anywhere from 10% to 25% and the Firefox ADI being down about 17%.

We have to figure out what the cause is, or revert back to the pp-zlb cluster if we can't figure it out quickly.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Attachment: Histogram of overall AMO log data by hour on the 5th and 6th

Comment 2

[Jeremy Orem [:oremj]]

We've reverted to Zeus.

Comment 3

[Wil Clouser [:clouserw]]

Is there a reason this bug isn't public?  If not please unhide it so we can link to it.