It's over now (so dry your eyes), but there was a pretty bad "blip" in BSD's response time for https://sendto.mozilla.org/page/contribute/givenow-seq?source=base_url#page-1 Like, of the 70+ seconds variety: http://www.webpagetest.org/result/141211_H8_5SE/ (Really, I'm just capturing this for the historical record.) We discovered this vis-a-vie: http://selenium.qa.mtv2.mozilla.com:8080/job/snippets.prod/50119/console (which probably won't survive Jenkins' retention policy, long-term, so here, in short form): 18:11:43 E AssertionError: Bad URL https://sendto.mozilla.org/page/contribute/givenow-seq?ref=EOYFR2014&utm_campaign=EOYFR2014&utm_source=firefox&utm_medium=snippet&utm_content=SelectAMT_test2#amount-10 found in /3/Firefox/default/default/default/en-US/release/default/default/default/
Simultaneous to that capture it would seem, we saw this alert come through: You are receiving this email because your Amazon CloudWatch Alarm "sendto.mozilla BSD HealthCheck Alarm" in the US - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 2 datapoints were less than or equal to the threshold (0.5). The most recent datapoints: [0.5, 0.5]." at "Thursday 11 December, 2014 02:10:46 UTC". View this alarm in the AWS Management Console: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarm=sendto.mozilla%20BSD%20HealthCheck%20Alarm Alarm Details: - Name: sendto.mozilla BSD HealthCheck Alarm - Description: - State Change: OK -> ALARM Threshold: - The alarm is in the ALARM state when the metric is LessThanOrEqualToThreshold 0.5 for 60 seconds.
I setup another couple monitors today: http://opsview.mofoprod.net:3000/viewport/EOY I have an open ticket with BSD and will be talking with Seth tomorrow after our mofo meeting.
JP, not sure which resolution, but I'm guessing this particular incident is no longer actionable :-)