1173905 - status.mozilla.org didn't accurately reflect "Service Disruption" for SUMO/https://support.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Projects, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Problem: 

tl;dr: the monitoring/alerting powering the status for SUMO on status.mozilla.org needs investigation.

Yesterday, June 10th, https://support.mozilla.org didn't reflect the proper status of "Service Disruption" -- it remained green ("Service is operating normally") throughout the HTTP 503 errors.

(Around 2:05pm PDT, according to users in #sumodev, the site started displaying HTTP 503 - Service Unavailable errors, intermittently, and lasted until the all clear around 3:15 or so.)

Looks like New Relic picked up the error and alerted, #moc was engaged, nagios fired, etc. -- so I want to scope this bug to just the disconnect between the "all clear" that https://status.mozilla.org/ displayed, and the reality of the site's disruption for almost an hour.

New Relic: https://rpm.newrelic.com/accounts/263620/applications/2779374/downtime (might not be around forever, so I'll attach a few screenshots for posterity).

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: CHK_s4tUIAEsLRC.png_large.png

1) status.mozilla.org status showing green for SUMO
2) site returning HTTP 503 - Service Temporarily Unavailable
3) New Relic showing HTTP 503 errors w/duration

Comment 2

[Ashish Vijayaram [:ashish]]

Attachment: pingdom incident list

For support.mozilla.org alerts, Pingdom informs oncall (via pagerduty) in 5 mins, statushub in 10 mins. The longest outage was 9 mins, so Pingdom didn't alert statushub.

The alerting policy seems to be in-line with other websites that share the same "criticality".

Comment 3

[Michael Cooper [:mythmon]]

I agree status.mozilla.org should have reported this. However, I'd like to note that the outage of the site was not complete. Throughout the entire window, the site would sometimes load, sometimes not. This maybe why status.mozilla.org was never triggered.

Comment 4

[Ashish Vijayaram [:ashish]]

Yeah, it was just the nature of the incident that made it dodge statusmo. I'm inclined to call this a one-off. I've checked the alerting pipeline and there's nothing wrong with that. I don't find a need to change the alerting policy as well.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Ashish Vijayaram [:ashish] from comment #4)
> Yeah, it was just the nature of the incident that made it dodge statusmo.
> I'm inclined to call this a one-off. I've checked the alerting pipeline and
> there's nothing wrong with that. I don't find a need to change the alerting
> policy as well.

So if https://www.mozilla.org were down for 9 minutes, intermittently, would status.mozilla.org exhibit the same "all clear/green" status?

Comment 6

[Ashish Vijayaram [:ashish]]

That is correct. statusmo will not change unless the outage is at least 10 mins long.