804658 - Brief outage of services on genericrhel6 prod pool

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Brandon Burton [:solarce]]

Failover of a component in the seamicro chassis that currently houses generic1-5.webapp.phx1 caused a brief outage of services hosted on the genericrhel6 cluster, including

* wiki.mozilla.org
* blog.mozilla.org
* tbpl.mozilla.org
* etc

Will post additional details shortly

Comment 1

[Brandon Burton [:solarce]]

Two factors played into this outage

1. Our blade server was set to draining due to maintenance work and was never put back into rotation
2. All five of our seamicro nodes were in the single chassis that had a component failover

To rectify this we're taking the following actions

1. We've re-enabled the blade and confirmed it's good to serve prod traffic
2. We're going to move generic4-5 to another seamicro chassis to prevent chassis failure causing an outage

We'll update this bug once #2 is complete and RF it

Comment 2

[matthew zeier [:mrz]]

We used to have a nagios alert that would flag on any zeus backend notes not marked as active.  Would this have caught that?

Comment 3

[Chris Turra [:cturra]]

(In reply to Brandon Burton [:solarce] from comment #1)
>
> 2. We're going to move generic4-5 to another seamicro chassis to prevent
> chassis failure causing an outage

this is now complete per bug 804669.

Comment 4

[Brandon Burton [:solarce]]

Per :cturra's work, RF!