1286963 - RCA 2016-07-14 - Buildbot DB Table Crash

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Root Cause Analysis, task)

Comment 1

[Keegan Ferrando [:fauweh]]

Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1286942
and WP entry https://whistlepig.mozilla.org/en-US/detail/729/

Please complete the following form to the best of your ability for RCA
https://docs.google.com/document/d/14XzguY_SoWKP9kfkGwayL7u6f7HSfdOoBG8w-8siAgo/edit

Sheeri, please update RCA doc or delegate as necessary.

Thanks!

Comment 2

[Sheeri Cabral [:sheeri]]

RCA completed; feel free to schedule the postmortem whenever. Attendees should include me, Matt Pressman, Callek, coop, hwine, linda and natalie.

Comment 3

[hwine]

Sheeri: can you please add the data requested in bug 1286942 comment 3 to the RCA please?

Given that this occurred so quickly after the mid day fail over, we'd like to see data that shows there were no failover issues.

Also, I've heard rumors that the DB has been failed back in response to similar db corruption issues occurring. Can you add to the timeline any failovers which have occurred since the P1 event, please?

Keegan: can you add :catlee to the invitee list, please?

Comment 4

[Keegan Ferrando [:fauweh]]

Postmortem scheduled for Tue, July 26, 9am – 10am with requested participants and calendar invite sent.

Comment 5

[Sheeri Cabral [:sheeri]]

The master is still buildbot2, which was what we failed over to in https://bugzilla.mozilla.org/show_bug.cgi?id=1286942#c8, so I don't think the rumors are true.

There was corruption of a small table on 7/18, but that was fixed real-time, and AFAIK there were no outages (and definitely no failover).

Adding the other info requested into the RCA.

Comment 6

[hwine]

s/a bug 1287817 for another outage on 2016-07-19

Comment 7

[Peter Radcliffe [:pir]]

I'm unclear on status here. There are postmortem results in the doc but it calls for a followup meeting.

mpressman, do you have state on action items and any bugs for them?

Comment 8

[Peter Radcliffe [:pir]]

:hwine since this caused you pain, are you happy with the postmortem results?

Comment 9

[hwine]

Yes - I see no value in going deeper. For history, the following actions discussed during this meeting have been taken:
 - buildbot database backend converted from MyISAM to InnoDB
 - alerts about replication lag have been fixed
 - observations into the RCA process where provided

Comment 10

[Peter Radcliffe [:pir]]

Thanks :hwine.

With a postmortem done and action items taken I think this can be closed.