1219321 - Etherpad data integrity issues

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Sheeri Cabral [:sheeri]]

Etherpad has data integrity issues, and we'd like to fail over the db to the slave to try to mitigate.

Comment 1

[Sheeri Cabral [:sheeri]]

Etherpad db was filling up disk space in /, so I moved the db to /data. 

In the process, the MyISAM table "store" was marked as crashed and needed to be repaired. The repair restored 120 records:

2015-10-28 16:27:59 13292 [Note] Found 4143174 of 4143054 rows when repairing './etherpad/store'
+----------------+--------+----------+---------------------------------------------------+
| Table          | Op     | Msg_type | Msg_text                                          |
+----------------+--------+----------+---------------------------------------------------+
| etherpad.store | repair | info     | Wrong bytesec: 108-108- 44 at 1116783308; Skipped |
| etherpad.store | repair | info     | Wrong bytesec: 115-115-105 at 1116778176; Skipped |
| etherpad.store | repair | info     | Wrong bytesec:  52- 71-116 at 1116785704; Skipped |
| etherpad.store | repair | info     | Wrong bytesec:  58-106-105 at 1116785548; Skipped |
| etherpad.store | repair | info     | Wrong bytesec:  58- 49-110 at 1116794512; Skipped |
| etherpad.store | repair | warning  | Number of rows changed from 4143054 to 4143174    |
| etherpad.store | repair | status   | OK                                                |
+----------------+--------+----------+---------------------------------------------------+
7 rows in set (35.71 sec)

This resulted in a 4 minute outage from 16:23 to 16:27 UTC (9:23 - 9:27 Pacific).

Comment 2

[Sheeri Cabral [:sheeri]]

Received a complaint that https://public.etherpad-mozilla.org/p/measurement-team-meeting-notes is blank after the outage. 

Etherpad's database is not set up in a way that we can extract text or history without using the API. There aren't any commandline tools available, so we'd like to fail over to the redundant slave (which didn't crash) in the hopes that the history/text is still there.

Comment 3

[Sheeri Cabral [:sheeri]]

needinfo'ing jakem, as this needs updating: https://mana.mozilla.org/wiki/display/websites/etherpad.mozilla.org#etherpad.mozilla.org-RestartEtherpad

Comment 4

[Sheeri Cabral [:sheeri]]

svn sysadmins repo r109826 committed to change config of etherpad db's to swap master and slave (configs only, nothing changes until the lb changes).

Comment 5

[Sheeri Cabral [:sheeri]]

:atoll stopped etherpad, I updated the load balancer, :atoll restarted etherpad. Functionality is good, unfortunately the etherpad that lost data, overwrote with an empty pad, so the slave did not have any history.

There may be other pads that lost data, but due to the nature of how etherpad stores data in the db, it's not possible to sleuth out how many pads were affected.

Comment 6

[Jake Maul [:jakem]]

Seems closable.