Closed Bug 1016620 Opened 12 years ago Closed 12 years ago

color - Elasticsearch on elasticsearch1.webapp.scl3.mozilla.com is CRITICAL: Elasticsearch Health is red

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Other
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nagiosapi, Unassigned)

References

()

Details

(Whiteboard: [id=nagios1.private.scl3.mozilla.com:362266])

Attachments

(1 file)

Automated alert report from nagios1.private.scl3.mozilla.com: Hostname: elasticsearch1.webapp.scl3.mozilla.com Service: color - Elasticsearch State: CRITICAL Output: Elasticsearch Health is red Runbook: http://m.allizom.org/color+-+Elasticsearch
Automated alert recovery: Hostname: elasticsearch1.webapp.scl3.mozilla.com Service: color - Elasticsearch State: OK Output: Elasticsearch Health is Green
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Attached file es_autolog_error.log —
TL,DR - Corrupt index on autolog. Attempted to fix from good copy; failed. Copied autolog from SCL3 prod cluster to autolog-bck in same cluster, deleted autolog index, created alias of "autolog" on the autolog-bck index. I tried the same "fix" as in https://bugzilla.mozilla.org/show_bug.cgi?id=1016215, but this only worked for a few moments before errors starting cropping up in the logs again. This time, I caught the full pattern of the errors, which is attached. Things start with a failed index merge, which leads to a failed recovery, which then leads to issues about who actually has a lock the index. I tried several things to fix the corrupt index by seeing if I could force a rebuild from good data, but was unable to get a good copy to stay up long enough to do so. There is some discussion of setting index.shard.check_on_startup to "fix" (https://github.com/elasticsearch/elasticsearch/issues/1871), but it's not clear to me if this can be applied only to a single index.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: