Closed
Bug 1016620
Opened 12 years ago
Closed 12 years ago
color - Elasticsearch on elasticsearch1.webapp.scl3.mozilla.com is CRITICAL: Elasticsearch Health is red
Categories
(mozilla.org Graveyard :: Server Operations: MOC, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: nagiosapi, Unassigned)
References
(
URL
)
Details
(Whiteboard: [id=nagios1.private.scl3.mozilla.com:362266])
Attachments
(1 file)
|
7.77 KB,
text/plain
|
Details |
Automated alert report from nagios1.private.scl3.mozilla.com:
Hostname: elasticsearch1.webapp.scl3.mozilla.com
Service: color - Elasticsearch
State: CRITICAL
Output: Elasticsearch Health is red
Runbook: http://m.allizom.org/color+-+Elasticsearch
| Reporter | ||
Comment 1•12 years ago
|
||
Automated alert recovery:
Hostname: elasticsearch1.webapp.scl3.mozilla.com
Service: color - Elasticsearch
State: OK
Output: Elasticsearch Health is Green
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment 2•12 years ago
|
||
TL,DR - Corrupt index on autolog. Attempted to fix from good copy; failed. Copied autolog from SCL3 prod cluster to autolog-bck in same cluster, deleted autolog index, created alias of "autolog" on the autolog-bck index.
I tried the same "fix" as in https://bugzilla.mozilla.org/show_bug.cgi?id=1016215, but this only worked for a few moments before errors starting cropping up in the logs again. This time, I caught the full pattern of the errors, which is attached. Things start with a failed index merge, which leads to a failed recovery, which then leads to issues about who actually has a lock the index.
I tried several things to fix the corrupt index by seeing if I could force a rebuild from good data, but was unable to get a good copy to stay up long enough to do so. There is some discussion of setting index.shard.check_on_startup to "fix" (https://github.com/elasticsearch/elasticsearch/issues/1871), but it's not clear to me if this can be applied only to a single index.
| Assignee | ||
Updated•11 years ago
|
Product: mozilla.org → mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•