Closed Bug 984128 Opened 12 years ago Closed 12 years ago

Elasticsearch4/5 (webapp) status is Yellow

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

x86
macOS
task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rwatson, Assigned: cliang)

References

Details

elasticsearch5.webapp.scl3.mozilla.com and elasticsearch4.webapp.scl3.mozilla.com Just alerted for yellow health. Documentation state to file a bug.
elasticsearch[1,3-5].webapp.scl3 were all alerting yellow, tried restarting ES on all the nodes since > half were in a bad state and now they're mostly showing red.
Severity: normal → major
Assignee: server-ops-webops → cliang
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Same thing again today.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
This morning, the cluster seemed to be in a worse state than it was yesterday. Most of the nodes were either unresponsive to local curl or reported back an error: [cliang@elasticsearch2.webapp.scl3 ~]$ curl -XGET http://localhost:9200/_cluster/health?pretty=true { "error" : "MasterNotDiscoveredException[waited for [30s]]", "status" : 503 } I did a full stop of the cluster and restarted it. My Current Working Theory is that the master node and non-communicative nodes suffered from some of the memory usage issues that are fixed in later versions of ES. We probably need to push the ES upgrades higher up on the priority list.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
The cluster has again slowed down terribly: [ashish@elasticsearch2.webapp.scl3 ~]$ time curl -XGET http://localhost:9200/_cluster/health?pretty=true { "cluster_name" : "es_prod_scl3", "status" : "green", "timed_out" : false, "number_of_nodes" : 5, "number_of_data_nodes" : 5, "active_primary_shards" : 30, "active_shards" : 60, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } real 4m12.100s user 0m0.004s sys 0m0.005s and frequently timing out as well: > [2014-03-18 04:51:37,592][WARN ][transport ] [elasticsearch2_scl3] Received response for a request that has timed out, sent [101008ms] ago, timed out [51177ms] ago, action [discovery/zen/fd/ping], node [[elasticsearch1_scl3][gRv2HTDKSdSGPFlzSDYGXw][inet[/10.22.81.100:9300]]], id [258884] Gonna kick ES on all nodes.
Had to kick ES again after it slowed to a crawl.
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.