Closed
Bug 984128
Opened 12 years ago
Closed 12 years ago
Elasticsearch4/5 (webapp) status is Yellow
Categories
(Infrastructure & Operations Graveyard :: WebOps: Other, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: rwatson, Assigned: cliang)
References
Details
elasticsearch5.webapp.scl3.mozilla.com and elasticsearch4.webapp.scl3.mozilla.com
Just alerted for yellow health.
Documentation state to file a bug.
Comment 1•12 years ago
|
||
elasticsearch[1,3-5].webapp.scl3 were all alerting yellow, tried restarting ES on all the nodes since > half were in a bad state and now they're mostly showing red.
Updated•12 years ago
|
Severity: normal → major
Updated•12 years ago
|
Assignee: server-ops-webops → cliang
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
| Reporter | ||
Comment 2•12 years ago
|
||
Same thing again today.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
| Assignee | ||
Comment 3•12 years ago
|
||
This morning, the cluster seemed to be in a worse state than it was yesterday. Most of the nodes were either unresponsive to local curl or reported back an error:
[cliang@elasticsearch2.webapp.scl3 ~]$ curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"error" : "MasterNotDiscoveredException[waited for [30s]]",
"status" : 503
}
I did a full stop of the cluster and restarted it.
My Current Working Theory is that the master node and non-communicative nodes suffered from some of the memory usage issues that are fixed in later versions of ES. We probably need to push the ES upgrades higher up on the priority list.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Comment 5•12 years ago
|
||
The cluster has again slowed down terribly:
[ashish@elasticsearch2.webapp.scl3 ~]$ time curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "es_prod_scl3",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 5,
"active_primary_shards" : 30,
"active_shards" : 60,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
real 4m12.100s
user 0m0.004s
sys 0m0.005s
and frequently timing out as well:
> [2014-03-18 04:51:37,592][WARN ][transport ] [elasticsearch2_scl3] Received response for a request that has timed out, sent [101008ms] ago, timed out [51177ms] ago, action [discovery/zen/fd/ping], node [[elasticsearch1_scl3][gRv2HTDKSdSGPFlzSDYGXw][inet[/10.22.81.100:9300]]], id [258884]
Gonna kick ES on all nodes.
Comment 6•12 years ago
|
||
Had to kick ES again after it slowed to a crawl.
Updated•7 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•