769973 - it-elasticsearch issues 30/6/2012

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Corey Shields [:cshields]]

the IT Elastic Search cluster is bit by today's leap second bug and needs some TLC.

Comment 1

[Brandon Burton [:solarce]]

Currently all three nodes in our ES cluster have been stopped cleanly, then rebooted.

Cluster should rebalance shortly and I'll close once it is "green" again

Comment 2

[Brandon Burton [:solarce]]

Based on current progress, I estimate another 45-50 minutes before all the shards rebalance and the cluster returns to green, but overall load is low and all ES operations are being processed within expected time frames

I'm going to R/F and reopen if needed.

Comment 3

[Brandon Burton [:solarce]]

woooo

bburton@andesite ~$ curl -v "http://elasticsearch1.webapp.phx1.mozilla.com:9200/_cluster/health?pretty=true"
* About to connect() to elasticsearch1.webapp.phx1.mozilla.com port 9200 (#0)
*   Trying 10.8.81.65... connected
* Connected to elasticsearch1.webapp.phx1.mozilla.com (10.8.81.65) port 9200 (#0)
> GET /_cluster/health?pretty=true HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: elasticsearch1.webapp.phx1.mozilla.com:9200
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 271
< 
{
  "cluster_name" : "phxweb",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 50,
  "active_shards" : 100,
  "relocating_shards" : 1,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
* Connection #0 to host elasticsearch1.webapp.phx1.mozilla.com left intact
* Closing connection #0
}%