914192 - brasstacks cannot reach ES cluster

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Mark Côté [:mcote]]

If I try to netcat to elasticsearch-zlb.webapp.scl3.mozilla.com:9200 from brasstacks1.dmz.scl3.mozilla.com, I don't get an Elastic Search response.  Not being able to reach the ES cluster breaks OrangeFactor.

Comment 1

[Brandon Burton [:solarce]]

This was impacted by some load balancing issues earlier this morning, work to resolve this is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=914154

I've confirmed that the LB VIP for ES is functional now

[[e-mail]3 ~]# host elasticsearch-zlb.webapp.scl3.mozilla.com
elasticsearch-zlb.webapp.scl3.mozilla.com has address 10.22.81.174

[[e-mail]3 ~]# nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
Connection to elasticsearch-zlb.webapp.scl3.mozilla.com 9200 port [tcp/wap-wsp] succeeded!


[[e-mail]3 ~]# curl elasticsearch-zlb.webapp.scl3.mozilla.com:9200
{
  "ok" : true,
  "status" : 200,
  "name" : "elasticsearch4_scl3",
  "version" : {
    "number" : "0.20.5",
    "snapshot_build" : false
  },
  "tagline" : "You Know, for Search"

Comment 2

[Mark Côté [:mcote]]

I don't have access to that bug.  I do see that the ES cluster is in a "red" state, though, and OF is still not working.

Comment 3

[Brandon Burton [:solarce]]

(In reply to Mark Côté ( :mcote ) from comment #2)
> I don't have access to that bug.  I do see that the ES cluster is in a "red"
> state, though, and OF is still not working.

Yes, discovered two of the five servers in the cluster weren't happy because of the network partitions.

We restarted their ES services and the cluster is green now, both the orangefactor_logs and tbpl indexes appear to be happy.