brasstacks cannot reach ES cluster

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: mcote, Assigned: bburton)

Tracking

Details

(Reporter)

Description

5 years ago
If I try to netcat to elasticsearch-zlb.webapp.scl3.mozilla.com:9200 from brasstacks1.dmz.scl3.mozilla.com, I don't get an Elastic Search response.  Not being able to reach the ES cluster breaks OrangeFactor.

Updated

5 years ago
Blocks: 914063
(Assignee)

Comment 1

5 years ago
This was impacted by some load balancing issues earlier this morning, work to resolve this is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=914154

I've confirmed that the LB VIP for ES is functional now

[root@brasstacks1.dmz.scl3 ~]# host elasticsearch-zlb.webapp.scl3.mozilla.com
elasticsearch-zlb.webapp.scl3.mozilla.com has address 10.22.81.174

[root@brasstacks1.dmz.scl3 ~]# nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
Connection to elasticsearch-zlb.webapp.scl3.mozilla.com 9200 port [tcp/wap-wsp] succeeded!


[root@brasstacks1.dmz.scl3 ~]# curl elasticsearch-zlb.webapp.scl3.mozilla.com:9200
{
  "ok" : true,
  "status" : 200,
  "name" : "elasticsearch4_scl3",
  "version" : {
    "number" : "0.20.5",
    "snapshot_build" : false
  },
  "tagline" : "You Know, for Search"
Assignee: server-ops-webops → bburton
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Priority: -- → P3
Resolution: --- → FIXED
(Reporter)

Comment 2

5 years ago
I don't have access to that bug.  I do see that the ES cluster is in a "red" state, though, and OF is still not working.
(Assignee)

Comment 3

5 years ago
(In reply to Mark Côté ( :mcote ) from comment #2)
> I don't have access to that bug.  I do see that the ES cluster is in a "red"
> state, though, and OF is still not working.

Yes, discovered two of the five servers in the cluster weren't happy because of the network partitions.

We restarted their ES services and the cluster is green now, both the orangefactor_logs and tbpl indexes appear to be happy.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.