mdn can't connect to elasticsearch-zlb.webapp.scl3.mozilla.com

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
P1
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: groovecoder, Assigned: solarce)

Tracking

Details

(Reporter)

Description

5 years ago
Started yesterday around noon?

https://errormill.mozilla.org/mdn/mdn/group/91489/
(Reporter)

Comment 1

5 years ago
priority bump - ongoing production stacktraces
Severity: normal → major
(Assignee)

Comment 2

5 years ago
We've been seeing some networking issues in SCL3 since then, including one a few minutes ago, which netops is working on

As of right now the ES's ZLB is reachable from all the prod mdn hosts

[root@developeradm.private.scl3 ~]# issue-multi-command developer nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
[2013-09-09 09:35:56] [developer1.webapp.scl3.mozilla.com] running: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
[2013-09-09 09:35:56] [developer2.webapp.scl3.mozilla.com] running: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
[2013-09-09 09:35:56] [developer3.webapp.scl3.mozilla.com] running: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200
[2013-09-09 09:35:56] [developer1.webapp.scl3.mozilla.com] finished: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200 (0.227s)
[developer1.webapp.scl3.mozilla.com] out: Connection to elasticsearch-zlb.webapp.scl3.mozilla.com 9200 port [tcp/wap-wsp] succeeded!
[2013-09-09 09:35:56] [developer2.webapp.scl3.mozilla.com] finished: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200 (0.308s)
[developer2.webapp.scl3.mozilla.com] out: Connection to elasticsearch-zlb.webapp.scl3.mozilla.com 9200 port [tcp/wap-wsp] succeeded!
[2013-09-09 09:35:56] [developer3.webapp.scl3.mozilla.com] finished: nc -z elasticsearch-zlb.webapp.scl3.mozilla.com 9200 (0.308s)
[developer3.webapp.scl3.mozilla.com] out: Connection to elasticsearch-zlb.webapp.scl3.mozilla.com 9200 port [tcp/wap-wsp] succeeded!
Assignee: server-ops-webops → bburton
Severity: major → normal
Status: NEW → ASSIGNED
Priority: -- → P1
(Assignee)

Comment 3

5 years ago
The cluster was also unhappy due to the networking partitions earlier. The cluster is now green again and all indexes are happy.
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.