963245 - MDN: ElasticHttpError: Non-OK response returned (500):

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Luke Crouch [:groovecoder]]

ElasticHttpError: Non-OK response returned (500): u'SearchPhaseExecutionException[Failed to execute phase [query], total failure; shardFailures {[_na_][mdnprod-main_index][0]: No active shards}{[_na_][mdnprod-main_index][1]: No active shards}{[_na_][mdnprod-main_index][2]: No active shards}{[_na_][mdnprod-main_index][3]: No active shards}{[_na_][mdnprod-main_index][4]: No active shards}]'

https://errormill.mozilla.org/mdn/mdn/group/154108/

Comment 1

[Brandon Burton [:solarce]]

We apologize for the disruption in service. Nagios alerted the MOC about the cluster health and this was escalated to WebOps as your bug was being filed.

The ElasticSearch cluster that provides services to MDN encountered a cluster-wide issue with trying to perform garbage collection due to hitting it's memory ceiling in combination with a particular index for another service

I'm investigating what caused the issues with the particular index as well as what tuning and upgrades are available, as many OOM fixes are in 0.90.x and later.

https://developer.mozilla.org/en-US/search?q=string is now returning results as expected for me.

Please let me know if I can answer any questions.

Comment 2

[Luke Crouch [:groovecoder]]

This is back. :(

https://errormill.mozilla.org/mdn/mdn/group/146983/

Comment 3

[Brandon Burton [:solarce]]

(In reply to Luke Crouch [:groovecoder] from comment #2)
> This is back. :(
> 
> https://errormill.mozilla.org/mdn/mdn/group/146983/

Still working on fixing the other index, which is causing some cluster instability, will leave this open until fully resolved but the mdn_prod index is now green

Comment 4

[Brandon Burton [:solarce]]

Cluster status is now green.

Comment 5

[Adrian J Fernandez [:Aj]]

This occurred again @ 12:44 pm PST.

Solarce is currently investigating.

Comment 6

[Brandon Burton [:solarce]]

Today's issue has been resolved, the mdn index was only available for a few minutes

bug 963824 is tracking the plan for permanent resolution of the issue

Comment 7

[Luke Crouch [:groovecoder]]

Getting these errors again.

https://rpm.newrelic.com/accounts/263620/applications/3172075/traced_errors/1484382839

Comment 8

[Chris Turra [:cturra]]

looks to be related to bug 993671. :cyliang comments on how the permanent fix is an es upgrade. i know there is a bug tracking that, but off the top of my head don't have context on it.

Comment 9

[Luke Crouch [:groovecoder]]

Again: https://rpm.newrelic.com/accounts/263620/applications/3172075/traced_errors/1492808873

Do we need an ES reboot?

Comment 10

[Luke Crouch [:groovecoder]]

Fixed by :cyliang.

Comment 11

[C. Liang [:cyliang]]

ES cossetted and rebooted.  The upgrade bug in question is 963824.  Discussions (outside of Bugzilla) about ameliorating this issue in other ways have started.

Comment 12

[Luke Crouch [:groovecoder]]

Again:

https://rpm.newrelic.com/accounts/263620/applications/3172075/traced_errors/1496813634

Comment 13

[Brandon Burton [:solarce]]

Continued issues with 0.20.x GC pauses are affecting this uptime, due to the shared nature and mixed use of the SCL3 cluster, moving forward with giving MDN its own cluster for prod, this is being tracked in bug 995457

Comment 14

[C. Liang [:cyliang]]

This happened again today.  (For future reference, bugs 1000488, 1000489, 1000490, and 1000493 have the Nagios alert texts.)

Comment 15

[Luke Crouch [:groovecoder]]

And again today. Can we get another restart? https://rpm.newrelic.com/accounts/263620/applications/3172075/traced_errors/1567702582

Comment 16

[Brandon Burton [:solarce]]

Passing to :cyliang as she's been working on the final resolution