912378 - Why did ES go down for 47 minutes?

Status: RESOLVED FIXED

(Marketplace Graveyard :: API, defect, P1)

Description

[Christopher Van Wiemeersch [:cvan]]

The Marketplace is blocked completely (i.e., stuck on the throbber) because of an API request. I can't diagnose what exactly the issue is but

https://marketplace.firefox.com/api/v1/apps/search/ is taking minutes to load - and in fact *never* finishes loading for me (not even a 502 Bad Gateway).

For measure https://marketplace-dev.allizom.org/ and https://marketplace-dev.allizom.org/api/v1/apps/search/ load just fine.

Comment 1

[Christopher Van Wiemeersch [:cvan]]

Issue has resolved itself. This is still very concerning, as the Marketplace was unresponsive, unable to load for over 30 minutes (since I noticed the issue) - could've been longer.

We should definitely figure out what happened and add nagios checks and ensure we have an action plan when ES goes down again in the future.

Comment 2

[Wil Clouser [:clouserw]]

47m comes from pingdom

Comment 3

[Jason Thomas [:jason]]

The logs show that all the nodes began aggressive GC during this time, and node es4.mktweb became unavailable and dropped out of the cluster. es4.mktweb remained unavailable until I restarted the service around 3:40 PDT.

I've added additional java7 related GC options to elasticsearch that should minimize the impact of GC running on these hosts.

Comment 4

[Wil Clouser [:clouserw]]

thanks