1007273 - Loggins elasticsearch dying

Status: RESOLVED FIXED

(Webmaker Graveyard :: DevOps, defect)

Description

[JP Schneider :jp]

× Oops! ElasticsearchException[java.lang.OutOfMemoryError: Java heap space]

Got this getting one day worth of data.

Comment 1

[JP Schneider :jp]

Followed these recs on the three ES servers:

http://stackoverflow.com/questions/18132719/how-to-change-elasticsearch-max-size

Comment 3

[JP Schneider :jp]

Since we're sending literally 3 million log messages per day, this is not keeping up.
I've been keeping a rolling three days of logs to ensure this service stays up.  I've also sent my es configs to Laura Thompson, our MoCo friend with lots of ES experience, in the hopes she can provide me some knowledge.

Additionally, on June 6th I'll be attending a class on the ELK stack which will cover scaling best practices.

Comment 4

[JP Schneider :jp]

As I'm writing my own node.js and looking to use messina, I note this in the readme:


If you'd like to combine a single request/response cycle into a single log message, simply set combinedOutput: true:

var messina = require('messina');
var log = messina('myapp');

app.use(log.middleware({ combinedOutput: true }));


We should do that.

Comment 5

[JP Schneider :jp]

Booyah.

Resolved by applying actual knowledge instead of wildly guessing about settings. :)

Comment 6

[Chris DeCairos (:cade)]

woo! \o/