1170204 - raise java heap on ES infra

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Robert Helmer [:rhelmer]]

The AWS ES infra isn't setting ES_HEAP_SIZE, so we're getting the default (1GB max heap)

I'm assuming this isn't intended, as the data nodes have ~60GB each, just sitting there :)

This came to my attention because we load tested over the weekend and ES data nodes are regularly falling over with java OOMs.

Comment 1

[Robert Helmer [:rhelmer]]

Total RAM by ES instance type:

* data nodes: 60GB
* interface nodes: 30GB
* master nodes: 32GB

It looks like all of them are getting the default right now, which is -Xms256m -Xmx1g

ES docs (https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html) say to just use ES_HEAP_SIZE, which sets min/max heap size to the same value.

Comment 2

[Robert Helmer [:rhelmer]]

PR: https://github.com/mozilla/socorro-infra/pull/148

Comment 3

[Daniel Maher [:phrawzty]]

(In reply to Robert Helmer [:rhelmer] from comment #1)
> Total RAM by ES instance type:
> 
> * interface nodes: 30GB

If the RAM quantities that's accurate, then this doesn't line up:

<% elsif @es_interface %>
ES_HEAP_SIZE=40g
<% end %>

Heap for the interface type is set to "40g", but total RAM on the system is apparently only 32GB...

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Daniel Maher [:phrawzty] from comment #3)
> (In reply to Robert Helmer [:rhelmer] from comment #1)
> > Total RAM by ES instance type:
> > 
> > * interface nodes: 30GB
> 
> If the RAM quantities that's accurate, then this doesn't line up:
> 
> <% elsif @es_interface %>
> ES_HEAP_SIZE=40g
> <% end %>
> 
> Heap for the interface type is set to "40g", but total RAM on the system is
> apparently only 32GB...

Good catch.. this change does not seem to have taken effect at all on the interface and master nodes in fact, although it did work on the data nodes.

Comment 5

[Robert Helmer [:rhelmer]]

Also, data nodes appear to have 36GB not 30GB as previously commented.

Comment 6

[Daniel Maher [:phrawzty]]

x-post from https://github.com/mozilla/socorro-infra/pull/167



Within the context of ES, heap sizes are governed by two factors: Lucene and Java.

Lucene operates out of the heap (it's functionally filesystem operations), and therefore has separate memory requirements. As a general rule, one should never allocate more than 50% of the total system RAM to the ES Java heap - the other half is left to the kernel to be used for the system, which includes Lucene.

Java catches a bad case of the dumb when it has more than 32GB of memory to deal with. Allocating more than that to the heap becomes a question of diminishing returns, and can even have detrimental effects for smaller allocation sizes; unless one is allocating hundreds of gigs it's not even worth dealing with the issue.

In summary, these numbers should not exceed 32GB, and must not exceed 50% of total system.