1346295 - Kibana 4.1 stack runs into memory issues under small queries

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Ben Bangert [:benbangert]]

The new Kibana 4.1 here:
https://autopush-kibana.prod.mozaws.net/_plugin/kibana/

Throws errors doing fairly basic queries, none of which ever caused issues on the old Kibana/Logging stack.

I have a dashboard there which illustrates the issue, "Autopush dashboard prototype".

Errors such as:
Visualize: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"ElasticsearchException[CircuitBreakingException[[fielddata] Data too large, data for [Fields.router_key] would be larger than limit of [2405459558/2.2gb... more


I had substantially more complex dashboards and displays on the old kibana/logging stack that worked without issue.

Comment 1

[Daniel Thorn [:relud]]

I increased the size of the elasticsearch cluster in use for autopush in prod, and that dashboard is working now. memory has increased by 4x per node (to 15 GiB), and I doubled the number of nodes from 2 to 4. I suspect you may still need more resources in your elasticsearch cluster, but I think this is a good start.

Comment 2

[Daniel Thorn [:relud]]

resolving this as fixed, please reopen if you see that it's still an issue.

Comment 3

[Ben Bangert [:benbangert]]

I'm still having issues when trying to get more than a few days of history on the main dashboard I made.

Comment 4

[Jeremy Orem [:oremj]]

I've bumped the instance sizes again and am now able to go out 7 days on that dashboard.