1319887 - Elastic Search under reporting results

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Lonnen :lonnen]

The healthcheck is failing and the front page is showing:

"Warning: Our database is experiencing troubles, the data you see might be wrong. The team has been notified of the issue.

The data for the week of Mon Nov 14 2016 is ~20% lower than expected."

We are investigating here: https://docs.google.com/document/d/14Ffc7_dJM_UAVAftcP_i6dqx7nU7-wmGMzWO8iCmuos/edit?ts=5835d9a8

Comment 1

[Lonnen :lonnen]

We're seeing OOM errors, and some full disks. The problems are directly related to FIELDDATA being too large, similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1288179.

We immediately resolved this and removed some pressure by closing week 21, 22, 23 which makes them unavailable for search and limits the size of fielddata.

I'll hang some follow on remediation tasks here that we identified in the process.


historic note: during the investigation we found a bunch of old indices that should have been reaped with very little data. They are not problematic. I suspect this is the 'submit old crashes' feature at work.

Comment 2

[Lonnen :lonnen]

We considered a few approaches:

* remove data from ES by dropping old indices
* close old indices
* adding more nodes

Jason didn't think adding more nodes would alleviate our specific problem. We chose to close old indices to reduce memory pressure on queries but retain the data on disk. We can easily reopen them after we've implemented other remediations or in an emergency and have the data back without a full reprocess.