1405104 - disk space low on prod ES node root volume

Status: RESOLVED FIXED

(Socorro :: Infra, task, P2)

Description

[Daniel Thorn [:relud]]

host i-450d96b2 has low space on /dev/xvda1 and is occasionally less than 1.5GiB remaining.

Comment 1

[Daniel Thorn [:relud]]

I modified the check to temporarily alert at 1.4GiB, so that it won't wake me up tonight, because it definitely looks like a very slow moving issue.

Comment 2

[Daniel Thorn [:relud]]

and it still warns at 1.5GiB

Comment 3

[Miles Crabill [:miles]]

Manually cleaned up ES logs, which are not rotated. Will write an ansible playbook that adds logrotate to the ES nodes, test it in stage first, and then run it in prod.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

As I understand it, this is important, but not the end of the world. Given that, I'm making this a P2.

Comment 5

[Miles Crabill [:miles]]

This disk space warning is resolved for this host, so I'm resolving this bug. I opened bug 1405586 for the issue of cleaning up logs in general.