919351 - node40.bunker.scl3 out of disk space

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, defect)

Description

[Adrian J Fernandez [:Aj]]

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              66G   63G   16K 100% /

61G     /var/lib/elasticsearch

technically it's not 100% truly full (if you account for reserved space).

Perhaps it will elasticsearch will automatically clear some space (re-allocate the shards) but filed for reference.

Comment 1

[Adrian J Fernandez [:Aj]]

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              66G   55G  7.8G  88% /

Looks like it kicked in.

Comment 2

[Adrian J Fernandez [:Aj]]

Re-occurred again and will probably (shorty) recover. However, lets try not to make this a re-occurring thing.

Comment 3

[Adrian J Fernandez [:Aj]]

Recovered, leaving open as this shouldn't page every day.

Comment 4

[Adrian J Fernandez [:Aj]]

So it's an everyday thing around this time.

Comment 5

[Eric Ziegenhorn :ericz]

Happened again:

[19:19] < nagios-scl3> | Sat 17:19:26 PDT [5283] node40.bunker.scl3.mozilla.com:Disk - All is CRITICAL: DISK CRITICAL - free space: / 6 MB (0% inode=96%): (http://m.allizom.org/Disk+-+All)

It's almost all in /var/lib/elasticsearch/bunker/nodes/0/indices which is 61G.

Comment 6

[Chris Turra [:cturra]]

bunker is no longer an active project on our team. marking this bug as r/wontfix.

Comment 7

[Eric Ziegenhorn :ericz]

:cturra, then should we remove these from Nagios?  It's still a burden on the MOC.  Should these nodes be decommissioned?

Comment 8

[Chris Turra [:cturra]]

:ericz - probably a question for :phrawzty. i don't have a state on if this project is being refreshed on the same hardware or not.

Comment 9

[Daniel Maher [:phrawzty]]

Bunker is alive and well, though it is no longer managed by webops; it is now acting as a back-end for DXR.

In other words, it should stay in Nagios. :)