1363527 - cruncher keeps running out of disk space

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Amy Rich [:arr] [:arich]]

cruncher-aws keeps sending puppet reports claiming that puppet can't complete because it's run out of disk space. e.g. 

Tue May 09 13:42:09 -0700 2017 Puppet (err): Failed to apply catalog: No space left on device - /var/lib/puppet/state/state.yaml20170509-26620-gm5vq5

Comment 1

[Amy Rich [:arr] [:arich]]

it looks like /tmp has a lot of cruft in it, but I'm not sure what's safe to delete.

Comment 2

[Amy Rich [:arr] [:arich]]

For starters, I removed everything from 2016. Whatever is leaving these temp files around should be modified to include a cleanup step.

Comment 3

[Alin Selagea [:aselagea]]

Previous disk space:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/cloud_root-lv_root
                       35G   18G   16G  53% /
none                  3.6G     0  3.6G   0% /dev/shm
/dev/xvda1             60M   28M   29M  50% /boot

Did some investigation and noticed three folders that are responsible for most of the used disk space:
1. /home --> 5.6G
2. /tmp --> 5.5G
3. /var --> 5.1G

    - /var/www/html/builds stores data for allthethings, slave_health and reportor (~5GB), but these are cleaned on a regular basis
    - /home/jlorenzo/l10n folder has 3.1GB and is dated back Feb 27.
    - /tmp/ contains copies of "buildbot-configs" and seem to be created on irregular basis. I could not find where they're generated, but I removed everything older than April 1.

Current disk space:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/cloud_root-lv_root
                       35G   12G   21G  37% /
none                  3.6G     0  3.6G   0% /dev/shm
/dev/xvda1             60M   28M   29M  50% /boot

Comment 4

[Alin Selagea [:aselagea]]

I think we're fine now and I don't expect future disk space issues in the near future on this machine. 

@Johan: regardless of that, could you please take a look at your home folder and do some cleaning if that stuff is no longer needed?
Thanks!

Comment 5

[Johan Lorenzo [:jlorenzo] - On PTO - Back on July 28th]

Thank you for spotting that. I deleted the 3.1GB of data I had in my home.

For reference, this data was generated by [1], in order to merge the l10n repos from beta to release. We don't need these files anymore. l10n repos will be re-downloaded during next merge. 

[1] https://hg.mozilla.org/build/braindump/raw-file/default/releases-related/beta2release_l10n.sh

Comment 6

[Alin Selagea [:aselagea]]

Thanks, Johan!

Let's mark this as resolved.