442843 - qm-moz2-unittest01 is out of disk space

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[:Gavin Sharp [email: [e-mail]]]

[30 21:26:07] <nagios> qm-moz2-unittest01:disk - /builds is CRITICAL: DISK CRITICAL - free space: /builds 0 MB (0% inode=98%):

Looks like this is causing the three unit test boxes on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla2 to go red, so filing as a blocker - not sure whether the IT support docs cover this scenario, but no unit test machine feedback is keeping the tree closed.

Comment 1

[Aravind Gottipati [:aravind]]

Yup, no docs for this case, over to build and release eng.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Moved 1.6Gb out of /builds/moz2_unittest/win2k3_03 to ~/win2k3_03 as a stop-gap fix. Current status:
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/sda1             8.3G  6.7G  1.2G  85% /
  tmpfs                 379M     0  379M   0% /dev/shm
  /dev/sda2             996M  120M  825M  13% /var
/  dev/sdb1              20G   18G  1.6G  92% /builds

Long term, this box needs something like this crontab job on qm-rhel02:
  # Delete slave logs after 5 days
  32 3 * * * find /build/master/trunk* -mtime +5 -exec rm {} \;

Builds are cycling now --> FIXED.

Comment 3

[Rob Campbell [:rc] (:robcee)]

because of the different directory names, I had to add 3 entries:

# Delete slave logs after 5 days
32 3 * * * find /build/master/linux* -mtime +5 -exec rm {} \;
32 4 * * * find /build/master/darwin* -mtime +5 -exec rm {} \;
32 5 * * * find /build/master/win* -mtime +5 -exec rm {} \;

Comment 4

[Rob Campbell [:rc] (:robcee)]

erm, corrected with proper directory names (/builds/moz2_unittest)