569113 - moz2-linux64-slave12 needs scratchbox uninstalled

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Phil Ringnalda (:philor)]

Since the successful http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1274992026.1274995958.25183.gz at 13:27 Thursday, every TraceMonkey Linux64 Bo run has been moz2-linux64-slave12 trying to free 7.00GB of space, only freeing 6.66GB (an inauspicious number), and burning.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Rail rightly points out that bug 566026 will help a lot here (assuming bug 568913 didn't steal all the space it needs). Of the 15G /builds partition in use now, df reports the breakdown in MB as
     1	logs
     1	lost+found
  7129	scratchbox
  3672	slave
 10800	total

Almost all of slave/ is the failing tracemonkey linux64 opt build. Can see why scratchbox isn't completely unnecessary on linux64, breaking that down there is
   176  ccache
  1137  compilers
   811  devkits
    44  ghost_shared
   172  tools
  4781  users

Most of that space in users/ is a bind mount from /builds/scratchbox/users/cltbld/builds/slave back to /builds/slave, so that we're double counting disk usage. 

Not all linux64 slaves seem to have scratchbox installed. :-S

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

moz2-linux64-slave12 is currently out of production, hence the green build on tracemonkey just now.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Ok, so only moz2-linux64-slave12 has scratchbox installed. I'm going to throw this over to jhford for the uninstall, since he knows all about this stuff.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

I have cleaned up this slave and there is now lots of free space

[cltbld@moz2-linux64-slave12 slave]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             8.0G  5.2G  2.4G  69% /
/dev/sdb1              15G  166M   14G   2% /builds
tmpfs                1006M     0 1006M   0% /dev/shm
/dev/sda2             798M  301M  457M  40% /var
10.2.71.136:/export/buildlogs/puppet-files
                      1.0T  563G  462G  55% /N

For future reference, to modify scratchbox in any way, you need to run:
 /scratchbox/sbin/sbox_ctl stop
 /scratchbox/sbin/sbox_umount_all
 umount /builds/slave
These three steps unmount all the bound directories.  Critical host system directories are mounted into the scratchbox, so a straight rm -rf /builds/scratchbox would break the host system.


And to remove scratchbox,
 rm -rf /scratchbox
 rm -rf /builds/scratchbox