1008147 - (b-linux64-ix-0010) b-linux64-ix-0010 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Carsten Book [:Tomcat]]

seems this slave is not happy ->

https://tbpl.mozilla.org/php/getParsedLog.php?id=39349537&tree=Mozilla-Inbound

abort: No space left on device

disabled this slave in slave alloc

Comment 1

[Pete Moore [:pmoore][:pete]]

Taking a look...

Comment 2

[Pete Moore [:pmoore][:pete]]

Whoops.

Carelessly, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1001518#c10 I ran:

[[e-mail] ~]$ python tools/buildfarm/maintenance/purge_builds.py -s 20
Deleting ./.repoconfig
Deleting ./.pki
Deleting ./tmp
Deleting ./nthomas
Deleting ./.pip
Deleting ./.mozilla
Deleting ./.ssh
Error: unable to free 20.00 GB of space. Free space only 0.00 GB
[[e-mail] ~]$

I think I should have cd'd into /builds/slaves first. Whoops. I should have checked before running.

I'll get this slave reimaged, as I've now destroyed it! :(

Comment 3

[Pete Moore [:pmoore][:pete]]

So ......... I've recovered directories:

  * /home/cltbld/.repoconfig
  * /home/cltbld/.pki
  * /home/cltbld/.pip
  * /home/cltbld/.ssh

from another slave (bld-linux64-ix-034) so I think a reimage is not necessary. From inspecting the files in these directories, they look generic so I believe they should be the same on bld-linux64-ix-034 and bld-linux64-ix-036, which I why I opted for a straight copy (via scp).

The /home/cltbld/.mozilla directory was a crash reports directory, so did not need to be copied.

The /home/cltbld/nthomas directory is lost, sorry Nick.

Also, /home/cltbld/tmp presumably was created by a release engineer, and has been lost too.

I tried to free up 20GB space running:

> python tools/buildfarm/maintenance/purge_builds.py -s 20 /builds/slave

It reported:

> "57.27 GB of space available".

Since this was not quite what I requested (20GB vs. 57GB), I checked to see if this was across multiple mounts:

> [[e-mail] ~]$ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda3             227G  158G   58G  74% /
> tmpfs                 3.9G     0  3.9G   0% /dev/shm
> /dev/sda1              97M   52M   41M  56% /boot

It might just be that the last directory to be freed was around 40GB(?) and so when it deleted it, the free space suddenly shot up?!

The last-to-be-deleted directory was:
> /builds/slave/rel-m-rel-lx_uv_4-000000000000

In any case, the machine now has available disk space again.

I will check buildbot is running ok, and reenable in slavealloc...

Pete

Comment 4

[Chris Cooper [:coop] (he/him)]

Please do not re-enable this slave. We are retiring linux hardware build slaves in bug 1106922.