1431886 - signing-linux-12 nearly ran out of space

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

Sat 00:32:59 UTC [7632] [] signing-linux-12.srv.releng.usw2.mozilla.com:disk - / is CRITICAL: DISK CRITICAL - free space: / 2370 MB (4% inode=97%): (http://m.mozilla.org/disk+-+/)Sat 01:46:58 UTC [7648] [] signing-linux-

12.srv.releng.usw2.mozilla.com:disk - / is OK: DISK OK - free space: / 6546 MB (13% inode=97%): (http://m.mozilla.org/disk+-+/)

we should investigate why and whether to bump the instance type or not.

May need to loop in someone with access to the machine: https://dxr.mozilla.org/build-central/source/puppet/manifests/moco-config.pp#202

Comment 1

[Aki Sasaki (not active)]

<aki> i cleaned it up. i think something around log rotation and not restarting. but also retriggers instead of reruns - fuzzy matching in cotv1 is very verbose.

This will be better with cotv2. We can also address https://github.com/mozilla-releng/signingscript/issues/45 .

Comment 2

[Aki Sasaki (not active)]

Also cleaned up the other 11.
I filed https://github.com/mozilla-releng/scriptworker/issues/180 . I believe Aryx talked to the sheriffs about not retriggering nightly jobs, but rerunning them instead.

Comment 3

[Aki Sasaki (not active)]

Steps:

- clobbered /builds/scriptworker/logs/worker.log.* (backups; we had a spike of fuzzy retriggers earlier in the week that made one of these logs exponentially larger)
- on some of these I also clobbered worker.log itself. That requires a scriptworker restart bc the filehandle is still open
- clobbered /var/log/messages , which was exponentially larger. Requires a syslog restart for the same reason
- rebooted to make a clean restart of the whole thing.