774513 - puppetagain masters are throwing disk space warnings

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jake Watkins [:dividehex]]

puppetagain masters are throwing disk space warnings.  I'm fairly sure this is from the Centos repo that got setup today.

See https://bugzilla.mozilla.org/show_bug.cgi?id=773379#c1

[16:08]  <nagios-releng-scl1> [77] releng-puppet1.build.mtv1:disk - / is WARNING: DISK WARNING - free space: / 12625 MB (8% inode=98%):
[16:09]  <nagios-releng-scl1> [78] releng-puppet1.srv.releng.scl3:disk - / is WARNING: DISK WARNING - free space: / 10676 MB (7% inode=98%):
[16:09]  <nagios-releng-scl1> [79] releng-puppet1.build.scl1:disk - / is WARNING: DISK WARNING - free space: / 10217 MB (7% inode=97%):

Comment 1

[Jake Watkins [:dividehex]]

also see https://bugzilla.mozilla.org/show_bug.cgi?id=764933#c3

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

images/ is pretty small; this was due to the 6.6GB of epel6, plus puppetagain, that I mirrored today.

I think we should just give these hosts more disk.  While we can delete the 6.2 repos when CentOS 6.3 is active, overall space usage is going to keep growing.  Three are on KVM, which I can expand.  The last is on ESX, so I'll coordinate with virtualization to get more space there.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

relabs-puppet.build.mtv1 - has 200G
releng-puppet1.build.scl1 - has 200G
releng-puppet1.build.mtv1 - will shortly have 200G (in process, 50m remaining)
releng-puppet1.srv.releng.scl3 - bug 774665