624213 - scl-production-puppet /var filling up

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

There are three large dirs on this server:
 - httpd logs
 - clientbucket (which seems odd since puppetd isn't running..)
 - /var/lib/puppet/reports

This is only a 996M partition, so something needs to go.  I blew away the clientbuckets, but what are those reports?  Do we need to keep those around?

Copying armen since I know he was working on the dashboard - are the reports related?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

copying Zandr since he was noticed these alerts in #build, too.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Yes, the reports are part of the prerequisites to the dashboard, so the theory goes that these were enabled as part of getting the dashboard set up on the scl1 production puppet master.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Turns out there were a couple of boxes (talos-r3-fed-044, talos-r3-fed64-004) which had rpm problems and were requesting updates every minute, which bloated the http logs. Managed to fix the fed64 box but the fed will need a reimage.

I used bzip on /var/log/http/*.? to free up space. We could configure logrotate to do this too. /var/lib/puppet/reports/talos-r3-fed-044.build.mozilla.org is 126M, we should probably put /var/lib/puppet/reports on /builds or something.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I think the better choice is to get rid of reports entirely, since they're not being used.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Bumping priority, since this is killing buildslaves.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m625213-puppet-manifests-r1.patch

remove --report from the run of puppet on fedora talos systems, which are the systems filling up the reports on this master.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I blew away the reports again, moments ago, so we've got some time on this one.

Comment 9

[Armen [:armenzg]]

Comment on attachment 504823 [details] [diff] [review]
m625213-puppet-manifests-r1.patch

Looks good :)

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 504823 [details] [diff] [review]
m625213-puppet-manifests-r1.patch

7612a30a9e1b

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Landed on all of the masters.