/var filled up on scl-production-puppet.build.scl1.mozilla.com

RESOLVED DUPLICATE of bug 624213

Status

Release Engineering
General
RESOLVED DUPLICATE of bug 624213
7 years ago
5 years ago

People

(Reporter: catlee, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
The /var partition (1GB big!!!) filled up on scl-production-puppet.build.scl1.mozilla.com, most of which was under /var/lib/puppet/reports
(Reporter)

Comment 1

7 years ago
(In reply to comment #0)
> The /var partition (1GB big!!!) filled up on
> scl-production-puppet.build.scl1.mozilla.com, most of which was under
> /var/lib/puppet/reports

Clobbered /var/lib/puppet/reports, restarted puppetmaster and syslog
(Reporter)

Comment 2

7 years ago
(In reply to comment #1)
> (In reply to comment #0)
> > The /var partition (1GB big!!!) filled up on
> > scl-production-puppet.build.scl1.mozilla.com, most of which was under
> > /var/lib/puppet/reports
> 
> Clobbered /var/lib/puppet/reports, restarted puppetmaster and syslog

Looks like this was responsible for keeping a large number of linux and mac slaves from running buildbot.

First nagios warning of this was today at 11:51am PST.

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 624213
(In reply to comment #1)
> (In reply to comment #0)
> > The /var partition (1GB big!!!) filled up on
> > scl-production-puppet.build.scl1.mozilla.com, most of which was under
> > /var/lib/puppet/reports
> 
> Clobbered /var/lib/puppet/reports, restarted puppetmaster and syslog

What was the nagios alert that made you look in that master?
What were the commands you run to get this back into shape?
I wanna know in case I hit it while on duty.
There's a nagios alert specifically about /var on scl-production-puppet - that's what made me see this error the first time it happened.

My solution was to rm -rf /var/lib/puppet/reports/* (and email release@ regarding why reports are turned on there)
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.