834849 - carbon-cache's console.log needs to be rotated regularly

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Ziegenhorn :ericz]

Graphite rotates all of it's log except console.log.  This file gets written to frequently and can get quite large.  Right now it's over 3GB on two of our graphite servers.  We should rotate this file regularly.

Comment 1

[Rick Bryce [:rbryce]]

Attachment: logroated config

Comment 2

[Rick Bryce [:rbryce]]

(In reply to Rick Bryce [:rbryce] from comment #1)
> Created attachment 707180 [details]
> logroated config

ericz - you probably know better how long we need to keep these log file, if it at all.  The attached config should suffice for now.

Comment 3

[Eric Ziegenhorn :ericz]

We probably don't need to keep it very long.  I see now that other carbon logs are being rotated but not cleaned up so we can probably apply a similar logrotate config to them.

Comment 4

[Eric Ziegenhorn :ericz]

console.log is all setup now.  I capped it at a smaller size because logrotate has to restart carbon-cache after rotating the log and while carbon-cache writes out any metrics in it's cache during the restart it is unavailable to accept new metrics, thus we want to minimize that time.

As previously noted the other carbon logs are rotated by carbon but never cleaned up.  I'll make a cron job to clean up those.

Comment 5

[Eric Ziegenhorn :ericz]

Added clean-old-graphite-logs root cron job to delete logs older than 15 days.