If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

carbon-cache's console.log needs to be rotated regularly

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: ericz, Assigned: ericz)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

5 years ago
Graphite rotates all of it's log except console.log.  This file gets written to frequently and can get quite large.  Right now it's over 3GB on two of our graphite servers.  We should rotate this file regularly.

Updated

5 years ago
Blocks: 833444

Comment 1

5 years ago
Created attachment 707180 [details]
logroated config

Comment 2

5 years ago
(In reply to Rick Bryce [:rbryce] from comment #1)
> Created attachment 707180 [details]
> logroated config

ericz - you probably know better how long we need to keep these log file, if it at all.  The attached config should suffice for now.
(Assignee)

Comment 3

5 years ago
We probably don't need to keep it very long.  I see now that other carbon logs are being rotated but not cleaned up so we can probably apply a similar logrotate config to them.
(Assignee)

Updated

5 years ago
Blocks: 833413
(Assignee)

Comment 4

5 years ago
console.log is all setup now.  I capped it at a smaller size because logrotate has to restart carbon-cache after rotating the log and while carbon-cache writes out any metrics in it's cache during the restart it is unavailable to accept new metrics, thus we want to minimize that time.

As previously noted the other carbon logs are rotated by carbon but never cleaned up.  I'll make a cron job to clean up those.
(Assignee)

Comment 5

5 years ago
Added clean-old-graphite-logs root cron job to delete logs older than 15 days.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.