778208 - auto compress/roll logs on socorro1.dev

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[Robert Helmer [:rhelmer]]

We have a dev server (socorro1.dev.dmz.phx1.mozilla.com aka "crash-stats-dev") which logs a significant amount of stuff to /var/log/socorro (and /var/log/socorro/kohana)

We don't have any log rolling/compression enabled on there, I know we do on stage/prod so it'd be nice to have the same thing here.

Comment 1

[Dumitru Gherman [:dumitru]]

That's only taking up 5 GB, and most of them are already compressed.
The main problem with disk space on this host is  /var/lib/hbase/crash_reports ... you know the story, Rob ;)

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Dumitru Gherman [:dumitru] from comment #1)
> That's only taking up 5 GB, and most of them are already compressed.
> The main problem with disk space on this host is 
> /var/lib/hbase/crash_reports ... you know the story, Rob ;)

Yep that is bug 778207 ;)

Comment 3

[Ashish Vijayaram [:ashish]]

So is this bug still valid? :)

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Ashish Vijayaram [:ashish] from comment #3)
> So is this bug still valid? :)

Yes, the hbase issue is separate - we still want to roll the logs on here I'd think.

Comment 5

[Eric Ziegenhorn :ericz]

These logs are actually already being rotated.  It seems like it is using a lot of disk still because they are just huge.  Particularly /var/log/socorro/syslog which is currently ~700MB before being compressed.  In the five days of the current syslog file there are over 5 million lines.  Not sure why socorro is so verbose but there is nothing more I can do here.