1211447 - Rotate buildbot bridge logs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

They are huge!

Comment 1

[Rail Aliiev [:rail]]

Attachment: bbb_logrotate.diff

I thought about using Python's log rotation, but this approach sounds more portable in case someone decides to log to stderr or using some other method.

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 8669663 [details] [diff] [review]
bbb_logrotate.diff

Review of attachment 8669663 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/buildbot_bridge/templates/buildbotbridge.logrotate.erb
@@ +10,5 @@
> +    rotate 10
> +    compress
> +    sharedscripts
> +    postrotate
> +        /usr/bin/supervisorctl restart buildbot_bridge_bblistener buildbot_bridge_tclistener buildbot_bridge_reflector

as long as we won't lose any messages/data by restarting, then I'm ok with this.

Comment 3

[Rail Aliiev [:rail]]

(In reply to Justin Wood (:Callek) from comment #2)
> Comment on attachment 8669663 [details] [diff] [review]
> bbb_logrotate.diff
> 
> Review of attachment 8669663 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/buildbot_bridge/templates/buildbotbridge.logrotate.erb
> @@ +10,5 @@
> > +    rotate 10
> > +    compress
> > +    sharedscripts
> > +    postrotate
> > +        /usr/bin/supervisorctl restart buildbot_bridge_bblistener buildbot_bridge_tclistener buildbot_bridge_reflector
> 
> as long as we won't lose any messages/data by restarting, then I'm ok with
> this.

I assume it's safe, because we restart the services when upgrade bbb: http://hg.mozilla.org/build/puppet/file/816e84f77e0f/modules/buildbot_bridge/manifests/services.pp#l31

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8669663 [details] [diff] [review]
bbb_logrotate.diff

remote:   https://hg.mozilla.org/build/puppet/rev/1e15a231ddcb
remote:   https://hg.mozilla.org/build/puppet/rev/79683f435fe4

Comment 5

[Rail Aliiev [:rail]]

Ran `/usr/sbin/logrotate -vf /etc/logrotate.conf` to verify. Logs are moved, compressed, the processes restarted and continue logging properly.

# ls -lh /builds/bbb/*log*
-rw-r--r-- 1 cltbld cltbld 9.0K Oct  5 10:50 /builds/bbb/bblistener.log
-rw-r--r-- 1 cltbld cltbld  70M Oct  5 10:48 /builds/bbb/bblistener.log-20151005.gz
-rw-r--r-- 1 cltbld cltbld  125 Oct  5 10:48 /builds/bbb/reflector.log
-rw-r--r-- 1 cltbld cltbld 5.1M Oct  5 10:49 /builds/bbb/reflector.log-20151005.gz
-rw-r--r-- 1 cltbld cltbld  139 Oct  5 10:48 /builds/bbb/tclistener.log
-rw-r--r-- 1 cltbld cltbld  20K Oct  5 10:49 /builds/bbb/tclistener.log-20151005.gz