1990765 - clean up kafka logs on hgweb nodes

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Julien Cristau [:jcristau]]

Bug 1896189 added a daily timer to run find /var/log/zookeeper /var/log/kafka -mtime +7 -delete.
That appears to run and work on the kafka nodes, but for some reason not on hgweb*, where we still have log files going back to 2018.

Comment 1

[Julien Cristau [:jcristau]]

OK I think I see what happened. The new timer was enabled but not started, and we haven't rebooted the hgweb nodes (which would have started it) since then. I've manually started it now so the command should run at 00:00Z from now on.

Comment 2

[Julien Cristau [:jcristau]]

Also did a one-off systemctl start kafka-logs-cleanup.service on hgweb4 to clear the nagios alert.

Comment 3

[Julien Cristau [:jcristau]]

Attachment: kafka-broker: make sure the log cleanup timer is started (bug 1990765)

Avoid requiring a reboot before we start cleaning up logs.

Comment 4

[Julien Cristau [:jcristau]]

Reopening to track the ansible fix.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/1b3aee022f08
kafka-broker: make sure the log cleanup timer is started r=sheehan