1861260 - Update the Push extension to add additional status check code in order to restart the pod if an error is occurring

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, enhancement)

Description

[David Lawrence [:dkl]]

When issues come up like the pod loses its connection tot he MySQL database (happened recently) we need to have a way for K8S to do a liveness check to restart the pod. We use Daemon::Generic which has a gd_check() method that can be defined that allows the daemon to do status checks from the commend line. Such as extensions/Push/bin/bugzilla-pushd.pl check which can be used by k8s. Currently check just returns if a PID is assigned. This would not have caught the MySQL issue encountered. By defining a gd_check() and then trying to connect to the database and get a count of queue jobs, we can get an better idea of the status of the pod. We currently do something similar for the JobQueue code that delivers emails so it would not be too much work to add it to the Push extensions as well.

https://github.com/mozilla-bteam/bmo/blob/master/Bugzilla/JobQueue/Runner.pm#L177-L188

Comment 1

[BMO Github Automation]

Attachment: [mozilla-bteam/bmo] Bug 1861260 - Update the Push extension to add additional status check code in order to restart the pod if an error is occurring (#2135)

Comment 2

[David Lawrence [:dkl]]

Merged
https://github.com/mozilla-bteam/bmo/commit/ce7841c990a49e97c4ce057bb2f4095c82dc2208