1338456 - [PulseGuardian] Does not clear overgrowing state resulting in warning spam

Status: RESOLVED INCOMPLETE

(Webtools :: Pulse, defect)

Description

[Bob Clary [:bc] (inactive)]

One of Autophone's servers (autophone-1) was off line for a bit this evening and it allowed the autophone1 queue/autophone1/builds queue to build up to a level of messages which resulted in a warning email.

This occurred at 2017-02-09 6:29 PM with the first message

Warning: your queue "queue/autophone1/builds" on exchange "could not be determined" is
overgrowing (4004 ready messages, 4004 total messages).

The queue will be automatically deleted when it exceeds 20000 messages.

Make sure your clients are running correctly and are cleaning up unused
durable queues.

I started seeing overgrowing / back to normal messages several times a minute even though I was not consuming the messages and it should have not been transitioning to normal then back to overgrowing.

...

Your queue "queue/autophone1/builds" on exchange "could not be determined" is
now back to normal (3614 ready messages, 3614 total messages).

...

Warning: your queue "queue/autophone1/builds" on exchange "could not be determined" is
overgrowing (4355 ready messages, 4355 total messages).

The queue will be automatically deleted when it exceeds 20000 messages.

Make sure your clients are running correctly and are cleaning up unused
durable queues.

...
 
at 6:59 I restarted the service and it consumed the messages

Your queue "queue/autophone1/builds" on exchange "could not be determined" is
now back to normal (0 ready messages, 0 total messages).

...

Then I continued to get the normal / overgrowing messages 2-3 times per minute. Watching Pulse Guardian it showed spikes of messages over the warning limit which were quickly cleared. I don't think these were real messages however.

I shut down the autophone process and deleted the autophone1 queues then restarted. I no longer received the spurious warnings.

I am ok for now but this might bite someone else in the future and I thought you might like to know.

Comment 1

[Chris Cooper [:coop] (he/him)]

bc: please re-open if this recurs.