mozillians queues have high rabbit unread messages

NEW
Assigned to

Status

Participation Infrastructure
Phonebook
8 months ago
8 months ago

People

(Reporter: sal, Assigned: ericz)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

8 months ago
Created attachment 8862629 [details]
mozillians queues

I asked #webops for help on this 


@nagios-scl3> Thu 21:35:04 UTC [5886] [webops] rabbit2.webapp.phx1.mozilla.com:Webapp Rabbit Unread Messages is WARNING: RABBITMQ_OVERVIEW WARNING - messages WARNING (5027) messages_ready WARNING (5027), messages_unacknowledged OK (0) (http://m.mozilla.org/Webapp+Rabbit+Unread+Messages)


@nagios-scl3> Thu 21:40:31 UTC [5888] [webops] rabbit1.webapp.phx1.mozilla.com:Webapp Rabbit Unread Messages is WARNING: RABBITMQ_OVERVIEW WARNING - messages WARNING (5053) messages_ready WARNING (5053), messages_unacknowledged OK (0) (http://m.mozilla.org/Webapp+Rabbit+Unread+Messages)
(Assignee)

Comment 1

8 months ago
Celery died for mozillians and airmo.  We restarted it and celery barfed thousands of queues with 1 to 3 messages each into Rabbit which caused some high load but those seem to be expiring now and everything appears to be settling down.  The main app queues appear to be good.
Assignee: nobody → eziegenhorn
(Reporter)

Comment 2

8 months ago
recoveries;  

@nagios-scl3> Thu 23:00:21 UTC [5947] [webops] rabbit1.webapp.phx1.mozilla.com:Webapp Rabbit Unread Messages is OK: RABBITMQ_OVERVIEW OK - messages OK (3551) messages_ready OK (3551) messages_unacknowledged OK (0) (http://m.mozilla.org/Webapp+Rabbit+Unread+Messages)

@nagios-scl3> Thu 23:05:04 UTC [5952] [webops] rabbit2.webapp.phx1.mozilla.com:Webapp Rabbit Unread Messages is OK: RABBITMQ_OVERVIEW OK - messages OK (3538) messages_ready OK (3538) messages_unacknowledged OK (0) (http://m.mozilla.org/Webapp+Rabbit+Unread+Messages)
You need to log in before you can comment on or make changes to this bug.