1364869 - [SUMO] Confirm rabbitmq is healthy and reachable from celery instances

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Giorgos Logiotatidis [:giorgos]]

Sentry reports many errors (about every second) probably related to connectivity issues between support-celery1.webapp.phx1.mozilla.com and RabbitMQ

Appeared on May 9th and still goes on.

Can you please check that rabbit-support-prod rabbitmq is running, healthy and reachable from support-celery1.webapp.phx1.mozilla.com and support-celery2.webapp.phx1.mozilla.com?


Sentry error https://sentry.prod.mozaws.net/operations/sumo/issues/206090/

Comment 1

[Eric Ziegenhorn :ericz]

I can't see that error, can you either paste it here or get me a login?  From my perspective celery and rabbit both looks perfectly happy, so I'd love to see those logs.  From past experience, celery doesn't recover from connection blips very well and may just need to be restarted.  Of course, now we've just done DB maintenance which could make it better or worse -- how does it look now?

Comment 2

[Giorgos Logiotatidis [:giorgos]]

What we're seeing in Sentry is similar to bug 1210881. It may be related to the extra stress db is experiencing 1363892. 

We verified that connectivity between celery and rabbitmq is fine so I'm marking this FIXED and will investigate StopIteration error in bug 1210881.