Closed Bug 939745 Opened 12 years ago Closed 12 years ago

MDN: celery consumer: connection to broker lost

Categories

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Community Platform
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: groovecoder, Assigned: cliang)

References

Details

45k of these since the network outage: https://errormill.mozilla.org/mdn/mdn-dev/group/41731/ I was looking into this because of bug 939730, which is affecting prod - not just dev. So, I'm bumping the priority of this as a potential cause of the prod blockage.
I restarted celery and it looks like messages are leaving RabbitMQ. Please let me know if this has fixed things.
Assignee: server-ops-webops → cliang
[fscholz 9:37] yay, looks good to me [fscholz 9:37] cyliang: ^ [cyliang 9:37] does the happy Snoopy dance [cyliang 9:37] Thanks.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Got a bunch of these on dev [1] and stage [2] when trying to re-index wiki documents into ElasticSearch. [1] https://errormill.mozilla.org/mdn/mdn-dev/group/41731/ [2] https://errormill.mozilla.org/mdn/mdn-stage/group/62109/ Did we restart all the celery clients including the ones that run tasks for dev and stage servers?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
My bad: I hadn't realized that this problem affected dev and stage as well. I've restarted the celery processes there. Can you please verify if this has fixed the re-index issues?
Hmm ... still getting these error events on dev. :( https://errormill.mozilla.org/mdn/mdn-dev/group/41731/
Ended up restarting RabbbitMQ after getting some anomalies in the webUI when trying to monitor messages in the queue (a lot of reference errors that would appear and disappear). This seems to have cleared dev and stage; a full search re-index on prod was initiated without warning. The errormill links referenced above all look clear for the past several hours, so I'm closing this bug. However, if things are still going awry, please let me know.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Can we - i.e., mdn dev team - get access to the web ui for RabbitMQ? Is it accessible on the VPN? Should I open another bug?
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.