901971 - mdn: errant celery behaviour (overloaded?)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task, P3)

Description

[Luke Crouch [:groovecoder]]

https://errormill.mozilla.org/mdn/mdn/group/61571/ is causing a bunch of production ISE's. Has something changed with our developer-prod celery node? Why is the 'celery' queue gone?

Please joine #mdndev to debug with us.

Comment 1

[Jake Maul [:jakem]]

I'm not entirely sure what caused this, but we restarted celery (on developer-celery1.webapp.scl3) as well as rabbitmq (on rabbit1/2.webapp.scl3) and the problem is resolved.

While troubleshooting this, we discovered that this rabbit queue is actually *very* large... over 700k messages. We cleared it, but the problem seems to be that every celery tasks generate an average of 1 more celery task! Thus the incoming rate is always higher than the outgoing rate.

Recently (last week) the volume of celery tasks did increase greatly. ElasticSearch indexing and building JSON metadata became celery tasks. Combined these now account for 99%+ of all celery jobs processed, according to the log on developer-celery1.webapp.scl3. That node is constantly near-max on CPU usage.

The current speculation is that an ES reindex (a celery task now) kicks off a json build (a celery task), which kicks off a render (not celery), which kicks off a reindex task (back to celery, repeat).

... or something like that.

Comment 2

[Daniel Maher [:phrawzty]]

See also :
https://groups.google.com/forum/#!topic/mozilla.dev.mdn/jCaPu1x6wTY

Comment 3

[Daniel Maher [:phrawzty]]

Hello,

Is this bug still valid ?  If so, I would like to assemble the list of specific, actionable items required in order to see it closed.  If not, I'll just close it off for now.

Thanks !

Comment 4

[Luke Crouch [:groovecoder]]

This one is done. I also filed bug 908204 so we can try to stay on top of this ourselves.