707522 - celery tasks may not be running on addons.mozilla.org (validation never completes)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Kumar McMillan [:kumar]]

Attachment: what I see in the celery task-ready graph

Some users in production are unable to upload addons because validation never completes (bug 707497). I can reproduce this on https://addons.mozilla.org/en-US/developers/addon/submit/2

The validation appears to run infinitely without errors which leads me to believe that rabbit is acknowledging the task but not running the task (or maybe there is an exception).

Ganglia shows task workers are connected (I think) but tasks seem to be piling up if I'm interpreting this right: https://ganglia.mozilla.org/phx1/graph.php?c=Addons%20Celery&h=celery2&v=69&m=rabbit_prod_zamboni_images_messages_ready&r=hour&z=medium&jr=&js=&st=1323024151&vl=messages&z=large

I cannot verify that there isn't an exception triggered by the validation though because today's log is unexpectedly unreadable:

[[e-mail]1 12]$ pwd
/var/log/clusterlogs/http_app_addons/2011/12
[[e-mail]1 12]$ ls -la 2011-12-04 
-rw------- 1 root root 376 Dec  4 03:48 2011-12-04

Any ideas?

btw, if you need me to help troubleshoot, I'm at 773-726-2397.  Thanks.

Comment 1

[Wil Clouser [:clouserw]]

Not only unreadable but mostly empty.  Yesterday's appears to be completely empty.

Comment 2

[Rick Bryce [:rbryce]]

:oremj restarted the celery service.  Starting to see log data now.  Please confirm

Comment 3

[Kumar McMillan [:kumar]]

Yep, I can upload and validate addons plus the logs are back and graphs look sane. Thanks everyone!