nagios has been paging about load for days on celery (which is all the annoying stuff in #flightdeck). Turns out it's the celeryd-builder-dev thread so the job is from builder. There are only a couple cron jobs running and they are in the middle of the night - perhaps this is a task that is triggered off of something user action? We restarted celery and emptied the queue so starting from scratch. Logs are on syslog1.webapp.phx1 in /var/log/clusterlogs/http_app_builder_dev/2012/01 If it doesn't start paging again, we can close this, but filing since I have it written out already.
This is happening again and was identified as builder again.
Do we know what jobs celery couldn't crunch through? Is it a job that lasts too long, or too many jobs at once?
celeery's tools aren't very good so it's hard to pinpoint the cause of the problem. Once it's identified (high load) we end up with a backlog of jobs so there are too many and they take too long. You can tell what is running from the logs, but that's about the best you can do with celery as far as I know. :(
is this still an issue (cleanup)
The builder project has been discontinued as of April 1st, 2014. The announcement last year: https://blog.mozilla.org/addons/2013/12/18/add-on-builder/ Next steps (still being written): https://developer.mozilla.org/en-US/Add-ons/SDK/Builder
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.