builder is making celery thrash



7 years ago
4 years ago


(Reporter: clouserw, Unassigned)





7 years ago
nagios has been paging about load for days on celery (which is all the annoying stuff in #flightdeck).  Turns out it's the celeryd-builder-dev thread so the job is from builder.

There are only a couple cron jobs running and they are in the middle of the night - perhaps this is a task that is triggered off of something user action?

We restarted celery and emptied the queue so starting from scratch.  Logs are on syslog1.webapp.phx1 in /var/log/clusterlogs/http_app_builder_dev/2012/01

If it doesn't start paging again, we can close this, but filing since I have it written out already.

Comment 1

7 years ago
This is happening again and was identified as builder again.
Do we know what jobs celery couldn't crunch through? Is it a job that lasts too long, or too many jobs at once?

Comment 3

7 years ago
celeery's tools aren't very good so it's hard to pinpoint the cause of the problem.  Once it's identified (high load) we end up with a backlog of jobs so there are too many and they take too long.

You can tell what is running from the logs, but that's about the best you can do with celery as far as I know. :(
is this still an issue (cleanup)

Comment 5

5 years ago
The builder project has been discontinued as of April 1st, 2014.

The announcement last year:

Next steps (still being written):
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
Product: → Graveyard
You need to log in before you can comment on or make changes to this bug.