948557 - [mozillians-dev] kick celery and maybe elasticsearch too

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[Andrei Hajdukewycz [:sancus]]

Attempts to reindex are taking forever and sometimes just failing partway through on dev. I suspect this is Celery processing tasks really slowly if at all, or possibly it's an ES issue... but celery is at fault here far more often.

Comment 1

[Chris Turra [:cturra]]

i'm not sure kicking the clery process will resolve what you're seeing. i am watching the celery logs for mozillians-dev and it's actively processing jobs at a pretty high rate.

Comment 2

[Andrei Hajdukewycz [:sancus]]

That's odd, why is it processing so many jobs? Maybe it's being flooded by... something?

Comment 3

[Chris Turra [:cturra]]

(In reply to Andrei Hajdukewycz [:sancus] from comment #2)
> That's odd, why is it processing so many jobs? Maybe it's being flooded
> by... something?

i wouldn't be able to answer that - i can just see the tasks coming in and being processed. the last minute saw 13 tasks complete.

[[e-mail]1 ~]# grep "2013-12-10 13:55" /var/log/celeryd-mozillians-dev.log | grep succeeded | wc -l
13

Comment 4

[Andrei Hajdukewycz [:sancus]]

Thanks for looking at this, I think I found the culprit. Running the chief update script every 15 minutes can be bad when you are doing db/celery intensive stuff in that script :p