984382 - [ReMo] Celery won't register tasks

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[Giorgos Logiotatidis [:giorgos]]

On reps prod (reps.m.o) celery will not register new tasks. 

We do run the same code on stage and things are OK (tasks register normally).

So maybe that's conf problem after the move to generic cluster?

Comment 1

[Giorgos Logiotatidis [:giorgos]]

We need some hands on this as soon as possible please. Blocks functionality on production.

Comment 2

[Brandon Burton [:solarce]]

I don't see many tasks executing, but I do see some

[generic-celery3.webapp.phx1.mozilla.com] out: [2014-03-14 08:05:15,850: INFO/MainProcess] Got task from broker: remo.base.tasks.send_mail_task[1f785b8b-4560-4064-96dc-c309e248f271]
[generic-celery3.webapp.phx1.mozilla.com] out: [2014-03-14 08:05:15,920: INFO/MainProcess] Task remo.base.tasks.send_mail_task[1f785b8b-4560-4064-96dc-c309e248f271] succeeded in 0.0391399860382s: None
[generic-celery3.webapp.phx1.mozilla.com] out: [2014-03-16 10:38:43,124: INFO/MainProcess] Got task from broker: remo.reports.tasks.send_remo_mail[be1cb469-7f57-4bcb-ad3b-52cb95707680]
[generic-celery3.webapp.phx1.mozilla.com] out: [2014-03-17 01:31:52,371: INFO/MainProcess] Got task from broker: remo.profiles.tasks.check_celery[6ec5251f-8987-40ad-8abf-d414325ea2ae]
[generic-celery3.webapp.phx1.mozilla.com] out: [2014-03-17 01:31:52,443: INFO/MainProcess] Task remo.profiles.tasks.check_celery[6ec5251f-8987-40ad-8abf-d414325ea2ae] succeeded in 0.0553591251373s: None

I've restarted celeryd on all three prod generic celery hosts

Can you provide some more details on where you're seeing issues/errors?

Comment 3

[Brandon Burton [:solarce]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #0)
> On reps prod (reps.m.o) celery will not register new tasks. 
> 
> We do run the same code on stage and things are OK (tasks register normally).
> 
> So maybe that's conf problem after the move to generic cluster?

As an aside, I noticed you're still bundling CeleryD 2.5.5 with reps.mo

[2014-03-18 05:38:33,140: WARNING/MainProcess] -------------- [e-mail] v2.5.5

You may want to look into upgrading the version you ship with reps.mo to 3.1.x

Comment 4

[Giorgos Logiotatidis [:giorgos]]

(In reply to Brandon Burton [:solarce] from comment #3)
> (In reply to Giorgos Logiotatidis [:giorgos] from comment #0)
> > On reps prod (reps.m.o) celery will not register new tasks. 
> > 
> > We do run the same code on stage and things are OK (tasks register normally).
> > 
> > So maybe that's conf problem after the move to generic cluster?
> 
> As an aside, I noticed you're still bundling CeleryD 2.5.5 with reps.mo
> 
> [2014-03-18 05:38:33,140: WARNING/MainProcess] --------------
> [e-mail] v2.5.5
> 
> You may want to look into upgrading the version you ship with reps.mo to
> 3.1.x

I know that this requires new compiled packages. Does generic cluster already what's needed? If not what's the procedure we should follow to make that happen?

Comment 5

[Brandon Burton [:solarce]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #4)
> (In reply to Brandon Burton [:solarce] from comment #3)
> > (In reply to Giorgos Logiotatidis [:giorgos] from comment #0)
> > > On reps prod (reps.m.o) celery will not register new tasks. 
> > > 
> > > We do run the same code on stage and things are OK (tasks register normally).
> > > 
> > > So maybe that's conf problem after the move to generic cluster?
> > 
> > As an aside, I noticed you're still bundling CeleryD 2.5.5 with reps.mo
> > 
> > [2014-03-18 05:38:33,140: WARNING/MainProcess] --------------
> > [e-mail] v2.5.5
> > 
> > You may want to look into upgrading the version you ship with reps.mo to
> > 3.1.x
> 
> I know that this requires new compiled packages. Does generic cluster
> already what's needed? If not what's the procedure we should follow to make
> that happen?

Ah, I wasn't aware of that, I don't know, but if you can provide a specific list we can check

Comment 6

[Brandon Burton [:solarce]]

Everything looks good in the logs http://genericadm.private.phx1.mozilla.com/chief/reps.prod/logs/0ea2346.1395147770 and the script https://github.com/mozilla/remo/blob/master/bin/update/update.py

Can we test this out in real time tomorrow or Thu?

Comment 7

[Giorgos Logiotatidis [:giorgos]]

I'm available on IRC to work on this.

Comment 8

[Giorgos Logiotatidis [:giorgos]]

OK I think I've a clue:

 * Celery must run with 'beat' enabled

Can you please verify this? 

See bug 918816 for more details.

Comment 9

[Brandon Burton [:solarce]]

This is my fault, I failed to port over the beat argument parts when setting up reps prod on generic

I've fixed that and it's running with beat now

[2014-03-20 06:50:19,441: WARNING/MainProcess] -------------- [e-mail] v2.5.5
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      amqp://[e-mail]:5672/reps-prod
- ** ----------   . loader:      djcelery.loaders.DjangoLoader
- ** ----------   . logfile:     /var/log/celeryd-reps.log@INFO
- ** ----------   . concurrency: 4
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        ON
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery


[Tasks]
  . calculate_longest_streaks
  . remo.base.tasks.send_mail_task
  . remo.profiles.tasks.check_celery
  . remo.profiles.tasks.check_mozillian_username
  . remo.profiles.tasks.send_generic_mail
  . remo.remozilla.tasks.fetch_bugs
  . remo.reports.tasks.send_ng_report_notification
  . remo.reports.tasks.send_remo_mail
  . remo.reports.tasks.send_report_digest
  . remo.reports.tasks.zero_current_streak
  . remo.voting.tasks.send_voting_mail
[2014-03-20 06:50:19,462: INFO/PoolWorker-2] child process calling self.run()
[2014-03-20 06:50:19,464: INFO/PoolWorker-3] child process calling self.run()
[2014-03-20 06:50:19,466: INFO/PoolWorker-4] child process calling self.run()
[2014-03-20 06:50:19,468: INFO/PoolWorker-5] child process calling self.run()
[2014-03-20 06:50:19,471: WARNING/MainProcess] [e-mail] has started.
[2014-03-20 06:50:19,471: INFO/Beat] child process calling self.run()
[2014-03-20 06:50:19,472: INFO/Beat] Celerybeat: Starting...
[2014-03-20 06:50:19,668: INFO/Beat] Scheduler: Sending due task remo.profiles.tasks.check_mozillian_username
[2014-03-20 06:50:19,688: INFO/Beat] Scheduler: Sending due task remo.reports.tasks.send_report_digest
[2014-03-20 06:50:19,692: INFO/Beat] Scheduler: Sending due task celery.backend_cleanup
[2014-03-20 06:50:19,717: INFO/MainProcess] Got task from broker: celery.backend_cleanup[0589b2e2-a24e-41c7-8ff0-b8a91a439ab0]
[2014-03-20 06:50:19,751: INFO/MainProcess] Task celery.backend_cleanup[0589b2e2-a24e-41c7-8ff0-b8a91a439ab0] succeeded in 0.00050687789917s: None
[2014-03-20 06:55:31,607: INFO/MainProcess] Got task from broker: celery.backend_cleanup[f23872dc-4c5a-4fc4-ab54-43a0ba9d5f5d]
[2014-03-20 06:55:31,668: INFO/MainProcess] Task celery.backend_cleanup[f23872dc-4c5a-4fc4-ab54-43a0ba9d5f5d] succeeded in 0.000600099563599s: None

Comment 10

[Giorgos Logiotatidis [:giorgos]]

Much better. Thanks solarce!

Comment 11

[Giorgos Logiotatidis [:giorgos]]

We have jobs managed by celery beat that run double the times on prod only. This can be caused if two celery beat servers run concurrently. 

:solarce can you please verify there is only one celery beat server?

Comment 12

[Brandon Burton [:solarce]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #11)
> We have jobs managed by celery beat that run double the times on prod only.
> This can be caused if two celery beat servers run concurrently. 
> 
> :solarce can you please verify there is only one celery beat server?

Ah, this could be a problem then, the generic cluster has three celery servers to fan out the load

Mozillians is running in the same fashion and I don't think it causes issues

I'll do some r&d into celerybeat and HA/fanout

Comment 13

[Brandon Burton [:solarce]]

It looks like there is no way around the "single celerybeat daemon" requirement, so we're going to get a VM provisioned and move the celerybeat processes to it

VM being requested in bug 994835

Comment 14

[Giorgos Logiotatidis [:giorgos]]

Stage has one only celery beat process running?

Comment 15

[Brandon Burton [:solarce]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #14)
> Stage has one only celery beat process running?

Correct, dev and stage only have a single celery server

Comment 16

[Giorgos Logiotatidis [:giorgos]]

(In reply to Brandon Burton [:solarce] from comment #12)
> 
> Mozillians is running in the same fashion and I don't think it causes issues
>

Well, that means that mozillians prod jobs run multiple times too but because of the nature of the jobs we didn't experience any problems. On Reps we send emails using Periodic Tasks and users started complaining about multiple emails. 

Do avoid the same issue on mozillians in the future, do you mind doing the same on mozillians prod too? I don't have access to bug 994835 to copy it myself.

Thanks!