696219 - setup celery for affiliates stage

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Osmose [:osmose, :mkelly]]

The celery nodes powering affiliates-dev and affiliates production need to be synced with the latest app code and then restarted.

This should become part of the normal push script to make sure they are always up to date.

Comment 1

[Jake Maul [:jakem]]

celery restarted for dev and prod.

Restarting this for prod isn't a big deal because that's on-demand, but are we sure we want to do so for dev? It takes a few seconds to shut down and start back up, and due to the way this is structured it would be running basically every 5 minutes.

Comment 2

[Jake Maul [:jakem]]

Prod update script is patched to do this during a push.

Let me know if there's something gentler we can do for celery on dev.

Comment 3

[Chris More [:cmore]]

Can't break prod anymore! I tested using this URL https://affiliates.mozilla.org/link/banner/200/1/1000 and it didn't insert invalid data into the database and thus making my_banners throw an exception. Works fine now.

Comment 4

[Osmose [:osmose, :mkelly]]

Dev updates every 15 minutes, so wouldn't the restart only run once every 15 minutes?

As long as celery won't drop jobs during the restart, I'm fine with a few seconds during the update of downtime.

Comment 5

[Matt Brandt [:mbrandt]]

Thanks for bumping Celery on prod.

Comment 6

[Jake Maul [:jakem]]

Unfortunately no, that's not quite how it works.

The admin node updates every 15 minutes. it pulls from your git, then checks into a local git repo. The servers update themselves from the admin node's repo every 5 minutes.

During a manual push, the admin node can trigger (force) remote commands to be run. During automatic pushes, it cannot. Hence, for celery to restart, it would have to run on the actual servers.

We could probably set up an entirely separate cronjob to restart celery. The timing might be tricky- you may end up in situations where it takes an extra 15 minute cycle for celery to restart. Let's do some research on this... I don't know what this looks like in puppet, and don't want to bolt on something we can't easily replicate later.


As far as dropping jobs, I have no idea how celery behaves during a restart. The AMO devs might have some info on this... I know they often have to deal with 10+ minute celery restarts... possibly to avoid this very problem, I really couldn't say.

Comment 7

[Osmose [:osmose, :mkelly]]

How about this then: We remove celery from dev, and add it to stage? Stage should really be representative of production, and it's only updated manually.

Comment 8

[Matt Brandt [:mbrandt]]

QA +1's this. thx mkelly and Jake

Comment 9

[Jake Maul [:jakem]]

I have commented out the BROKER/CELERY settings in affiliates-dev's settings/local.py. Let me know if any more than that is required to disable celery there. I'm dropping severity on this since dev/stage/prod should all be fully functional now... this is now just a new celery setup for stage.

I agree with setting stage up to use celery... "more like prod" is always good. I'll need to get some help on this, but I don't expect it to be too hard. I just want to make sure it gets done properly (puppetized, etc), and I don't really know a lot about celery/rabbitmq. :)

Comment 10

[Jake Maul [:jakem]]

Renormalizing priority levels... P4 is "normal" now.

Comment 11

[Jason Crowe [:jd]]

Closing this bug as it was done in a new bug. Bug 833944