680572 - [amo] Make code updates smoother

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Jeff Balogh (:jbalogh)]

Restarting the python processes to pick up new code gives us an awful load spike. It makes us scared to push updates when we're at high load and makes our graphs look bad.

My ideas:
* Take boxes out of the zeus pool (1-5(ish) at a time) and let them update in peace
* Try out gunicorn instead of mod_wsgi

Other than fear, this is the big blocker from us pushing more often.

Comment 1

[Jeff Balogh (:jbalogh)]

Load: http://cl.ly/2S011N1x3w282a1V3m1J
Responses per second: http://cl.ly/1q1Q2c0P3g3n413p2P42

Comment 2

[Corey Shields [:cshields]]

justdave uses the "few at a time" method for bugzilla updates, and he might have a script already to do that we could adapt to the deploy script.

but yeah, +1 to taking a couple of nodes out of zlb at a time and doing the updates.

Comment 3

[Jeremy Orem [:oremj]]

Yeah, I looked at that script a few months ago. Problem is, using that method with amo makes pushes take a really really long time, because the server count is so much higher and AMO has too much load to take large chunks out at a time.

Comment 4

[Jeremy Orem [:oremj]]

We won't attempt any fixes until after the PHX move.

Comment 6

[Jeremy Orem [:oremj]]

I don't think this is a problem anymore after the move to gunicorn.