572230 - Second SUMO staging environment

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[James Socol [:jsocol, :james]]

Having two staging environments that were both similar to production (see bug 517314) would allow:

* Increased parallelism. QA could work with our .next milestone while we are finishing up issues that, like our current set, block a release.
* Improve confidence in our code and reduce backed-out pushes. The closer to a production environment we can get in staging, the less likely we are to find issues during pushes.

mrz says this is could happen in July, which works out fine for me.

Comment 1

[matthew zeier [:mrz]]

Phong, can you make sure we have the hardware we need for this?  oremj can offer suggestions.

Comment 2

[James Socol [:jsocol, :james]]

Would we be able to move our current staging server off its VM and on to this hardware, as well?

Comment 3

[matthew zeier [:mrz]]

following up from irc, we'll put this in phoenix.  need a local stage db pair there too i imagine.

Comment 4

[James Socol [:jsocol, :james]]

Matthew/Shyam: any timeline for this?

Comment 5

[Shyam Mani [:fox2mike]]

I'd say about 2 weeks out. Passing over to Phong to make sure machine is racked up.

Comment 6

[James Socol [:jsocol, :james]]

Can we get an updated ETA? We'd like to start using this environment.

Comment 7

[Phong Tran [:phong]]

https://inventory.mozilla.org/systems/show/2210/
https://inventory.mozilla.org/systems/show/2211/

We need to set these up.

Comment 8

[Shyam Mani [:fox2mike]]

James,

With everything that's on my plate now, I'd give you an estimate of end August or in the first week of September.

If there is a really pressing need for this setup before that timeframe, then I might be able to pull a few hours at night or something and get it going before then.

Thoughts?

Also, I have 2 machines, but I'm only planning to use one. This will have support-stage.mozilla.com and next-support-stage.mozilla.com. The spare one can probably be utilized for some other stage project or a db server.

Comment 9

[James Socol [:jsocol, :james]]

(In reply to comment #8)
> With everything that's on my plate now, I'd give you an estimate of end August
> or in the first week of September.

Is there any way you or someone else can help accelerate that? We'd really like to start using both (for a "trunk" branch and a "current release" branch) as soon as possible, and were told these would be ready in July.

> Also, I have 2 machines, but I'm only planning to use one. This will have
> support-stage.mozilla.com and next-support-stage.mozilla.com. The spare one can
> probably be utilized for some other stage project or a db server.

As long as the everything that needs to run on these boxes (like celery, I expect) can be set up in parallel without ever causing conflicts. We will be very sad pandas if one staging environment takes down the other. Hopefully that doesn't mean VMs. We'd all really like the performance of real hardware on stage.

If we can't guarantee that separation, then I'd feel better if they were on separate boxes.

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to comment #9)

> Is there any way you or someone else can help accelerate that? We'd really like
> to start using both (for a "trunk" branch and a "current release" branch) as
> soon as possible, and were told these would be ready in July.

Alright, done. Mid next week, at most. I'll push back a few other things and get this going. You have any naming preferences? tags or something that I need to pull from? Also, I'm planning to use the existing dbs.
 
> As long as the everything that needs to run on these boxes (like celery, I
> expect) can be set up in parallel without ever causing conflicts. We will be
> very sad pandas if one staging environment takes down the other. Hopefully that
> doesn't mean VMs. We'd all really like the performance of real hardware on
> stage.

I think that should be too much of an issue. Would be nice to see some sad sumo pandas once in a while though ;)
 
> If we can't guarantee that separation, then I'd feel better if they were on
> separate boxes.

I'm fairly sure amo runs multiple celery instances etc on the same box, so shouldn't be an issue.

Comment 11

[James Socol [:jsocol, :james]]

(In reply to comment #10)
> Alright, done. Mid next week, at most. I'll push back a few other things and
> get this going. You have any naming preferences? tags or something that I need
> to pull from? Also, I'm planning to use the existing dbs.

Thanks, Shyam. It means a lot to us and will help all of us and QA as well.

Comment 12

[Shyam Mani [:fox2mike]]

So while setting up kitsune, (running pip -r install requirements-dev.txt)

Downloading/unpacking PIL==1.1.7 (from -r requirements.txt (line 22))
  Could not find any downloads that satisfy the requirement PIL==1.1.7 (from -r requirements.txt (line 22))
No distributions at all found for PIL==1.1.7 (from -r requirements.txt (line 22))

So I commented it out for now and continued, but something for you to look at?

Comment 13

[Shyam Mani [:fox2mike]]

Also, ran into this: 

[root@dm-app-sumo01 kitsune]# pip install -r requirements-dev.txt 
Obtaining test-utils from git+git://github.com/jbalogh/test-utils.git@c4c31905a95e59dcc8919c1030b23848ad7fbca6#egg=test-utils (from -r requirements-dev.txt (line 13))
  Updating ./src/test-utils clone (to c4c31905a95e59dcc8919c1030b23848ad7fbca6)
You are not currently on a branch, so I cannot use any
'branch.<branchname>.merge' in your configuration file.
Please specify which remote branch you want to use on the command
line and try again (e.g. 'git pull <repository> <refspec>').
See git-pull(1) for details.
  Complete output from command /usr/bin/git pull -q:
  
----------------------------------------
Command /usr/bin/git pull -q failed with error code 1

So I'm not sure how much of the kitsune setup actually works :) Let's debug over IRC?

Comment 14

[Shyam Mani [:fox2mike]]

Alright, so a short status update :

1) We have a version of sumo on hardware, running on http://dm-app-sumo01.mozilla.org/

2) Still needs a bunch of things configured - including ssl and zeus.

3) It will be called master-support.mozilla.com or master.support.mozilla.com

4) We will start off with one hardware instance for now, and continue to use mrapp-stage04 with support-stage. Moving forward, we can setup another instance on hardware, this is probably q4 or ahead.

5) Stuff to be done :
* zeus
* rabbit/celery
* db (wipe out and get a new one for this, right now it's sharing support-stage)
* sphinx (same as above)
* naming
* SSL certs (self signed okay, since this is stage).

James, let me know if I've missed out anything. Once the above is complete, I'm going to call this closed.

Comment 15

[Shyam Mani [:fox2mike]]

And auto-update stuff, as well as crons.

Comment 16

[Shyam Mani [:fox2mike]]

(In reply to comment #14)
> 5) Stuff to be done :
> * zeus

Done, this is behind the production zeus cluster, btw.

> * rabbit/celery

Done.

> * db (wipe out and get a new one for this, right now it's sharing
> support-stage)

Done. master_support_mozilla_com on tm-stage01-master01

> * sphinx (same as above)

Done. master.support, sql port is 3387 and access from khan enabled.

> * naming

Done.

> * SSL certs (self signed okay, since this is stage).

Done.


Auto-updates as well as crons are now in place as well.

Partay, http://master.support.mozilla.com/ is now live :) Reopen this bug in case of issues related to this OR file new bugs and block this one and I'll get to them.

Comment 17

[Shyam Mani [:fox2mike]]

Also, we'll be downgrading the RAM on the box from 18GB to 6GB since 18 is overkill for a stage machine and wasn't originally intended for this machine. Tracking that in Bug 590505 and we'll ask you guys for some downtime before we do that.