483416 - pm-app-amodev01 setup

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

Sometimes bugs are only reproducible in the production environment.  In a case like this I'd like to be able to do:

1) IT pulls a box from the production server pool, but it remains completely set up to talk to production.

2) They change whatever config variables they need to (in AMO's case, probably setting DEBUG=2)

3) They give webdev a URL where they can hit the box directly

Comment 1

[Dave Miller [:justdave]]

This ought to be easy enough to pull off...  just set up a testing VIP in the Netscaler (or in zeus) and point that VIP at the box in question...  Even if we don't pull it out of the production pool, we can at least be sure they're hitting the box we're playing with the debug stuff on...  give them that IP and have them put it in their hosts file.

Comment 2

[Wil Clouser [:clouserw]]

I'm not sure if you're saying to turn on debug on a production server but I don't think it's a good idea until it's out of the pool.  Debug exposes a lot of info and uses considerably more cpu/ram.

Also, I'd prefer not to use the hosts file.  With the way browsers cache DNS it can be difficult to tell if the new address has kicked in yet.

Comment 3

[matthew zeier [:mrz]]

oremj, what do you think?  You think this could live as a VM and only be included into load balancing when debugging?

Comment 4

[Jeremy Orem [:oremj]]

If we don't want to use the hosts file I could set up a new internal amo vip on different sub-domain.  That VIP would only point to our "production" VM, which would always have debugging turned on and could only be accessed from behind the corporate firewall.  We won't be able to set it up exactly like a production machine, because the code syncing will overwrite the DEBUG define.

Would hosting it on a different domain name matter?

Comment 5

[Wil Clouser [:clouserw]]

(In reply to comment #3)
> oremj, what do you think?  You think this could live as a VM and only be
> included into load balancing when debugging?

I don't know enough about the back end to speak intelligently about our setup but if this means having a VM that is only used when webdev asks a question I don't think that's the best way to go about it.  That's the kind of thing that gets forgotten when something changes (config, file paths, whatever) and then we end up debugging why it doesn't work instead of the actual problem.  I think using an actual production machine is a better idea because then there is no question about what configs or file permissions or whatever.  Additionally, if one server starts acting up we can use the headers on AMO to figure out which one it is and pull that one out and work on it specifically.

(In reply to comment #4)
> If we don't want to use the hosts file I could set up a new internal amo vip on
> different sub-domain.  That VIP would only point to our "production" VM, which
> would always have debugging turned on and could only be accessed from behind
> the corporate firewall.  We won't be able to set it up exactly like a
> production machine, because the code syncing will overwrite the DEBUG define.
> 
> Would hosting it on a different domain name matter?

Hosting it on a different domain would matter.  Since I'd like to stay as close to it being production as possible I may be out of luck on the hosts file issue.  If so, that's fine.  I'll find an add-on that displays the IP of the server or something.

Comment 6

[matthew zeier [:mrz]]

(In reply to comment #5)
> (In reply to comment #3)
> > oremj, what do you think?  You think this could live as a VM and only be
> > included into load balancing when debugging?
> 
> I don't know enough about the back end to speak intelligently about our setup
> but if this means having a VM that is only used when webdev asks a question I
> don't think that's the best way to go about it.  That's the kind of thing that
> gets forgotten when something changes (config, file paths, whatever) and then
> we end up debugging why it doesn't work instead of the actual problem.  I think
> using an actual production machine is a better idea because then there is no
> question about what configs or file permissions or whatever. 

I'm not suggesting that the VM be divergent from production - from your perspective it wouldn't be any different than a physical production AMO server.  I would expect it to look not unlike mrapp01 which gets treated like an AMO node even though it only runs memcached.


> Additionally, if
> one server starts acting up we can use the headers on AMO to figure out which
> one it is and pull that one out and work on it specifically.

This, of course, can be done right now.

Comment 7

[Jeremy Orem [:oremj]]

I'll start working on this again next week when Wil is back.

Comment 8

[matthew zeier [:mrz]]

I want to revisit this and move forward with it.

Want to setup an AMO-like server (VM) that lead AMO devs can access.  Host should be exactly like production (minus production traffic).  

This should facilitate debugging during content pushes.

We'll try to follow these guidelines:
1. webdev will have root-like access to view files
2. host will be treated like production - configs & content will be pushed like other pm-app-amo servers
3. webdev won't make any changes
4. IT will push production load to it only when necessary to debug production issues with webdev
5. What Wil said in comment #0.

Sound good?

Comment 9

[Jeremy Orem [:oremj]]

I'm not sure this bug will help the sphinx bug, since sphinx doesn't run on the app nodes. If we do put sphinx on this node it will no longer be like the rest of the AMO cluster.

Comment 10

[matthew zeier [:mrz]]

Host up @ 10.2.83.11.  Storage interface @ 10.253.0.66.

Needs:
1. AMO puppet config
2. openssh+lpk & sudo for certain webdev folk
3. monitoring

Comment 11

[matthew zeier [:mrz]]

I think this is done and ddash used it (or is still using it).

We can talk about the specifics of access Thursday.