672218 - New jenkins service in PHX

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Corey Shields [:cshields]]

As discussed during work week, jenkins is a critical service to webdevs and should be given some better resources to do its job. We will spin up a new dedicated server in phx1 to do this.

Comment 1

[Jeff Balogh (:jbalogh)]

AMO is now running tests on RHEL5 and deploying on RHEL6. And oremj installs packages for RHEL6 systems but doesn't have rpms for RHEL5. And hudson runs tests slower than my laptop.

Can we get one of those shiny phx boxes hooked up to the existing hudson? I don't think we need to move the whole system over right away, unless that's easier.

Comment 2

[Shyam Mani [:fox2mike]]

It is chicken and egg, one can't come without the other. 

And no, hooking up phx -> sjc is not a good plan. 

If this is critical, then we'll have to setup stuff in phx :|

Comment 3

[Wil Clouser [:clouserw]]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> It is chicken and egg, one can't come without the other. 
> 
> And no, hooking up phx -> sjc is not a good plan. 
> 
> If this is critical, then we'll have to setup stuff in phx :|

There is cross DC communication happening already.  If you don't want to add some more to send the results of tests being sent back and forth, then yeah, let's set the whole thing up in phx.  It takes 30 minutes to run the tests right now every time a developer commits.  Anything we can do to bring that down I'm all for.

Comment 4

[Shyam Mani [:fox2mike]]

I don't want to add on to the cross DC communication at all.

I'm sure you'd also like this to come up with all the history intact? :) 

I'll try and get something going by end of next week.

Comment 5

[Wil Clouser [:clouserw]]

I don't care too much about the history, but if it's easy, sure.

Is your plan to move the entirety of hudson.mozilla.org to phx and just run all jobs there?  If so, that's fine, but I don't want to have 2 hudson.mozilla.org domains to try to explain to developers.

An alternative to this may be to reuse all the AMO hardware that is idle in MPT for additional slaves.  Maybe that's not as long term though.

Comment 6

[Wil Clouser [:clouserw]]

Also, it looks like there are ~30 jobs on Jenkins right now and I have no idea how many are on RHEL 5 vs 6 or if it matters to those projects.  Maybe a good move would be to set up a non-default RHEL 5 slave that those projects can use and email their owners to ask them to test on RHEL 6 and move them over when they work.

Or maybe they'll work, but there should be communication there.

Comment 7

[Shyam Mani [:fox2mike]]

(In reply to Wil Clouser [:clouserw] from comment #5)
> I don't care too much about the history, but if it's easy, sure.

I'm guessing it is, I'll look into it.

> Is your plan to move the entirety of hudson.mozilla.org to phx and just run
> all jobs there?  If so, that's fine, but I don't want to have 2
> hudson.mozilla.org domains to try to explain to developers.

Yes. Everything moves to phx1 and runs on RHEL 6, stuff that needs RHEL 5, we'll add a RHEL 5 slave and they can only build on that. 

> An alternative to this may be to reuse all the AMO hardware that is idle in
> MPT for additional slaves.  Maybe that's not as long term though.

MPT goes away soon enough that this will be a problem. By moving away from MPT, we avoid this altogether.

Comment 8

[Shyam Mani [:fox2mike]]

(In reply to Wil Clouser [:clouserw] from comment #6)

> Or maybe they'll work, but there should be communication there.

There will be. When I have something to communicate about :)

I was thinking about something along the lines of move everything, and if people want 5, they can build on the 5 slave, else they build on the master which is 6.

I will send out an email before I do all this, and ask for input.

Comment 9

[Shyam Mani [:fox2mike]]

Corey,

Can I have 2 blades for this please? 12-18 GB of RAM, 8 cores or higher. Thanks!

Comment 10

[Corey Shields [:cshields]]

(In reply to Shyam Mani [:fox2mike] from comment #9)
> Corey,
> 
> Can I have 2 blades for this please? 12-18 GB of RAM, 8 cores or higher.
> Thanks!

we've been working on this already, will let you finish it out.  Touch base with jabba about any work he has done in puppet:

Last login: Fri Sep  9 06:31:44 2011 from 10.8.74.5
[[e-mail]1 ~]#

https://inventory.mozilla.org/systems/show/3764/

Comment 12

[James Socol [:jsocol, :james]]

(Adding "PHX" to the summary so I can find this easier.)

WebQA has a goal of eventually running their Selenium tests off the same Jenkins instance as webdev. Not sure if that affects anything at the moment, but it's something worth keeping in mind. Will have to work out questions of where their servers run, physically and networkly.

Comment 13

[Jeff Balogh (:jbalogh)]

Just to pile on some more, this is making it hard for ozten to set up tests that require ldap and/or node.js since he wants to run those in a vagrant VM.

fox2mike: do you still expect the ETA to be around 10/15?

Comment 14

[Shyam Mani [:fox2mike]]

(In reply to Jeff Balogh (:jbalogh) from comment #13)
> Just to pile on some more, this is making it hard for ozten to set up tests
> that require ldap and/or node.js since he wants to run those in a vagrant VM.
> 
> fox2mike: do you still expect the ETA to be around 10/15?

+/- a couple of days at most. I have one thing to finish before I get to this and that's higher priority.

Comment 15

[Shyam Mani [:fox2mike]]

Just another update : I've worked with individual stakeholders and gotten their projects to build fine on the new machine. We're just waiting on one more project to be "fixed" and then we should be good to migrate this from sjc1 -> phx1.

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Shyam Mani [:fox2mike] from comment #15)
> Just another update : I've worked with individual stakeholders and gotten
> their projects to build fine on the new machine. We're just waiting on one
> more project to be "fixed" and then we should be good to migrate this from
> sjc1 -> phx1.

Case Conductor, and then WebQA, right?

Comment 17

[Shyam Mani [:fox2mike]]

(In reply to Stephen Donner [:stephend] from comment #16)

> Case Conductor, and then WebQA, right?

Pretty much :) A good thing for WebQA to do would be to prepare a list of stuff you've installed on the mac mini etc (dependencies we'd need for your stuff) and the like. 

I've still not had time to login and poke around :)

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Shyam Mani [:fox2mike] from comment #17)
> (In reply to Stephen Donner [:stephend] from comment #16)
> 
> > Case Conductor, and then WebQA, right?
> 
> Pretty much :) A good thing for WebQA to do would be to prepare a list of
> stuff you've installed on the mac mini etc (dependencies we'd need for your
> stuff) and the like. 
> 
> I've still not had time to login and poke around :)

We use pip install and virtualenv (along with Selenium, beautifulsoup, etc.), but I don't think/know of any permanent libraries we need on the Jenkins box -- we do and will use some plugins, though.  http://qa-selenium.mv.mozilla.com:8080/pluginManager/installed has the list, but we can probably prune it a little.

Comment 19

[Shyam Mani [:fox2mike]]

https://ci.mozilla.org/ is now live. The older jenkins and hudson instance has been shut down. The URLs will still be valid, even over SSL.

If you have any immediate issues because of this migration, please comment on this bug, else file new bugs.

I'm leaving this open for new blockers I'm filing for monitoring and backups to be fixed before I close this out.

Comment 20

[Shyam Mani [:fox2mike]]

Going to call this one fixed :) The remaining blocker has been open for a while and is on netops.