592793 - tools-staging-master vm can't keep up with sendchanges

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: General, task)

Description

[alice nodelman [:alice] [:anode]]

The tools-staging-master vm is hosted on the resource constrained ESX host in mountain view - thus it is unable to keep up with all of the sendchanges from production build machines.

We can keep things afloat here by reducing the talos masters on tools-staging-master to only test moz-central and ignore all other branches/tests.  The long term solution is having the esx host upgraded.

Comment 1

[matthew zeier [:mrz]]

What luck!  Hardware just arrived today.  This is likely to be after the 42 builders though.

Comment 2

[matthew zeier [:mrz]]

Ben - can you spin up this VM for Alice on the KVM cluster @ Castro?

Comment 3

[Ben Kero [:bkero]]

Sure,  I'll do that this afternoon.

Comment 4

[alice nodelman [:alice] [:anode]]

Any news here?

Comment 5

[Ben Kero [:bkero]]

Sorry for the delay, I needed to finish setting up the KVM cluster for you.

The machine should be available at tools-staging-master02.build.mtv1.mozilla.com.  Let me know if you have any issues with the VM, or if you needed any hardware(RAM, CPUs, drive space) added, or if you need any SSH keys put on there.

Comment 6

[alice nodelman [:alice] [:anode]]

So, we have a new staging master?

Comment 7

[matthew zeier [:mrz]]

Punting this back to Alice.  You have a root login to that box.

Comment 8

[alice nodelman [:alice] [:anode]]

Let's not punt so fast...

I'd like tools-staging-master02 to be a copy of tools-staging-master as I've already done a bunch of setup work on tools-staging-master, as have my coworkers.

Can you pull a copy of tools-staging-master and apply it to tools-staging-master02?

Comment 9

[matthew zeier [:mrz]]

No, not possible (or easy).  One's an ESX VM and the other is a KVM VM.

Comment 10

[alice nodelman [:alice] [:anode]]

What's the image on tools-staging-master02?

Comment 11

[matthew zeier [:mrz]]

Pretty sure it's a fresh CentOS install but I'll let Ben comment.

Comment 12

[Ben Kero [:bkero]]

Yep, it's a fresh centos 5.5 install.

Comment 13

[Ben Kero [:bkero]]

How should we proceed on this?  If I learn a bit about ESX I should be able to dump the image for tools-staging-master, and import it into KVM.

Comment 14

[alice nodelman [:alice] [:anode]]

If it's possible to get a copy of the current state of tools-staging-master and put it onto tools-staging-master02 that would be great, but I'm unsure what is involved.

Setting up from scratch is going to be time consuming on the auto-tools team side (upwards of a week, I bet, to get all the ducks in a row).

Comment 15

[cmtalbert]

Does this help?

http://forum.proxmox.com/threads/2218-Help-needed-VMWARE-to-Proxmox-%28KVM%29-for-a-Linux-server.

Comment 16

[Ben Kero [:bkero]]

I'm going to need to research ESX to accomplish this.  I'll be doing this anyway for another bug.  I'm first going to try to do this without any downtime, but will let you know the situation as I proceed.

Comment 17

[Phong Tran [:phong]]

(In reply to comment #13)
> How should we proceed on this?  If I learn a bit about ESX I should be able to
> dump the image for tools-staging-master, and import it into KVM.

If you can take a short downtime, I can copy the VM to another location.

Comment 18

[alice nodelman [:alice] [:anode]]

Short downtimes are a-okay if it gets the existing, configured master somewhere where it can run more efficiently.

Comment 19

[Ben Kero [:bkero]]

Phong,

did this ever get copied somewhere?

If not, I have enough ESX chops now to copy this somewhere for replication.

Comment 20

[Amy Rich [:arr] [:arich]]

I'm going through the releng ops queue and just checking to see where we stand on this bug.  Ben/Alice, have the two of you coordinated a downtime and gotten a new vm spun up?

Comment 21

[alice nodelman [:alice] [:anode]]

I believe that a downtime was attempted but was not successful...

Comment 22

[Ben Kero [:bkero]]

I can create this VM at any time, except I think this also follows the same problem as talos-addon-master1, where you'll need a new reference image.

Would you like me to deploy with the tools-staging-master reference image, or wait for a new one here too?

Comment 23

[alice nodelman [:alice] [:anode]]

This is dependent on bug 659512 - having a copy of the old tools-staging-master won't be helpful as it doesn't work with the latest releng master set up.

Comment 24

[Amy Rich [:arr] [:arich]]

Since we started over fresh with tools-staging-master02, where do we stand on this bug?

Comment 25

[alice nodelman [:alice] [:anode]]

I think that we could mark this is a WONTFIX:

- the new vm should be better able to keep up
- we are no longer linked to the releng masters that were generating the glut of sendchanges and are instead generating sendchanges locally using scripts