Closed Bug 735381 Opened 14 years ago Closed 14 years ago

instantiate new vms for linux and linux64 releng builders

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: arich, Unassigned)

References

Details

The releng BU is seeing movement, so I wanted to file this bug as a placeholder for building the new linux and linux64 builder vms on the new esx cluster in scl3. We will be replacing the hosts in sjc1 called: moz2-linux64-slaveNN where NN is 01-12 moz2-linux-slaveNN where NN is 01-50 With new vms in the build.releng.scl3.mozilla.com VLAN/IP space (exact names tbd). We will be replacing the hosts in sjc1 called: try-linux-slaveNN where NN is 01-30 try-linux64-slaveNN where NN is 01-10 With new vms in the try.releng.scl3.mozilla.com VLAN/IP space (exact names of new vms tbd). The templates for the vms we're replacing are in the "templates" folder of the build virtual datacenter on nm-ops03.mozilla.org. The template for linux hosts: CentOS-5.0-ref-tools-vm The template for linux64: Centos-5-x64-ref We'd like to start with those templates, and then make the following modifications before instantiating new vms: * make sure iptables is off * increase the /builds partition to 40G * increase the / partition to 12G (since it's at 89% full at 8G) * perform a boundary alignment on it Please also grab a copy of the buildbot-master-ref template as well since we may deploy more buildbot masters on the vmware cluster in the future.
Blocks: releng-scl3
Sounds good. How about I bring up a single instance of each VM type in private.scl3.mozilla.com so you can log in and check it out, after I finish the above mentioned changes?
This bug depends on getting the core infra in the releng.scl3.mozilla.com up before we can crank out vms (and on getting buildbot masters up to control things before we can put them into production), but front loading this with example vms on networks that are already configured sounds great to me.
Assignee: server-ops → dparsons
I've begun the process of copying the templates to scl3.
Status: NEW → ASSIGNED
Adding catlee, coop, and joduinn since I was just pointed at an earlier bug 716964 (that I wasn't cced on) which talked about different specs. So, back to the discussion of how do we resize these vms in scl3 to replace the ones in sjc1 given the hosts that we are replacing in the initial description of this bug. Are we going to go with 50 vms instead of 102? What specs do we need. How do we split them up try vs build?
It was my understanding that those questions were answered in bug 716964 -- half as many VMs with about twice the beef in each. So: 50 VMs, split evenly between 32- and 64-bit 100GB disk 2 CPU 4 GB RAM I'd prefer to give these a single disk (including /builds) rather than try to do multiple partitions. I'll leave the naming call to Amy -- we'll need to get the bitlength in there. Maybe bld-centos5-32-vm-NNN and bld-centos5-vm-NNN? Within bitlengths, they should be numbered sequentially, regardless of their eventual try/build allocation.
Okay, the last comment I saw in the referenced bug said "we'll revisit this when it's time to create the vms." Which, to me, meant it hadn't been settled yet. We'll need to know how many try vs how many build for each platform, since they will be going on different vlans.
Yeah, that comment was heading off an off-topic conversation. Catlee -- you agreed to a 50/50 split of VMs, but the split of the existing VMs is 80/20. Also, only about 8% of the 32-bit VMs are try, and none of the 64-bit are. How should we apportion the 50? 32 build ____ 32 try ____ 64 build ____ 64 try + ____ 50
OK, so the current mix is 32 build 46 32 try 34 64 build 12 64 try + 10 102 So the proportional new mix is 32 build 22 32 try 17 64 build 6 64 try + 5 50 sound good?
LGTM. We have more load on 32-bit machines because we also do android builds there. I can see switching everything over to 64-bit at some point, but that's a problem for another bug. Thanks!
:dustin & :arr, can you please confirm the following specs: Quantity of VMs: 50 Specs: all exactly the same, just based on different templates Specs: 1 virtual hard drive, 100GB. 2 partitions: /, 99GB; swap, 1GB 2 CPUs 4GB RAM Is that right? Finally, can someone tell me what the FQDNs will be?
How about we call them bld-centos5-32-vmw-001 to bld-centos5-32-vmw-039 and bld-centos5-64-vmw-001 to bld-centos5-64-vmw-011?
Are those all .build.releng.scl3.mozilla.com ?
Try has it's own vlan. bld-centos5-32-vmw-001 - bld-centos5-32-vmw-022 .build.releng.scl3.mozilla.com bld-centos5-32-vmw-023 - bld-centos5-32-vmw-039 .try.releng.scl3.mozilla.com bld-centos5-64-vmw-001 - bld-centos5-64-vmw-006 .build.releng.scl3.mozilla.com bld-centos5-64-vmw-007 - bld-centos5-64-vmw-011 .try.releng.scl3.mozilla.com build = vlan252, 10.26.52/22 try = vlan264, 10.26.64/22 The esx boxes should eventually be vlan trunked to everything but the test vlans: winbuild = vlan236, 10.26.36/22 wintry = vlan244, 10.26.44/22 srv = vlan248, 10.26.48/22 build = vlan252, 10.26.52/22 mobile = vlan260, 10.26.60/22 try = vlan264, 10.26.64/22 dmz = vlan74, 10.26.74/24 private = vlan75, 10.26.75/24 (see bug 736566)
And apologies if this is repetitive, but just to be clear, the 64-bit machines are 64-bit, and the 32-bit machines are 32-bit, so they're not all the same. As far as you're concerned try and build (within a bitlength) are the same.
Just checking back in here to see if anything is blocked on relops at this point and when you might have something ready for releng to verify.
Blocks: 739791
These hosts now all have dns entries. They do not have inventory entries since I was waiting on MAC addresses for them. If this is an issue, please let me know.
Per irc conversation with Dan, targeting next week to get one of these vms up for releng to verify.
I cribbed a script to stick stuff into inventory from rtucker and added these sans MAC and dhcp scope.
Specs are (per bug 716964) 100GB disk 2 CPU 4 GB RAM
To be clear, I should say one vm of each type, 64 bit and 32 bit.
Update from Dan. Releng now has two vms (one 64 bit, one 32 bit) that it can validate before we create all of the new vms from the same template. Dan: I've corrected the root pw to match the current one in the releng.txt file. I've copied over the 32-bit and 64-bit releng templates, updated them to the new virtual hardware, gave them 4GB RAM and 2 CPUs, and increased hard drive space to 100GB. Please test them. 64-bit template: 10.26.55.220 32-bit template: 10.26.67.220 Please let me know what you find and if anything needs to be changed. When you tell me these VMs are correct, I can clone them to the quantities specified in 735381. Notes: (1) I disabled puppet on each template because it prevented the VM from booting completely (it's looking for its old puppetmaster and it doesn't ever background itself…) (2) There is no LVM. The virtual disk has two partitions: / and swap (2GB). If this is a problem, let me know. (3) There's some sort of funny stuff going on with /home/cltbld/.ssh and /builds/slave involving bind mounts in /etc/fstab. I'm not sure if this transferred correctly between the old vmdk and the new one. Please investigate. Only the 32-bit template seems to have this.
(In reply to Amy Rich [:arich] [:arr] from comment #21) > Update from Dan. Releng now has two vms (one 64 bit, one 32 bit) that it > can validate before we create all of the new vms from the same template. > Dan: I've corrected the root pw to match the current one in the releng.txt > file. How can I access these machines? Were the hostnames posted somewhere that I missed?
The IPs for these are in comment 21.
I'm going to poke at these tonight/tomorrow. Three things I would note straight off about the Linux 32-bit ref image: 1) seems to have an older cltbld passwd 2) /builds/slave is owned by root rather than cltbld 3) scratchbox is still installed. I'll likely need to sync these VMs up with puppet before I'll be able to build anything, which may prove difficult without hostnames. We'll see. Once we're sorted here, we should probably also take a new ref image with the changes above already made to speed up the puppet-ing cycle.
I don't think we updated the VM images very often, so none of that surprises me. I added linux32-temp.try.releng.scl3.mozilla.com linux64-temp.build.releng.scl3.mozilla.com to DNS temporarily. Let me know if those don't work, or if you have trouble hooking them up to puppet.
I got both VMs to puppetize this morning. The only failures were a missing clang package (someone forgot to sync it to scl3), and the following error on both 32- and 64-bit: err: //Node[linux32-temp]/vm/Mount[builds]/ensure: change from present to mounted failed: Execution of '/bin/mount -o noatime /builds' returned 32: mount: special device /dev/sdb1 does not exist err: //Node[linux64-temp]/vm/Mount[builds]/ensure: change from present to mounted failed: Execution of '/bin/mount -o noatime /builds' returned 32: mount: special device /dev/sdb1 does not exist Should that device exist on these VMs, or do we need a new puppet class? If we can puppetize, I'm pretty sure we can build, but I'll kick off a build on each VM for good measure.
It's all one big disk on these boxes, no separate /builds. I also added root's authorized_keys to these hosts.
(In reply to Chris Cooper [:coop] from comment #26) > If we can puppetize, I'm pretty sure we can build, but I'll kick off a build > on each VM for good measure. Builds ran fine. I'll file a follow-up bug to make sure we have a new builder VM class for scl3.
(In reply to Chris Cooper [:coop] from comment #28) > Builds ran fine. I'll file a follow-up bug to make sure we have a new > builder VM class for scl3. Bug 744067.
Dan: the vms you created have been modified and are ready to be turned back into a template and then have all of the new vms created (per comment 13). If you get me a list of hostnames and mac addresses, I can update inventory and get them all booted up.
As a followup, this is high priority for cshields (as well as releng) because he needs us to consolidate onto fewer blades in sjc1 so he can move chassis to scl3.
OK, shutting down the VMs now for cloning.
All the VMs have been created. I emailed the MAC addresses to arr as requested for inventory / DHCP setup.
The clones, um, well they didn't clone well. Fixing now, will update asap, likely Friday.
New clones built, they're happy now. MAC addresses emailed to Amy.
I've added the mac and scope info for each of these and powered them on. I noticed that the ref hostname was set in /etc/sysconfig/network, so I removed that line and rebooted them, and they've all come up with their proper hostnames now (via dhcp). releng, please verify that the hosts look good (and optionally set the hostname with puppet after the fact if you must). I'm going to change the two templates to use dhcp as well so no manual hostname changing needs to take place when bringing up a new vm (just like we do on kvm).
Assignee: dparsons → nobody
Component: Server Operations: Virtualization → Release Engineering
QA Contact: dparsons → release
This is fixed from our standpoint. Releng work continues in bug 744067.
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.