389426 - upgrade community Linux machines to the new ref platform

Status: RESOLVED FIXED

(Webtools Graveyard :: Tinderbox, defect, P2)

Description

[Robert Kaiser]

The community tinderbox Linux machines for SeaMonkey and Sunbird trunk builds should probably be upgraded to / replaced with the new ref platform

Comment 1

[Chris Cooper [:coop] (he/him)]

(copied from bug 392438)

from offline discussion:

- we'll bring up a new VM on our internal build network and make sure community
builds work on it
- shutdown existing community VM
- copy (not VMmotion) the new VM to the community build machine
- start new VM on community build machine.

Some discussion about how much of this should be done by build group rather
then by community members.

Comment 2

[J. Paul Reed [:preed]]

This is blocked on new hardware for another community ESX box, as the current one is full (disk space).

Comment 3

[Stuart Parmenter]

this is blocking 362682 which needs to be landed asap.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Per discussion with stuart, some of the VMs on this host which need updating are:

Linux cb-sea-linux-tbox Dep release
Linux cb-sb-linux-tbox Dep Lt-Release
Linux cb-sb-linux-tbox Dep Sb-Release
Linux cb-sb-linux-tbox Clbr Sb-Trunk-l10n

Comment 5

[J. Paul Reed [:preed]]

I looked on this host, and while the machine is almost maxed out on CPU/memory, it still does have some drive space left, it looks like.

There should be enough space to put the new ref VMs and get them up and running. CPU and memory (especially) is gonna be a bit tight, but if we're shutting down the old ref VMs, it should only be tight for a day or two.

mrz ordered new hardware, but I don't think this is blocked on that hardware order anymore.

Comment 6

[Daniel Boelzle [:dbo]]

When is this going to happen?

Comment 7

[J. Paul Reed [:preed]]

(In reply to comment #6)
> When is this going to happen?

I'm thinking I'll get the new machines created today, but we let the run for 1-2 days first to make sure everything's OK. It's taken about 1-2 days for the machines in the build network, so I'd guess it'd take that long to do these, but may run into problems with the cb network.

Stay tuned!

Comment 8

[J. Paul Reed [:preed]]

Attachment: feature_newref branch config changes for Seamonkey

Comment 9

[Robert Kaiser]

Comment on attachment 282423 [details] [diff] [review]
feature_newref branch config changes for Seamonkey

>Index: tinder-config.pl
>===================================================================
> # Extra build name, if needed.
>-$BuildNameExtra = 'release';
>+$BuildNameExtra = 'Newref';

We could even name it "Nightly" right away, as that should be what we have in the future and we don't need to rename it another time then.
But r=me either way.

Comment 10

[J. Paul Reed [:preed]]

Both of these are reporting into their respective Tinderbox pages:

http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey

and

http://tinderbox.mozilla.org/showbuilds.cgi?tree=Sunbird

They're building right now, but I'm not 100% sure they'll complete successfully yet... might have to go a couple of rounds before we get to green.

I'll be keeping an eye on them.

Comment 11

[J. Paul Reed [:preed]]

So, both of these completed their builds just fine; I'm gonna coordinate with coop to hand them over to the community project reps.

For now, though, this shouldn't block bug 362682 anymore, since there's coverage on Seamonkey and Sunbird, for now.

(If we don't get them handed over quickly after this lands, we can turn update and symbol push back on for configs, so those keep chugging along.)

Lowing severity appropriately.

Comment 12

[Martin Schröder [:mschroeder]]

Lightning should also be brought up on the new VM. Sunbird and Lightning always lived on the same VM afaik.

Comment 13

[J. Paul Reed [:preed]]

(In reply to comment #12)
> Lightning should also be brought up on the new VM. Sunbird and Lightning always
> lived on the same VM afaik.

Lightning builds now reporting into the Sunbird Tinderbox page, from the same VM.

Comment 14

[Robert Kaiser]

The Sunbird L10n repack process should probably also move to sb-newref-linux-tbox - though after bug 397285 you could even perform that in the same cycle that also does the original builds, if wanted.

Comment 15

[J. Paul Reed [:preed]]

Over to coop to scrub the machines, get them into the CB network, and do the handoff, a la "Meet in the train station in Madrid at midnight; I'll be the guy with the briefcase..."

Comment 16

[Chris Cooper [:coop] (he/him)]

I've scrubbed the passwords on the box and shutdown the tinderboxes. Re-assigning to server-ops to get the vmfs files moved over to the community VM host (I'd do it myself, but this involves crossing the firewall). The VMs are currently on bm-vmware05, and the vmfs files are on netapp-d-002 (sbnr-linux-tbox and seanr-linux-tbox). I just need the VMs moved -> I can bring them back up once they are moved. BTW, it seems that sbnr-linux-tbox ended up as read-only

ause/kairo: once moved, do you need the newref VMS up ASAP as replacements or would you prefer a defined outage window?

Comment 17

[Robert Kaiser]

As long as stuart doesn't re-land bug 362682, we are not depending on the newref for SeaMonkey, so you can plan doing this when it's convenient for you.

Comment 18

[Dave Miller [:justdave]]

These two VMs are now in the process of moving to the cb-vmware01 server.  I'll post here again once they're finished. (Looks like it's going to take a while)

Comment 19

[Dave Miller [:justdave]]

OK, moves completed.  Over to coop.

Comment 20

[Chris Cooper [:coop] (he/him)]

Oops, we'll still need community network DHCP updates for the new MAC addr for these two boxes:

cb-sb-linux-tbox  - 00:50:56:80:09:4C
cb-sea-linux-tbox - 00:50:56:80:64:72

Comment 21

[Dave Miller [:justdave]]

As best I can tell, the cb network doesn't have any DHCP, the machines that are there now appear to have static IP assignments.

Comment 22

[Chris Cooper [:coop] (he/him)]

OK, I'm in the process of changing the config scripts on the newref VMs to assume the IPs from the old ones. If there is anything that IT needs to do on their end for this, please let me know before I go too far.

Comment 23

[Chris Cooper [:coop] (he/him)]

I'm conceding defeat on this for tonight. 

Static networking is still not working on the newref boxes, so I've restarted tinderbox on the old VMs for now. If anyone from IT has any insight on how static IPs are supposed to work on the community network and wants to take a look in the interim, be my guest.

Comment 24

[Chris Cooper [:coop] (he/him)]

mrz gave me some pointers on my ifcfg, but I was still unable to get it to work. Passing back to IT.

Comment 25

[Dave Miller [:justdave]]

Sorry, I was still working on this and not checking my email frequently enough to notice you grabbed it again.  They need new IPs assigned from the community-build network's IP range, which I should have shortly.

Comment 26

[Dave Miller [:justdave]]

ok, your new IPs are:

63.245.210.22 cb-sea-newref-linux-tbox
63.245.210.23 cb-sb-newref-linux-tbox

The old IPs probably would have worked in hindsight except that the VMs were plugged into the wrong virtual switch.

Your ifcfg-eth0 should look something like this:

DEVICE=eth0
BOOTPROTO=static
HWADDR=00:50:56:80:09:4C
IPADDR=63.245.210.23
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=Ethernet

As long as you're taking the old ones offline you can reuse those IPs if you want, let me know so I can take the new ones back out if you do.  The new IPs will let you get both the old and new online at the same time if you need them though.

Comment 27

[Chris Cooper [:coop] (he/him)]

I can't get the new IPs to work, and I've tried with the suggested NETMASK of 255.255.255.0 and the previous NETMASK of 255.255.255.224.

Can someone from IT please log into these new VMs and get the networking setup correctly? Going back and forth via the bug is not the most efficient way to get these machines up.

Comment 28

[matthew zeier [:mrz]]

Can't figure out root login - ping'd coop, waiting for him.

Comment 29

[matthew zeier [:mrz]]

Addresses work, /etc/resolv.conf had internal DNS and was probably causing some problems.  I can get to yahoo.com from both.

Comment 30

[Robert Kaiser]

I think the bug report itself should only be marked FIXED once the new tinderboxen work correctly and the respective community members have access to them. ;-)

Comment 31

[Robert Kaiser]

back to coop is probably best :)

Comment 32

[Chris Cooper [:coop] (he/him)]

Checked in the config changes, updating them on the VMs now.

Comment 33

[Chris Cooper [:coop] (he/him)]

Tinderbox mail is not making it out from these boxes. 

Are there mail settings that need to change too now that these VMs are in the community network?

Comment 34

[matthew zeier [:mrz]]

Probably - where are they trying to send mail to?

Comment 35

[Chris Cooper [:coop] (he/him)]

The returned mail was headed to [e-mail]

Comment 36

[matthew zeier [:mrz]]

Added:

access-list sandbox-outbound line 31 permit tcp object-group try-servers  host  10.2.74.14 eq 25

Comment 37

[Chris Cooper [:coop] (he/him)]

Still getting the following in returned mail on the two VMs:

550 5.1.2 <[e-mail]>... Host unknown (Name ser
ver: mail.build.mozilla.org: host not found)

Comment 38

[Dave Miller [:justdave]]

can you forward me the entire bounce please?

Comment 39

[Chris Cooper [:coop] (he/him)]

Working now...mail.build.mozilla.org is an alias to smtp.mozilla.org that doesn't resolve on the community network.

Comment 40

[Robert Kaiser]

verified for the SeaMonkey part, looks like the machine is running correctly and I can log into it via the jumphost.