568268 - re-image 20 try talos minis as try-mac builders and move out of sandbox and into build network

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

The following slaves need to be re-imaged/re-created if VMs as the current builder platform and put into the build network - please update nagios as well.

xp/vista --> win32:

qm-pxp-try{01-10}.m.o  --> try-win32-slave{30-40}.build.m.o
qm-pvista-try{01-11}.m.o --> try-win32-slave{41-52}.build.m.o

tiger/leopard --> mac (10.5 ref image):

qm-ptiger-try{01,02}.m.o --> try-mac-slave{20,21}.build.m.o
qm-pleopard-try{01-16}.m.o --> try-mac-slave{22-38}.build.m.o

ubuntu --> linux:

qm-pubuntu-try{01-16}.m.o --> try-linux-slave{20-36}.build.m.o

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This seems a little wonky to me, so hold off on starting for now.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Ok - so after consulting with nthomas, we're just going to take 20 of these to turn into mac builders as a temporary measure while we work on getting talos-r3 accepting try unittests.

Please re-image 20 of the above boxes with the 10.5 ref platform -> try-mac-slave{20-40}.build


The rest should still be pulled offline and put aside somewhere as we will be figuring out what to do with them.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Any ETA on these mac images?  The mac builds on try are backing up when we get up to a full load during the day.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> Any ETA on these mac images?  The mac builds on try are backing up when we get
> up to a full load during the day.

Phong: how come these werent done in scheduled colo trips on friday (28th) and tues (1st)? Can this be done in tomorrow's colo trip?


Raising priority as this is blocking TryServer wait times.

Comment 5

[Phong Tran [:phong]]

We don't have enough storage for more win32 VM slaves.  I've already created 4 more linux VMs in another bug.

Comment 6

[Phong Tran [:phong]]

I was in PHX last week.  Justin was out most of last week and then I was also on-call.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #5)
> We don't have enough storage for more win32 VM slaves.  I've already created 4
> more linux VMs in another bug.

I'll change the summary if it's unclear.  Right now we're only talking about getting mac minis re-imaged as try-mac slaves.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

fyi: qm-pleopard-try{12-16} and qm-pubuntu-try{12-16} are here in 650castro in the QA lab. 

All these are offline and can be reimaged at any time.

Comment 9

[Justin Lazaro [:jlaz] (use needinfo)]

qm-pleopard-try(12-16) and qm-pubuntu-try(12-15) re-imaged with 10.5 image.  phong/jdow will bring the rest back from MPT tomorrow.

Comment 10

[Justin Dow [:jabba]]

(In reply to comment #2)


> 
> The rest should still be pulled offline and put aside somewhere as we will be
> figuring out what to do with them.

All of them have been pulled offline. We're bringing them back to castro to finish up the other ten try-mac-slaves, and will await instructions for the other ~35 minis.

Comment 11

[Armen [:armenzg]]

Lukas and John, on bug 563479 I have added 10 10.6 builders (re-purposed from rev2 machines). Right now the unit tests are disabled for them and therefore the load is small but I would like to see that we have the same amount of 10.5 and 10.6 try slaves. Can we clone some 10.6 slaves out of these ~35 minis that jabba mentions?

Comment 12

[Justin Dow [:jabba]]

The twenty try-mac-slaves (20-39) are all online. I don't have an exact count on how many are remaining right now. When I get back to the office, I'll open a new tracking bug for what to do with those.

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Was there anything different or unusual about the re-imaging of these machines?  Did the image (current 10.5) ref image apply cleanly?  

https://bugzilla.mozilla.org/show_bug.cgi?id=570445#c1 outlines how these newly imaged macs are not acting like our usual mac images.

Comment 14

[Justin Dow [:jabba]]

The image applied cleanly. We only have one image for 10.5 builders, so I don't think we could have used the wrong image. The only thing I can think of is they didn't get the resistors on them. Is that needed for builders? I can go plug resistors into slaves 20-29 to see if that helps.

Comment 15

[Justin Dow [:jabba]]

The DeployStudio software seems to reset the Bluetooth preferences, so an extra step is required when deploying images going forward. I've documented the step in https://wiki.mozilla.org/PostImage .

Lukas fixed the 20 minis in this bug.