1299508 - Please rename and reimage the Win 7 machines moved to Win XP & Win 8 pools

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Alin Selagea [:aselagea]]

On bug 1299468 there is work to move 80 Windows 7 machines to the Windows XP pool and another 31 Windows 7 machines to Windows 8 in order to better deal with the current backlog.

We'd need to updated nagios and inventory and to reimage these machines.

Comment 1

[Alin Selagea [:aselagea]]

Here's a quick sketch:

t-w732-ix-151
t-w732-ix-152
t-w732-ix-153
...
...
t-w732-ix-229
t-w732-ix-230

will become:

t-xp32-ix-143
t-xp32-ix-144
t-xp32-ix-145
...
...
t-xp32-ix-221
t-xp32-ix-222


and 

t-w732-ix-231
t-w732-ix-232
t-w732-ix-233
..
..
t-w732-ix-260
t-w732-ix-261

will become

t-w864-ix-205
t-w864-ix-206
t-w864-ix-207
..
..
t-w864-ix-234
t-w864-ix-235

Comment 2

[Amy Rich [:arr] [:arich]]

Q: can you tell me which have SOL so we can switch the video cards back for XP? That will help determine which machines we turn into XP nodes.

I also assume that it's fine to have the onboard card off for w8?

Comment 3

[Q]

I can check. I think there may be a way to check via ipmi, if not it can be done with ssh scripts. 

Win 8 works the same way as XP and should normally have the card on but I haven't tested to see if it will compensate with the onboard set to off.

Comment 4

[Amy Rich [:arr] [:arich]]

Inventory and nagios have been updated. I needed to remove the DHCP file stuff for the w8 machines by hand through the gui, because I couldn't see a good way to do that via invtool.

Waiting for DNS/DHCP to catch up before I try to reimage any machines.

Comment 5

[Amy Rich [:arr] [:arich]]

I've kicked off a reimage on  t-w864-ix-205 - t-w864-ix-215

All of the hosts have had their bootdev set to pxe, so a reimage should just be a matter of rebooting them in batches of 10 at this point.

On Tuesday, Q is going to look into enabling SOL for them and switching the onboard graphics machines back on for the XP machines.

Comment 6

[Amy Rich [:arr] [:arich]]

I've now kicked off t-w864-ix-216 - t-w864-ix-225 as well. That leaves t-w864-ix-226 - t-w864-ix-235 and all of the xp machines.

I'm ccing all of the buildduty folks so they can pick up with this on Monday while the US and Canada are out on holiday. When you kick off the reimages, leave about 20-30 minutes in between batches of 10 so you don't overload the imaging server.

Once reimaged, the XP machines still need the graphics work mentioned above, but I think that the w8 machines should be okay as is (please confirm resolution before putting them into prod).

Comment 7

[Amy Rich [:arr] [:arich]]

I've also kicked off t-w864-ix-226 - t-w864-ix-235 and t-w732-ix-143 - t-w732-ix-154.

Comment 8

[Amy Rich [:arr] [:arich]]

Since I was around, I kicked off reimages for all of them, but I haven't checked that all of the reimages worked, so that's where buildduty should likely pick up tomorrow (with the knowledge that the XP machines still need the onboard graphics enabled, so they'll be at the wrong res)).

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Some machines seems to be changed but some of them have problems and are still seen as windows7 or windows8 when they should be Windows XP or are unreachable.

Those that are changed and are OK : t-xp32-ix-174 - t-xp32-ix-222 (from windows XP) and T-w864-ix-(212,218,219,220,225,226,228,229,231,232,234 and 235) fron Windows 8

We also have some unraechable machines:                                                                t-xp32-ix-(144,145,146,147,148,149,151,152,154,155,159,160,161,162,165,168)
t-w864-ix-(206,207,208,208,209,210,211,213,214,216,217,221,222,223,224,227,230)

And some machines that should have been changed but are not or are wrongly changed:

t-xp32-ix-143 is seen as w732-ix-231 
t-xp32-ix-150 is seen as t-w864-ix-212
t-xp32-ix-153 is seen as t-w732-ix-241
t-xp32-ix-156  seen as t-w864-ix-218
t-xp32-ix-157 seen as t-w864-ix-219
t-xp32-ix-158 seen as t-w864-ix-220
t-xp32-ix-163 seen as t-w864-ix-225
t-xp32-ix-164 seen as t-w864-ix-226
t-xp32-ix-166 seen as t-w864-ix-228
t-xp32-ix-167 seen as t-w864-ix-229
t-xp32-ix-169 seen as t-w864-ix-231
t-xp32-ix-170 seen as t-w864-ix-232
t-xp32-ix-171 seen as t-w732-ix-259
t-xp32-ix-172 seen as t-w864-ix-234
t-xp32-ix-173 seen as t-w864-ix-235


t-w864-ix-205 is seen as t-w732-ix-231
t-w864-ix-215 is seen as t-w732-ix-241
T-w864-ix-233 is seen as t-w732-ix-259

Please check these machines with problems and see if the problem can be solve.
Thank you

Comment 10

[Amy Rich [:arr] [:arich]]

andrei: please try reimaging those that failed and NI Q if problems persist.

Comment 11

[Alin Selagea [:aselagea]]

Attachment: win8_status.txt

I checked the Windows 8 machines (205-235) and things don't look too good. Most of them are unreachable and trying to reboot/re-image them does nothing. Only some of them managed to install the correct OS, but they are running at a small resolution as they're not using the dedicated video card (see attachment).

Taking a look at windamin server, I noticed the following:
    - /machines/windows/7/SCL3/tester/ contains 281 machines (so the 111 ones from this bug and another 20 from bug 1297173 are still there)
    - /machines/windows/8/SCL3/tester/ has 14 T-XP32-IX entries, also some other entries seem to be missing
    - /machines/windows/XP/SCL3/tester/ has T-W864-IX entries
Should we worry about these?

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Also on windows XP after re-imaging some machines ,no machine was fixed  and the bellow list need to be investigate by RelOps team:

t-xp32-ix-173 --> seen as t-w864-ix-235
t-xp32-ix-172 --> seen as t-w864-ix-234
t-xp32-ix-171 --> unreachable
t-xp32-ix-170 --> seen as t-w864-ix-232
t-xp32-ix-169 --> seen as t-w864-ix-231
t-xp32-ix-168 --> unreachable
t-xp32-ix-167 --> unreachable
t-xp32-ix-166 --> unreachable
t-xp32-ix-165 --> unreachable
t-xp32-ix-164 --> unreachable
t-xp32-ix-163 --> unreachable
t-xp32-ix-162 --> unreachable
t-xp32-ix-161 --> unreachable
t-xp32-ix-160 --> unreachable
t-xp32-ix-159 --> unreachable
t-xp32-ix-158 --> unreachable
t-xp32-ix-157 --> unreachable
t-xp32-ix-156 --> seen as t-w864-ix-218
t-xp32-ix-155 --> unreachable
t-xp32-ix-154 --> unreachable
t-xp32-ix-153 --> seen as t-w864-ix-215
t-xp32-ix-152 --> unreachable
t-xp32-ix-151 --> unreachable
t-xp32-ix-150 --> unreachable
t-xp32-ix-149 --> unreachable
t-xp32-ix-148 --> unreachable
t-xp32-ix-147 --> unreachable
t-xp32-ix-146 --> unreachable
t-xp32-ix-145 --> unreachable
t-xp32-ix-144 --> seen as t-w732-ix-232
t-xp32-ix-143 --> unreachable

Comment 13

[Amy Rich [:arr] [:arich]]

I'm not sure why t-w864-ix-206 had an IP address conflict (the info was correct in the spreadsheet), but I've fixed that, at least.

Comment 14

[Mark Cornmesser [:markco]]

It looks like t-xp32-ix-173 and t-xp32-ix-172 are now coming up as xp machines. However, they may need to have their on board graphic cards reenabled. The fakemon vbs doesn't seem to be working.

Comment 15

[Q]

As covered in other bugs Fakemon for xp requires the onboard to be on due to static ids for monitors. It can be rewritten with some more logic or we can make the onboard primary in the bios ( it isn't really disabled)

Comment 16

[Mark Cornmesser [:markco]]

I am currently looking at t-xp32-ix-171. The management ip is pingable and I am amble to send ipmi commands to it, but it doesn't appear to be installing. In the previous 2 I was able to monitor the installation through the kvm console, but with this one there is no change to the console.

Comment 17

[Mark Cornmesser [:markco]]

I am seeing an Ip conflict on t-xp32-ix-168 and it is coming up as t-w732-ix-256.

Comment 18

[Amy Rich [:arr] [:arich]]

I wonder if inventory somehow screwed up in the translation... https://docs.google.com/spreadsheets/d/1I38-lTHwaxjS1pPjLxmzCM9cTQuFw9bkuN-ADsKEIiE/edit#gid=1881968051 (last two tabs) has the correct I information, so feel free to go fix anything that's out of place.

Comment 19

[Mark Cornmesser [:markco]]

Yeah, I think it is inventory. I was working with and after the correcting and recorrecting, after mixing up a few numbers. 

I will go through and make sure the xp inventory entries looks like the spreadsheet. After that I will reinstall the issued xp machines.

Comment 20

[Mark Cornmesser [:markco]]

Updated all the xp inventory entries and reinstalled a few. I am going to start kicking off groups of 10 installs.

Comment 21

[Amy Rich [:arr] [:arich]]

(In reply to Amy Rich [:arr] [:arich] from comment #18)
It looks like the inventory scripts did, in fact not do the right thing. Unfortunately inventory is really no longer supported by IT, so can't really file a bug for that. Need to make sure the w8 hosts were not similarly affected (they probably were).

Comment 22

[Mark Cornmesser [:markco]]

Out of the 80 xp machines I am down to 4 that still have some type of inventory/ip address issue: 

151 
154
159
190

The rest are up, but they still the need the graphic cards bios changes made.

Comment 23

[Mark Cornmesser [:markco]]

I am reinstalling the above mentioned 4. The install may had been kicked off too closely to inventory entries being updated and DNS did not catch up.

Comment 24

[Mark Cornmesser [:markco]]

The above mentioned 4 are now up and good.

Comment 25

[Amy Rich [:arr] [:arich]]

Needs Q to modify the graphics cards.

Comment 26

[Mark Cornmesser [:markco]]

Updated the Win 8 inventory entries and re-installed. All but 134 has come back up. I will need to the onboard card set as default before i can trouble shoot it further. In addtion the other machines may need the onboard graphics card set to default because the fake mon vbs seems to be running but the resolution is not being set correctly. It is also possible this set of machine are not doing a full Group Policy update because a majority of machines are coming up with a message about Metro being updated. I have verified that they have come up in the correct OU.

Comment 27

[Q]

All xp machines are updated with bios settings for the video cards corrected and SOL enabled. Resolutions looks correct and the y should be good to add into the pool.

The win 8 machines had a few hangs during recon fig but they should be done tomorrow.

Comment 28

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Few xp machines still have some problems:

t-xp32-ix-143 --> unreachable
t-xp32-ix-153 --> unreachable
t-xp32-ix-189 --> unreachable
t-xp32-ix-197 --> unreachable
t-xp32-ix-208 --> unreachable
t-xp32-ix-222 --> bad resolution 1024 x 768

Comment 29

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

update:

t-xp32-ix-153 --> bad resolution 1024 x 768
t-xp32-ix-189 --> bad resolution 1024 x 768
t-xp32-ix-197 --> bad resolution 1024 x 768
t-xp32-ix-208 --> bad resolution 1024 x 768
t-xp32-ix-222 --> bad resolution 1024 x 768

Comment 30

[Q]

I missed 222 by accident.

Rebooted the below and resolution is now correct:
t-xp32-ix-153 
t-xp32-ix-189 
t-xp32-ix-197
t-xp32-ix-208

Comment 31

[Alin Selagea [:aselagea]]

I checked all t-w864-ix-[205-235] hosts and they all look good. t-xp32-ix-153 needed a reimage as it was getting stuck in a loop when booting the OS. I'll go on and mark this bug as resolved, further isolated issues can be treated on the individual problem tracking bugs.

Thanks everyone for your work!