705352 - Repurpose Rev3 Snowleopard machines for other testing silos

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Armen [:armenzg]]

Please take all talos-r3-snow-XXX machines (except the ref one) and repurpose them as other testing slaves.

I want to add these many slaves to each testing silo:
talos-r3-fed	 13
talos-r3-fed64	 12
talos-r3-leopard  7
talos-r3-w7	 14
talos-r3-xp	 13

For the record we need to rename the old values for (I hope I'm not missing any):
- DNS
- inventory
- nagios

We will fix the releng side of this in bug 683734.

Please keep talos-r3-snow-ref around as we never know who might want to ask to get a re-image (unless we can take a snapshot and save it somewhere else).

Here is the list of new slaves (I hope I got it right):
01- talos-r3-fed-064
02- talos-r3-fed-065
03- talos-r3-fed-066
04- talos-r3-fed-067
05- talos-r3-fed-068
06- talos-r3-fed-069
07- talos-r3-fed-070
08- talos-r3-fed-071
09- talos-r3-fed-072
10- talos-r3-fed-073
11- talos-r3-fed-074
12- talos-r3-fed-075
13- talos-r3-fed-076
01- talos-r3-fed64-060
02- talos-r3-fed64-061
03- talos-r3-fed64-062
04- talos-r3-fed64-063
05- talos-r3-fed64-064
06- talos-r3-fed64-065
07- talos-r3-fed64-066
08- talos-r3-fed64-067
09- talos-r3-fed64-068
10- talos-r3-fed64-069
11- talos-r3-fed64-070
12- talos-r3-fed64-071
01- talos-r3-leopard-060
02- talos-r3-leopard-061
03- talos-r3-leopard-062
04- talos-r3-leopard-063
05- talos-r3-leopard-064
06- talos-r3-leopard-065
07- talos-r3-leopard-066
01- talos-r3-w7-066
02- talos-r3-w7-067
03- talos-r3-w7-068
04- talos-r3-w7-069
05- talos-r3-w7-070
06- talos-r3-w7-071
07- talos-r3-w7-072
08- talos-r3-w7-073
09- talos-r3-w7-074
10- talos-r3-w7-075
11- talos-r3-w7-076
12- talos-r3-w7-077
13- talos-r3-w7-078
14- talos-r3-w7-079
01- talos-r3-xp-063
02- talos-r3-xp-064
03- talos-r3-xp-065
04- talos-r3-xp-066
05- talos-r3-xp-067
06- talos-r3-xp-068
07- talos-r3-xp-069
08- talos-r3-xp-070
09- talos-r3-xp-071
10- talos-r3-xp-072
11- talos-r3-xp-073
12- talos-r3-xp-074
13- talos-r3-xp-075

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the correspondance of old to new machines

talos-r3-snow-001       talos-r3-fed-064
talos-r3-snow-002       talos-r3-fed-065
talos-r3-snow-003       talos-r3-fed-066
talos-r3-snow-004       talos-r3-fed-067
talos-r3-snow-005       talos-r3-fed-068
talos-r3-snow-006       talos-r3-fed-069
talos-r3-snow-007       talos-r3-fed-070
talos-r3-snow-008       talos-r3-fed-071
talos-r3-snow-009       talos-r3-fed-072
talos-r3-snow-010       talos-r3-fed-073
talos-r3-snow-011       talos-r3-fed-074
talos-r3-snow-012       talos-r3-fed-075
talos-r3-snow-013       talos-r3-fed-076
talos-r3-snow-014       talos-r3-fed64-060
talos-r3-snow-015       talos-r3-fed64-061
talos-r3-snow-016       talos-r3-fed64-062
talos-r3-snow-017       talos-r3-fed64-063
talos-r3-snow-018       talos-r3-fed64-064
talos-r3-snow-019       talos-r3-fed64-065
talos-r3-snow-020       talos-r3-fed64-066
talos-r3-snow-021       talos-r3-fed64-067
talos-r3-snow-022       talos-r3-fed64-068
talos-r3-snow-023       talos-r3-fed64-069
talos-r3-snow-024       talos-r3-fed64-070
talos-r3-snow-025       talos-r3-fed64-071
talos-r3-snow-026       talos-r3-leopard-060
talos-r3-snow-027       talos-r3-leopard-061
talos-r3-snow-028       talos-r3-leopard-062
talos-r3-snow-029       talos-r3-leopard-063
talos-r3-snow-030       talos-r3-leopard-064
talos-r3-snow-031       talos-r3-leopard-065
talos-r3-snow-032       talos-r3-leopard-066
talos-r3-snow-033       talos-r3-w7-066
talos-r3-snow-034       talos-r3-w7-067
talos-r3-snow-035       talos-r3-w7-068
talos-r3-snow-036       talos-r3-w7-069
talos-r3-snow-037       talos-r3-w7-070
talos-r3-snow-038       talos-r3-w7-071
talos-r3-snow-039       talos-r3-w7-072
talos-r3-snow-040       talos-r3-w7-073
talos-r3-snow-041       talos-r3-w7-074
talos-r3-snow-042       talos-r3-w7-075
talos-r3-snow-043       talos-r3-w7-076
talos-r3-snow-044       talos-r3-w7-077
talos-r3-snow-045       talos-r3-w7-078
talos-r3-snow-046       talos-r3-w7-079
talos-r3-snow-047       talos-r3-xp-063
talos-r3-snow-048       talos-r3-xp-064
talos-r3-snow-049       talos-r3-xp-065
talos-r3-snow-050       talos-r3-xp-066
talos-r3-snow-051       talos-r3-xp-067
talos-r3-snow-052       talos-r3-xp-068
talos-r3-snow-053       talos-r3-xp-069
talos-r3-snow-054       talos-r3-xp-070
talos-r3-snow-055       talos-r3-xp-071
talos-r3-snow-056       talos-r3-xp-072
talos-r3-snow-057       talos-r3-xp-073
talos-r3-snow-058       talos-r3-xp-074
talos-r3-snow-059       talos-r3-xp-075

This will need to be updated in:
 * DNS
 * DHCP
 * Inventory
 * Nagios
 * DeployStudio
at which point I'll hand this over to Jake for some re-imaging and labeling fun in scl1.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

in sed-script format, since I know I'll need this :)

s/talos-r3-snow-001/talos-r3-fed-064/g
s/talos-r3-snow-002/talos-r3-fed-065/g
s/talos-r3-snow-003/talos-r3-fed-066/g
s/talos-r3-snow-004/talos-r3-fed-067/g
s/talos-r3-snow-005/talos-r3-fed-068/g
s/talos-r3-snow-006/talos-r3-fed-069/g
s/talos-r3-snow-007/talos-r3-fed-070/g
s/talos-r3-snow-008/talos-r3-fed-071/g
s/talos-r3-snow-009/talos-r3-fed-072/g
s/talos-r3-snow-010/talos-r3-fed-073/g
s/talos-r3-snow-011/talos-r3-fed-074/g
s/talos-r3-snow-012/talos-r3-fed-075/g
s/talos-r3-snow-013/talos-r3-fed-076/g
s/talos-r3-snow-014/talos-r3-fed64-060/g
s/talos-r3-snow-015/talos-r3-fed64-061/g
s/talos-r3-snow-016/talos-r3-fed64-062/g
s/talos-r3-snow-017/talos-r3-fed64-063/g
s/talos-r3-snow-018/talos-r3-fed64-064/g
s/talos-r3-snow-019/talos-r3-fed64-065/g
s/talos-r3-snow-020/talos-r3-fed64-066/g
s/talos-r3-snow-021/talos-r3-fed64-067/g
s/talos-r3-snow-022/talos-r3-fed64-068/g
s/talos-r3-snow-023/talos-r3-fed64-069/g
s/talos-r3-snow-024/talos-r3-fed64-070/g
s/talos-r3-snow-025/talos-r3-fed64-071/g
s/talos-r3-snow-026/talos-r3-leopard-060/g
s/talos-r3-snow-027/talos-r3-leopard-061/g
s/talos-r3-snow-028/talos-r3-leopard-062/g
s/talos-r3-snow-029/talos-r3-leopard-063/g
s/talos-r3-snow-030/talos-r3-leopard-064/g
s/talos-r3-snow-031/talos-r3-leopard-065/g
s/talos-r3-snow-032/talos-r3-leopard-066/g
s/talos-r3-snow-033/talos-r3-w7-066/g
s/talos-r3-snow-034/talos-r3-w7-067/g
s/talos-r3-snow-035/talos-r3-w7-068/g
s/talos-r3-snow-036/talos-r3-w7-069/g
s/talos-r3-snow-037/talos-r3-w7-070/g
s/talos-r3-snow-038/talos-r3-w7-071/g
s/talos-r3-snow-039/talos-r3-w7-072/g
s/talos-r3-snow-040/talos-r3-w7-073/g
s/talos-r3-snow-041/talos-r3-w7-074/g
s/talos-r3-snow-042/talos-r3-w7-075/g
s/talos-r3-snow-043/talos-r3-w7-076/g
s/talos-r3-snow-044/talos-r3-w7-077/g
s/talos-r3-snow-045/talos-r3-w7-078/g
s/talos-r3-snow-046/talos-r3-w7-079/g
s/talos-r3-snow-047/talos-r3-xp-063/g
s/talos-r3-snow-048/talos-r3-xp-064/g
s/talos-r3-snow-049/talos-r3-xp-065/g
s/talos-r3-snow-050/talos-r3-xp-066/g
s/talos-r3-snow-051/talos-r3-xp-067/g
s/talos-r3-snow-052/talos-r3-xp-068/g
s/talos-r3-snow-053/talos-r3-xp-069/g
s/talos-r3-snow-054/talos-r3-xp-070/g
s/talos-r3-snow-055/talos-r3-xp-071/g
s/talos-r3-snow-056/talos-r3-xp-072/g
s/talos-r3-snow-057/talos-r3-xp-073/g
s/talos-r3-snow-058/talos-r3-xp-074/g
s/talos-r3-snow-059/talos-r3-xp-075/g

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This will need to be updated in:
 * DNS - DONE
 * DHCP - DONE
 * Inventory - DONE
 * Nagios - DONE
 * DeployStudio - need pointers from arr on how to do this.

Comment 4

[Amy Rich [:arr] [:arich]]

All of the non-fedora minis have been reimaged (but still need to be physically re-lableed).  The fedora (fed and fed64) minis will require someone to be physically in the datacenter to reimage since they use clonezilla and not deploystudio.

As far as keeping the ref machine, do we need to keep the physical machine, or is having the DS image around good enough, Armen?

Comment 5

[Amy Rich [:arr] [:arich]]

After talking with Armen, we don'tneed to keep the physical host as long as we keep the last r3-snow ds image.  I recommend that we use the ref machine as talos-r3-xp-067 since talos-r3-snow-051 has a dead logic board and is not cost effective to fix.  This will require moving the ref machine from mtv1 to scl1.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

bug 706601 filed for that purpose

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I'll get a CSV containing all of the renaming info, including rack position, and then bump this to jake.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the requested CSV - I assume this will look purty in open office or something.

old,new,system_rack,rack_order
talos-r3-snow-001,talos-r3-fed-064,101-07,1.01
talos-r3-snow-002,talos-r3-fed-065,101-07,1.02
talos-r3-snow-003,talos-r3-fed-066,101-07,1.03
talos-r3-snow-004,talos-r3-fed-067,101-07,1.04
talos-r3-snow-005,talos-r3-fed-068,101-07,1.05
talos-r3-snow-006,talos-r3-fed-069,101-07,1.06
talos-r3-snow-007,talos-r3-fed-070,101-07,1.07
talos-r3-snow-008,talos-r3-fed-071,101-08,1.01
talos-r3-snow-009,talos-r3-fed-072,101-08,1.02
talos-r3-snow-010,talos-r3-fed-073,101-08,1.03
talos-r3-snow-011,talos-r3-fed-074,101-08,1.04
talos-r3-snow-012,talos-r3-fed-075,101-08,1.05
talos-r3-snow-013,talos-r3-fed-076,101-08,1.06
talos-r3-snow-014,talos-r3-fed64-060,101-08,1.07
talos-r3-snow-015,talos-r3-fed64-061,101-09,1.01
talos-r3-snow-016,talos-r3-fed64-062,101-09,1.02
talos-r3-snow-017,talos-r3-fed64-063,101-09,1.03
talos-r3-snow-018,talos-r3-fed64-064,101-09,1.04
talos-r3-snow-019,talos-r3-fed64-065,101-09,1.05
talos-r3-snow-020,talos-r3-fed64-066,101-09,1.06
talos-r3-snow-021,talos-r3-fed64-067,101-09,1.07
talos-r3-snow-022,talos-r3-fed64-068,101-10,1.01
talos-r3-snow-023,talos-r3-fed64-069,101-10,1.02
talos-r3-snow-024,talos-r3-fed64-070,101-10,1.03
talos-r3-snow-025,talos-r3-fed64-071,101-10,1.04
talos-r3-snow-026,talos-r3-leopard-060,101-10,1.05
talos-r3-snow-027,talos-r3-leopard-061,101-10,1.06
talos-r3-snow-028,talos-r3-leopard-062,101-10,1.07
talos-r3-snow-029,talos-r3-leopard-063,101-11,1.01
talos-r3-snow-030,talos-r3-leopard-064,101-11,1.02
talos-r3-snow-031,talos-r3-leopard-065,101-11,1.03
talos-r3-snow-032,talos-r3-leopard-066,101-11,1.04
talos-r3-snow-033,talos-r3-w7-066,101-11,1.05
talos-r3-snow-034,talos-r3-w7-067,101-11,1.06
talos-r3-snow-035,talos-r3-w7-068,101-11,1.07
talos-r3-snow-036,talos-r3-w7-069,101-12,1.01
talos-r3-snow-037,talos-r3-w7-070,101-12,1.02
talos-r3-snow-038,talos-r3-w7-071,101-12,1.03
talos-r3-snow-039,talos-r3-w7-072,101-12,1.04
talos-r3-snow-040,talos-r3-w7-073,101-12,1.05
talos-r3-snow-041,talos-r3-w7-074,101-12,1.06
talos-r3-snow-042,talos-r3-w7-075,101-12,1.07
talos-r3-snow-043,talos-r3-w7-076,101-13,1.01
talos-r3-snow-044,talos-r3-w7-077,101-13,1.02
talos-r3-snow-045,talos-r3-w7-078,101-13,1.03
talos-r3-snow-046,talos-r3-w7-079,101-13,1.04
talos-r3-snow-047,talos-r3-xp-063,101-13,1.05
talos-r3-snow-048,talos-r3-xp-064,101-13,1.06
talos-r3-snow-049,talos-r3-xp-065,101-14,1.01
talos-r3-snow-050,talos-r3-xp-066,101-14,1.02
talos-r3-snow-051,talos-r3-xp-067,101-14,1.03 (bug 706601)
talos-r3-snow-052,talos-r3-xp-068,101-14,1.04
talos-r3-snow-053,talos-r3-xp-069,101-14,1.05
talos-r3-snow-054,talos-r3-xp-070,101-14,1.06
talos-r3-snow-055,talos-r3-xp-071,101-14,1.07
talos-r3-snow-056,talos-r3-xp-072,101-13,30.06
talos-r3-snow-057,talos-r3-xp-073,101-13,30.07
talos-r3-snow-058,talos-r3-xp-074,101-14,30.01
talos-r3-snow-059,talos-r3-xp-075,101-14,30.02

Comment 9

[Jake Watkins [:dividehex]]

All fed's and fed64's in the list have been imaged and all have been relabeled.

Comment 10

[Jake Watkins [:dividehex]]

If releng has any problems with them, just open a new bug.

Comment 11

[Armen [:armenzg]]

Thank you guys! You guys rock :D

Comment 12

[Armen [:armenzg]]

Hi,
I believe the DNS did not get updated.
Would you be able to fix this for me? Thanks!

Armens-MacBook-Air:~ armenzg$ host talos-r3-fed-063.build.mozilla.org
talos-r3-fed-063.build.mozilla.org is an alias for talos-r3-fed-063.build.scl1.mozilla.com.
talos-r3-fed-063.build.scl1.mozilla.com has address 10.12.50.228
Armens-MacBook-Air:~ armenzg$ host talos-r3-fed-064.build.mozilla.org
Host talos-r3-fed-064.build.mozilla.org not found: 3(NXDOMAIN)
Armens-MacBook-Air:~ armenzg$ host talos-r3-fed64-060.build.mozilla.org
Host talos-r3-fed64-060.build.mozilla.org not found: 3(NXDOMAIN)
Armens-MacBook-Air:~ armenzg$ host talos-r3-leopard-060.build.mozilla.org
Host talos-r3-leopard-060.build.mozilla.org not found: 3(NXDOMAIN)
Armens-MacBook-Air:~ armenzg$ host talos-r3-w7-066.build.mozilla.org
Host talos-r3-w7-066.build.mozilla.org not found: 3(NXDOMAIN)
Armens-MacBook-Air:~ armenzg$ host talos-r3-xp-063.build.mozilla.org
Host talos-r3-xp-063.build.mozilla.org not found: 3(NXDOMAIN)

Comment 13

[Armen [:armenzg]]

This seems to have got fixed. Thanks!
Is this correct?

BTW the machines were snow leopard machines and I assume they might still have the dongles attached.
I am not sure how this could affect Fedora32/64 and XP machines but I would not like to figure it out.
Did the dongles get removed? If not, when could they be removed?

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I'm 99% sure the dongles got removed - Jake mentioned that there were 52 available for the w7's.  Yes, this got finished last night, and apparently I forgot to close the bug :)

Comment 15

[Armen [:armenzg]]

Hi,
I just noticed that the Windows 7 slaves are not activated.
Could you please fix this for me?

Is it possible to add activating Windows 7 as SOP? It can be easily be missed for releng even if we have it as one of our steps:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave#Windows_7_.28unmanaged.29

Today I am working full on this so I might notice other issues for the other OSes.

Comment 16

[Armen [:armenzg]]

The following slaves cannot be reached by ssh. They are PING UP though:
talos-r3-fed-070
talos-r3-fed-073
talos-r3-fed-074
talos-r3-fed64-064
talos-r3-fed64-071

Comment 17

[Armen [:armenzg]]

After a reboot talos-r3-fed64-072 got into the same state. Odd...

Comment 18

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #17)
> After a reboot talos-r3-fed64-072 got into the same state. Odd...

I meant: talos-r3-fed-072.

That's all for today.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Over to Matt to activate the w7 hosts here.

Can you VNC to the fedora boxes?  If not, I wonder if they're failing to puppetize?  Please open a new bug if they will need hands-on attention.

Comment 20

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #19)
> Can you VNC to the fedora boxes?  If not, I wonder if they're failing to
> puppetize?  Please open a new bug if they will need hands-on attention.
I tried to VNC before (and double checked now) and I have filed bug 712004 to take care of them.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Matt activated these last night.

Comment 22

[Armen [:armenzg]]

Thanks guys. I am putting them now on staging.

Comment 23

[Armen [:armenzg]]

Hi,
It seems that one of the Win7 slaves did not get activated:
* talos-r3-w7-072

Could you please get to it? Thanks!

Comment 24

[Amy Rich [:arr] [:arich]]

I activated talos-r3-w7-072.