934976 - Migrate 80 win64-rev1 machines to win64-rev2

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

Tracking the migration of 80 win64-rev1 machines to win64-rev2, per bug 781277 comment 53.
Will be doing 20 per day from Tuesday-Friday this week.

Comment 1

[John Hopkins (:jhopkins)]

From https://etherpad.mozilla.org/win64-rev2:

DAY 1 BATCH 1
prod: w64-ix-slave06
prod: w64-ix-slave07
prod: w64-ix-slave08
prod: w64-ix-slave09
prod: w64-ix-slave10
try: w64-ix-slave30
try: w64-ix-slave31
try: w64-ix-slave32
try: w64-ix-slave33
try: w64-ix-slave34

DAY 1 BATCH 2
prod: w64-ix-slave11
prod: w64-ix-slave12
prod: w64-ix-slave13
prod: w64-ix-slave14
prod: w64-ix-slave15
try: w64-ix-slave35
try: w64-ix-slave36
try: w64-ix-slave37
try: w64-ix-slave38
try: w64-ix-slave39

DAY 2 BATCH 1
prod: w64-ix-slave16
prod: w64-ix-slave17
prod: w64-ix-slave18
prod: w64-ix-slave19
prod: w64-ix-slave20
try: w64-ix-slave40
try: w64-ix-slave41
try: w64-ix-slave44
try: w64-ix-slave45
try: w64-ix-slave46

DAY 2 BATCH 2
prod: w64-ix-slave21
prod: w64-ix-slave23
prod: w64-ix-slave24
prod: w64-ix-slave25
prod: w64-ix-slave26
try: w64-ix-slave47
try: w64-ix-slave48
try: w64-ix-slave49
try: w64-ix-slave50
try: w64-ix-slave51

DAY 3 BATCH 1
prod: w64-ix-slave27
prod: w64-ix-slave28
prod: w64-ix-slave29
prod: w64-ix-slave42
prod: w64-ix-slave43
try: w64-ix-slave52
try: w64-ix-slave53
try: w64-ix-slave54
try: w64-ix-slave55
try: w64-ix-slave56

DAY 3 BATCH 2
prod: w64-ix-slave75
prod: w64-ix-slave76
prod: w64-ix-slave77
prod: w64-ix-slave78
prod: w64-ix-slave79
try: w64-ix-slave57
try: w64-ix-slave58
try: w64-ix-slave59
try: w64-ix-slave60
try: w64-ix-slave61

DAY 4 BATCH 1
prod: w64-ix-slave81
prod: w64-ix-slave82
prod: w64-ix-slave83
prod: w64-ix-slave84
prod: w64-ix-slave87
try: w64-ix-slave62
try: w64-ix-slave63
try: w64-ix-slave64
try: w64-ix-slave65
try: w64-ix-slave66

DAY 4 BATCH 2
prod: w64-ix-slave88
prod: w64-ix-slave89
prod: w64-ix-slave90
prod: w64-ix-slave91
prod: w64-ix-slave92
try: w64-ix-slave67
try: w64-ix-slave68
try: w64-ix-slave69
try: w64-ix-slave70
try: w64-ix-slave71

Comment 2

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day 1 batch 1 -> rev2

Comment 3

[John Hopkins (:jhopkins)]

Comment on attachment 827373 [details] [diff] [review]
[buildbot-configs] day 1 batch 1 -> rev2

https://hg.mozilla.org/build/buildbot-configs/rev/218d8bb30f31

Comment 4

[Mark Cornmesser [:markco] OOO 2024/04/15]

Day 1 batch 1, with the exception of w64-ix-slave10, is now being re-imaged.

Comment 5

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day1 batch2 -> rev2

Comment 6

[John Hopkins (:jhopkins)]

Comment on attachment 827585 [details] [diff] [review]
[buildbot-configs] day1 batch2 -> rev2

Landed on default and production:
 http://hg.mozilla.org/build/buildbot-configs/rev/2ff07c61c7a3
 http://hg.mozilla.org/build/buildbot-configs/rev/b6efcd390964

Comment 7

[Mark Cornmesser [:markco] OOO 2024/04/15]

Day 1 batch 2 and w64-ix-slave10 are now in process of being re-imaged.

Comment 8

[John Hopkins (:jhopkins)]

attachment 827585 [details] [diff] [review] is in production

Comment 9

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day2 batch1 -> rev2

Comment 10

[John Hopkins (:jhopkins)]

Comment on attachment 828046 [details] [diff] [review]
[buildbot-configs] day2 batch1 -> rev2

https://hg.mozilla.org/build/buildbot-configs/rev/7030ec0898ca

Comment 11

[Mark Cornmesser [:markco] OOO 2024/04/15]

Day 2 batch one are now being imaged with the exception of:
w64-ix-slave17
w64-ix-slave19
w64-ix-slave20
w64-ix-slave41
Which are waiting on a reboot.

Comment 12

[John Hopkins (:jhopkins)]

attachment 828046 [details] [diff] [review] in production

Comment 13

[Mark Cornmesser [:markco] OOO 2024/04/15]

Day 2 batch 2 are now being imaged. Except for w64-ix-slave24, which is currently undetectable by IPMI and is included in the blocking bug above. 

w64-ix-slave17 is now imaging as well.

Comment 14

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day2 batch2 -> rev2

Comment 15

[Mark Cornmesser [:markco] OOO 2024/04/15]

w64-ix-slave19, w64-ix-slave20, and w64-ix-slave41are now in the process of being imaged. However, all machines are moving slow.

Comment 16

[John Hopkins (:jhopkins)]

Comment on attachment 828225 [details] [diff] [review]
[buildbot-configs] day2 batch2 -> rev2

https://hg.mozilla.org/build/buildbot-configs/rev/1c274b5aa15f

Comment 17

[John Hopkins (:jhopkins)]

Comment on attachment 828225 [details] [diff] [review]
[buildbot-configs] day2 batch2 -> rev2

Also landed in http://hg.mozilla.org/build/buildbot-configs/rev/c70e886ddecd

Comment 18

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day3 batch1 -> rev2

Comment 19

[John Hopkins (:jhopkins)]

Comment on attachment 828715 [details] [diff] [review]
[buildbot-configs] day3 batch1 -> rev2

https://hg.mozilla.org/build/buildbot-configs/rev/addf27088f70

Comment 20

[John Hopkins (:jhopkins)]

in production

Comment 21

[John Hopkins (:jhopkins)]

Attachment: day3batch2 and day4batch1 -> rev2

Comment 22

[John Hopkins (:jhopkins)]

Comment on attachment 829282 [details] [diff] [review]
day3batch2 and day4batch1 -> rev2

https://hg.mozilla.org/build/buildbot-configs/rev/14acbec073cc

Comment 23

[John Hopkins (:jhopkins)]

in production

Comment 24

[John Hopkins (:jhopkins)]

Attachment: [buildbot-configs] day4 batch2 -> rev2

Comment 25

[Hal Wine [:hwine] use NI!]

Comment on attachment 829613 [details] [diff] [review]
[buildbot-configs] day4 batch2 -> rev2

<stamp />

Comment 26

[John Hopkins (:jhopkins)]

Comment on attachment 829613 [details] [diff] [review]
[buildbot-configs] day4 batch2 -> rev2

Remove "#MERGE DAY" comments (after discussing with hwine) and landed in:

 https://hg.mozilla.org/build/buildbot-configs/rev/12f0e9f60362
 https://hg.mozilla.org/build/buildbot-configs/rev/098f09e1ffba (production)

Comment 27

[John Hopkins (:jhopkins)]

in production