1551337 - Deployed Ronin Puppet configure Windows 10 to Datacenters

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Windows OS, task)

Comment 1

[Mark Cornmesser [:markco]]

Will require re-imaging of all Windows Moonshot nodes. The applicable task sequence is Datacenter geckotwin1064hw PreBootstrap.

Comment 2

[Mark Cornmesser [:markco]]

Jmaher: is there any concern with starting this deployment on 2019-05-28?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

that is after beta is released- so yes it is a great time to do it.

Comment 4

[Mark Cornmesser [:markco]]

Jmaher: Could you take a look at https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ea6863c6de2d1a2eac0739e2bfd7a6c5bca1aea&selectedJob=250415783 , please? Are these failing tests concerning? How are the overall results looking?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I did some retriggers and it looks good. I am not sure what your base revision is.

Comment 6

[Mark Cornmesser [:markco]]

Ms-017 through ms - 045 and ms-316 through ms-345 has been deployed.

Comment 7

[Mark Cornmesser [:markco]]

This has been deployed to MDC1. The last nodes will finish installing within the next 30 minutes.

Comment 8

[Zsolt Fay [:zfay]]

I've deployed this to chassis 9, 10 and 11 on MDC2.
Found a few workers with boot problems: t-w1064-ms-{367, 370, 387, 420, 454, 460, 461, 472, 476, 477}
Issued a cartridge power reset and tried re-imaging them again. Some of them remained faulty.
@markco can you check the ones that are still faulty? The faulty ones are t-w1064-ms-{370, 460, 472, 476, 477}

Comment 9

[Attila Craciun [:arny]]

Deployed chassis 12, except worker t-w1064-ms-525.

Comment 10

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Deployed to chassis 13 (all of them), everything went ok.

Comment 11

[Attila Craciun [:arny]]

Deployed chassis 14, except worker t-w1064-ms-{581,582,599,600}.

Comment 12

[Mark Cornmesser [:markco]]

(In reply to Zsolt Fay [:zfay] from comment #8)

I've deployed this to chassis 9, 10 and 11 on MDC2.
Found a few workers with boot problems: t-w1064-ms-{367, 370, 387, 420, 454, 460, 461, 472, 476, 477}
Issued a cartridge power reset and tried re-imaging them again. Some of them remained faulty.
@markco can you check the ones that are still faulty? The faulty ones are t-w1064-ms-{370, 460, 472, 476, 477}

These look good now. All logs are similar to:

Jun 13 08:37:14 T-W1064-MS-367.mdc2.mozilla.com-1 puppet-run: 2019-06-13 14:21:10 +0000 Puppet (notice): Applied catalog in 28.12 seconds#015
Jun 13 08:39:09 T-W1064-MS-367.mdc2.mozilla.com-1 generic-worker: No task claimed. Idle for 1m18.0684145s (will exit if no task claimed in 1h58m41.9315855s).#0

Comment 13

[Mark Cornmesser [:markco]]

All the nodes are good except 525 and 600. Currently trying to reimage those 2.

Comment 14

[Mark Cornmesser [:markco]]

525 is in the wrong vlan. Moved to https://bugzilla.mozilla.org/show_bug.cgi?id=1559196 .

Comment 15

[Mark Cornmesser [:markco]]

600 appears to a network issue. Moved to https://bugzilla.mozilla.org/show_bug.cgi?id=1559209.