Add hardware support to the OpenCloudConfig

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
a year ago
10 months ago

People

(Reporter: markco, Assigned: markco)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Details

Attachments

(1 attachment)

Comment hidden (empty)
(Assignee)

Updated

a year ago
Assignee: relops → mcornmesser
(Assignee)

Comment 1

a year ago
Currently trying to get this to work through he opencloudconfig located at https://github.com/mozilla-releng/OpenCloudConfig

The list of prerequisites thus far:
Framework management 4.5 
.net Framework 4.0
Powershell 5.0 
And create a firewall rule allowing Windows Remote Management HTTP-In.
(Assignee)

Comment 2

a year ago
I have hit snag on running rundsc.ps1. It errors out when it goes to run  DynamicConfig.ps1:

PSDesiredStateConfiguration\Configuration : Unable to connect to the remote server
At C:\Users\Administrator\AppData\Local\Temp\DynamicConfig.ps1:8 char:1
+ Configuration DynamicConfig {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Configuration], WebExcepti
   on
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Configuration

Which results to the following message in the event viewer: 

Log Name:      System
Source:        Microsoft-Windows-WinRM
Date:          1/30/2017 5:38:24 PM
Event ID:      10148
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      T-W864-IX-011
Description:
The WinRM service is listening for WS-Management requests. 

 User Action 
 Use the following command to see the specific IPs on which WinRM is listening: 

 winrm enumerate winrm/config/listener

However, the winrm is is listening and listening on what appears to be the correct IPs:

C:\Users\Administrator>winrm quickconfig
WinRM service is already running on this machine.
WinRM is already set up for remote management on this computer.


C:\Users\Administrator>winrm enumerate winrm/config/listener
Listener
    Address = *
    Transport = HTTP
    Port = 5985
    Hostname
    Enabled = true
    URLPrefix = wsman
    CertificateThumbprint
    ListeningOn = 10.26.40.41, 127.0.0.1, 169.254.145.172, ::1, fe80::5efe:10.26
.40.41%12, fe80::28c3:6c19:3589:c5f1%11, fe80::f57f:e461:16e3:91ac%13


Q: Grenade: DO you guys have any ideas on what could cause this kind of behavior? Any suggestions on how to move forward?
Flags: needinfo?(rthijssen)
Flags: needinfo?(q)
(Assignee)

Comment 3

a year ago
Correction: The WinRM service is not listening for WS-Management requests.
couple of thoughts pop into my head.
run `Enable-PSRemoting -Force` in the powershell console. it's supposed to enable stuff. also, is the ps shell an elevated one? it needs to be. i can see that it's the admin account, but it also needs to be an elevated ps console session. failing that, let me know how to connect to the instance and i will attempt to debug live.
Flags: needinfo?(rthijssen)

Comment 5

a year ago
Worked in IRC to resolve cancelling need info.
Flags: needinfo?(q)
(Assignee)

Updated

a year ago
Summary: Investigate the possibility of transitioning datacenter Windows hardware nodes to Desired State Config → Add hardware support to the OpenCloudConfig
(Assignee)

Comment 6

a year ago
The "The WinRM service is not listening for WS-Management requests" error was irrelevant. 

(Invoke-WebRequest -Uri 'http://169.254.169.254/latest/meta-data/public-keys' -UseBasicParsing).Content was the line cuasing the error in comment 2.
Depends on: 1336050
(Assignee)

Comment 7

a year ago
Worked around the issue in comment 6 with a conditional statement based on the existence of the EC2 service.

Also it turns out we do not need to make any changes to the firewall configuration prior to running rundsc.ps1.

Currently I have gotten to a point where dsc is applying to the machine. Now I need to chase down the various errors that occur.
(Assignee)

Comment 8

a year ago
Most of the initial errors from the manifest (gecko-t-win7-32-hw.json) regards there not being a y drive. This easily rectify with pointing everything to the C: drive. 

The other notable error:
VERBOSE: [T-W864-IX-011]: LCM:  [ End    Set      ]  [[Script]GpgKeyImport]  in 23.7260 seconds.
PowerShell DSC resource MSFT_ScriptResource  failed to execute Set-TargetResource functionality with error message: 
Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server" 
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : localhost
PowerShell DSC resource MSFT_ScriptResource  failed to execute Set-TargetResource functionality with error message:
Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server"
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : localhost
(Assignee)

Updated

a year ago
Depends on: 1336286
(Assignee)

Updated

a year ago
Depends on: 1337104
(Assignee)

Comment 9

a year ago
From this code (where "$meta_data = https://github.com/markcor/OpenCloudConfig/tree/master/userdata/meta_data"):
 
 
 
$instancekey = (Invoke-WebRequest -Uri "$meta_data/latest/meta-data/public-keys" -UseBasicParsing).Content
  if ($instancekey.StartsWith('0=aws-provisioner-v1-managed:')) {
    # provisioned worker
    $workerType = $instancekey.Split(':')[1]
  } else {
    # ami creation instance
    $workerType = $instancekey.Replace('0=mozilla-taskcluster-worker-', '')
  }
  Write-host $workerType
  if ($workerType) {
    $manifest = (Invoke-WebRequest -Uri ('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/master/userdata/Manifest/{0}.json?{1}' -f $workerType, [Guid]::NewGuid()) -UseBasicParsing | ConvertFrom-Json)
  } else {
    switch -wildcard ((Get-WmiObject -class Win32_OperatingSystem).Caption) {
      'Microsoft Windows 7*' {
        $manifest = (Invoke-WebRequest -Uri ('https://raw.githubusercontent.com/markcor/OpenCloudConfig/master/userdata/Manifest/gecko-t-win7-32-hw.json?{0}' -f [Guid]::NewGuid()) -UseBasicParsing | ConvertFrom-Json)
      }
      'Microsoft Windows 10*' {
        $manifest = (Invoke-WebRequest -Uri ('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/master/userdata/Manifest/gecko-t-win10-64-hw.json?{0}' -f [Guid]::NewGuid()) -UseBasicParsing | ConvertFrom-Json)
      }
      default {
        $manifest = ('{"Items":[{"ComponentType":"DirectoryCreate","Path":"$env:SystemDrive\\log"}]}' | ConvertFrom-Json)
      }
    }
}
 
 
 
I am getting this error. 
 
PSDesiredStateConfiguration\Configuration : 400: Invalid request
At C:\Users\Administrator\AppData\Local\Temp\DynamicConfig.ps1:8 char:1
+ Configuration DynamicConfig {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Configuration], WebExcepti
   on
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Configuration
Errors occurred while processing configuration 'DynamicConfig'.
At
C:\windows\system32\WindowsPowerShell\v1.0\Modules\PSDesiredStateConfiguration\PSDesiredStateConfiguration.psm1:3705
char:5
+     throw $ErrorRecord
+     ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (DynamicConfig:String) [], InvalidOperationException
    + FullyQualifiedErrorId : FailToProcessConfiguration
(Assignee)

Comment 10

a year ago
Created attachment 8836839 [details] [review]
Add support to rundcs.ps1 and DynamicConfig.ps1

I scrapped everything that had to deal with a faux metadata in the datacenter. 

Any thoughts or suggestions
Attachment #8836839 - Flags: feedback?(rthijssen)
(Assignee)

Updated

a year ago
Depends on: 1339198
Comment on attachment 8836839 [details] [review]
Add support to rundcs.ps1 and DynamicConfig.ps1

loogs good. the only thing i think it needs is a block to set $LocationType in DynamicConfig.ps1 (exactly like the one you have in rundsc.ps1) since the vars aren't shared between the scripts.
Attachment #8836839 - Flags: feedback?(rthijssen) → feedback+
(Assignee)

Updated

a year ago
Depends on: 1342461
Depends on: 1342892
(Assignee)

Comment 12

a year ago
Will land with: https://github.com/mozilla-releng/OpenCloudConfig/pull/49 (Bug 1338324).
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
I noticed in slave health that t-w864-ix-011 and t-w864-ix-012 machines are disabled from slavealloc, and that in the comments they are both pointing to this bug.
Do you still need the machines for further tests or can I place them back in production ?
Flags: needinfo?(mcornmesser)
(Assignee)

Comment 14

a year ago
(In reply to Sebastian Pacurar [:spacurar] from comment #13)
> I noticed in slave health that t-w864-ix-011 and t-w864-ix-012 machines are
> disabled from slavealloc, and that in the comments they are both pointing to
> this bug.
> Do you still need the machines for further tests or can I place them back in
> production ?

We will still needs these for few more weeks.
Flags: needinfo?(mcornmesser)
Markco do you still need t-w864-ix-011 and t-w864-ix-012, or can we place them back in production?
Flags: needinfo?(mcornmesser)
(Assignee)

Comment 16

a year ago
Yes. It may be another week or 2.
Flags: needinfo?(mcornmesser)

Updated

11 months ago
Blocks: 1366828
Just want to check if t-w864-ix-011 and t-w864-ix-012 are still needed :-)
Flags: needinfo?(mcornmesser)
(Assignee)

Comment 18

10 months ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #17)
> Just want to check if t-w864-ix-011 and t-w864-ix-012 are still needed :-)

I am finally done with these machines. I have kicked off a reinstall on both of them.
Flags: needinfo?(mcornmesser)
You need to log in before you can comment on or make changes to this bug.