1451837 - Write a continuous hardware management/monitoring module for OCC

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Comment 1

[Mark Cornmesser [:markco]]

Attachment: userdata/rundsc.ps1

Rob: Looking for feed back on the changes in rundsc.ps1

It include:
hw-diskmanage function 
Setting a variable to reflect which datacenter the node is in
Set ntp server 
Call hw-diskmanage function 

Also do you see a reason why the shutdown command the hw-diskmanage function does not actually work?

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Mark Cornmesser [:markco] from comment #1)
> Created attachment 8974570 [details]
> userdata/rundsc.ps1
> 
> Rob: Looking for feed back on the changes in rundsc.ps1
> 
> It include:
> hw-diskmanage function 
> Setting a variable to reflect which datacenter the node is in
> Set ntp server 
> Call hw-diskmanage function 
> 
> Also do you see a reason why the shutdown command the hw-diskmanage function
> does not actually work?

looks good to me.
i can't see any problem with the shutdown command. my guess would be that the line isn't getting hit. i like to add a "/d p:4:1" arg to the command to reduce log noise but not having the arg wouldn't affect the shutdown success or failure.

Comment 3

[Mark Cornmesser [:markco]]

There is a 5 node test pool using this version of rundsc.ps1:
t-w1064-ms-030
t-w1064-ms-038
t-w1064-ms-039
t-w1064-ms-059
t-w1064-ms-114

I have notify buildduty that these nodes have a different configuration and ask them to quarantine if they begin causing issues.

Comment 4

[Mark Cornmesser [:markco]]

The test pool is hitting an occasional DNS issue where it is unable to pull the DNS domain. This in turn affects the setting of the NTP server. There is then a process (under the hood) triggered that continuously tries to set the ntp server to the nonexistent server (infoblox1.private..mozilla.com). Though the node continues to pick p tasks and produce green tests. Rundsc doesn't seem to run correctly after this happens. 

To work around this I have added a reboot to the else statement when setting the location variable: 

    Write-Log -message "Domian is $DNSDomain" -severity 'INFO'
    if ($DNSDomain -like '*mdc1*') { 
      $datacenter = "MDC1"
      reg ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" /v "NV Domain" /d mdc1.mozilla.com /f
      Write-Log -message "Node is located in MDC1" -severity 'INFO'
    } 
    elseif ($DNSDomain -like '*mdc2*') { 
      $datacenter = MDC2
      reg ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" /v "NV Domain" /d mdc2.mozilla.com /f
      Write-Log -message "Node is located in MDC2" -severity 'INFO'
    } 
    else {
      Write-Log -message  "Cannot determine Physical location" -severity 'ERROR'
      Remove-Item -Path $lock -force -ErrorAction SilentlyContinue
      & shutdown @('-s', '-t', '0', '-c', 'Cannot determine Physical location;Possible DNS issue', '-f', '-d', '1:1') | Out-File -filePath $logFile -append
   }
}
Hopefully the node will reboot and try again. There are better ways deal with it, but this is the path of least resistance. I am going to reinstall the 2 affected nodes and let the test pool sit for a few more days. Hopefully this patch can land by week's end.

Comment 5

[Mark Cornmesser [:markco]]

Attachment: Run dism cleanup command after diskspace warning

We started getting some disk space warning this morning. This will clear up several gigs of space. 


I am including it on this bug because this is a stop gap for a function that will be included in the GenericWorker upgrade patch.