1474678 - HW Win 10 intermittently can't get to external resources

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Mark Cornmesser [:markco]]

There were some machines looping on the GW wrapper script while looking for the manifest completion flag.  It looks like the node can not get to outside of the datacenter.

This is from MS-036 @ 17:49 UTC: 
**********************
Windows PowerShell transcript start
Start time: 20180710174920
Username: WORKGROUP\SYSTEM
RunAs User: WORKGROUP\SYSTEM
Machine: T-W1064-MS-036 (Microsoft Windows NT 10.0.15063.0)
Host Application: powershell.exe -File C:\dsc\rundsc.ps1
Process ID: 1624
PSVersion: 5.1.15063.0
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.15063.0
BuildVersion: 10.0.15063.0
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
**********************
Transcript started, output file is C:\log\20180710174919.dsc-run.log
WARNING: Network connectivity may not be available, unable to reach remote sources.
WARNING: Unable to bootstrap the required package provider due to problems with network connectivity. Please fix your network connection. If this is not possible, refer to 'Get-Help Install-PackageProvider' or http://go.microsoft.com/fwlink/?LinkId=626941 for guidance on installing the package provider manually.
WARNING: MSG:UnableToDownload «https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409» «»
WARNING: Unable to download the list of available providers. Check your internet connection.
WARNING: Unable to download from URI 'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409' to ''.
Install-PackageProvider : No match was found for the specified search criteria for the provider 'NuGet'. The package 
provider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags.
At C:\dsc\rundsc.ps1:1228 char:5
+     Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Fo ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvider:InstallPackageProvider) 
[Install-PackageProvider], Exception
    + FullyQualifiedErrorId : 
NoMatchFoundForProvider,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackageProvider
Install-PackageProvider : No match was found for the specified search criteria
for the provider 'NuGet'. The package provider requires 'PackageManagement'
and 'Provider' tags. Please check if the specified package has the tags.
At C:\dsc\rundsc.ps1:1228 char:5
+     Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Fo ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvi
   der:InstallPackageProvider) [Install-PackageProvider], Exception
    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.Pac
   kageManagement.Cmdlets.InstallPackageProvider

WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2/'.
PackageManagement\Install-Package : No match was found for the specified search criteria and module name 
'xPSDesiredStateConfiguration'. Try Get-PSRepository to see all available registered module repositories.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], 
Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
PackageManagement\Install-Package : No match was found for the specified
search criteria and module name 'xPSDesiredStateConfiguration'. Try
Get-PSRepository to see all available registered module repositories.
At C:\Program
Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809
char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPacka
   ge:InstallPackage) [Install-Package], Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.Pac
   kageManagement.Cmdlets.InstallPackage

WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2/'.
PackageManagement\Install-Package : No match was found for the specified search criteria and module name 
'xWindowsUpdate'. Try Get-PSRepository to see all available registered module repositories.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], 
Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
PackageManagement\Install-Package : No match was found for the specified
search criteria and module name 'xWindowsUpdate'. Try Get-PSRepository to see
all available registered module repositories.
At C:\Program
Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809
char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPacka
   ge:InstallPackage) [Install-Package], Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.Pac
   kageManagement.Cmdlets.InstallPackage

Exception calling "DownloadFile" with "2" argument(s): "The remote name could not be resolved: 
'raw.githubusercontent.com'"
At C:\dsc\rundsc.ps1:51 char:5
+     (New-Object Net.WebClient).DownloadFile(('{0}?{1}' -f $url, [Guid ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException
Exception calling "DownloadFile" with "2" argument(s): "The remote name could
not be resolved: 'raw.githubusercontent.com'"
At C:\dsc\rundsc.ps1:51 char:5
+     (New-Object Net.WebClient).DownloadFile(('{0}?{1}' -f $url, [Guid ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

Unblock-File : Cannot find path 'C:\windows\TEMP\xDynamicConfig.ps1' because it does not exist.
At C:\dsc\rundsc.ps1:53 char:5
+     Unblock-File -Path $target
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\windows\TEMP\xDynamicConfig.ps1:String) [Unblock-File], 
ItemNotFoundException
    + FullyQualifiedErrorId : FileNotFound,Microsoft.PowerShell.Commands.UnblockFileCommand
Unblock-File : Cannot find path 'C:\windows\TEMP\xDynamicConfig.ps1' because
it does not exist.
At C:\dsc\rundsc.ps1:53 char:5
+     Unblock-File -Path $target
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\windows\TEMP\xDynamicConfig.
   ps1:String) [Unblock-File], ItemNotFoundException
    + FullyQualifiedErrorId : FileNotFound,Microsoft.PowerShell.Commands.Unblo
   ckFileCommand

. : The term 'C:\windows\TEMP\xDynamicConfig.ps1' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try 
again.
At C:\dsc\rundsc.ps1:54 char:7
+     . $target
+       ~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\windows\TEMP\xDynamicConfig.ps1:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
. : The term 'C:\windows\TEMP\xDynamicConfig.ps1' is not recognized as the
name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is
correct and try again.
At C:\dsc\rundsc.ps1:54 char:7
+     . $target
+       ~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\windows\TEMP\xDynamicConfig.
   ps1:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

xDynamicConfig : The term 'xDynamicConfig' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try 
again.
At line:1 char:1
+ xDynamicConfig -OutputPath C:\windows\TEMP\xDynamicConfig
+ ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (xDynamicConfig:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
xDynamicConfig : The term 'xDynamicConfig' is not recognized as the name of a
cmdlet, function, script file, or operable program. Check the spelling of the
name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ xDynamicConfig -OutputPath C:\windows\TEMP\xDynamicConfig
+ ~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (xDynamicConfig:String) [], Comm
   andNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Resolve-Path : Cannot find path 'C:\windows\TEMP\xDynamicConfig' because it does not exist.
    + CategoryInfo          : ObjectNotFound: (C:\windows\TEMP\xDynamicConfig:String) [Resolve-Path], 
ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.ResolvePathCommand
PS>TerminatingError(Start-DscConfiguration): "C:\windows\TEMP\xDynamicConfig is not a valid directory."
Start-DscConfiguration : C:\windows\TEMP\xDynamicConfig is not a valid directory.
At C:\dsc\rundsc.ps1:59 char:5
+     Start-DscConfiguration -Path "$mof" -Wait -Verbose -Force
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-DscConfiguration], ArgumentException
    + FullyQualifiedErrorId : 
System.ArgumentException,Microsoft.PowerShell.DesiredStateConfiguration.Commands.StartDscConfigurationCommand
Start-DscConfiguration : C:\windows\TEMP\xDynamicConfig is not a valid
directory.
At C:\dsc\rundsc.ps1:59 char:5
+     Start-DscConfiguration -Path "$mof" -Wait -Verbose -Force
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-DscConfiguration], Argu
   mentException
    + FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.De
   siredStateConfiguration.Commands.StartDscConfigurationCommand

**********************
Windows PowerShell transcript end
End time: 20180710174925
**********************

This seems to be hitting on start up. And at the very least can get to the log aggregator shortly afterwards. 

Jul 10 10:47:38 T-W1064-MS-036.mdc1.mozilla.com User32: The process C:\windows\system32\shutdown.exe (T-W1064-MS-036) has initiated the restart of computer T-W1064-MS-036 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Application: Unresponsive   Reason Code: 0x40005   Shutdown Type: restart   Comment: reboot to rouse the generic worker#015
Jul 10 10:47:38 T-W1064-MS-036.mdc1.mozilla.com Service_Control_Manager: The sshd service terminated unexpectedly.  It has done this 1 time(s).#015
Jul 10 10:50:38 T-W1064-MS-036.mdc1.mozilla.com Microsoft-Windows-WinRM: The WinRM service is not listening for WS-Management requests.      User Action    If you did not intentionally stop the service, use the following command to see the WinRM configuration:      winrm enumerate winrm/config/listener#015

Comment 1

[Kendall Libby [:fubar] (he/him)]

Generally, everything in MDC1/2 is locked down; I don't recall offhand what the agreement was for releng networks, and what access they are allowed, but it's possible that we're missing an allowed exit point here.  (eg stuff in IT's vlans needs to either use the DC proxies or have an explicit allow from EIS/netops)  Of course, that should apply to random nodes, but maybe it's worth having folks look at the panorama logs?

Comment 2

[Mark Cornmesser [:markco]]

Typically we can get out to these. OCC would not work if it couldn't. 

I have also found mlx4_bus errors around 17:49:

SingleFunc_14_0_0: Driver startup failed because the hca could not be initialized. 
SingleFunc_14_0_0: file \SystemRoot\temp\SingleFunc_14_0_0_c000_poll-tout-034_fwtrace_0.log was created due to command timeout.



I am also going to to attach the logs that were created at the same time

Comment 3

[Mark Cornmesser [:markco]]

Attachment: SingleFunc_14_0_0_c000_poll-tout-034_crspace_0.log,

Comment 4

[Mark Cornmesser [:markco]]

Attachment: SingleFunc_14_0_0_c000_poll-tout-034_crspace_1.log

Comment 5

[Mark Cornmesser [:markco]]

Attachment: SingleFunc_14_0_0_c000_poll-tout-034_crspace_2.log

Comment 6

[Mark Cornmesser [:markco]]

Attachment: SingleFunc_14_0_0_c000_poll-tout-034_fwtrace_0.log

Comment 7

[Mark Cornmesser [:markco]]

This problem persists even with the upgrade of the Mellanox drivers.

https://papertrailapp.com/groups/1141234/events?focus=961542578333966379&selected=961542578333966379

Comment 8

[Mark Cornmesser [:markco]]

This is no longer occurring.