1481064 - (T-W1064-MS-119) [MDC1] T-W1064-MS-119 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Missing from taskcluster.
The worker has been re-imaged, needs verification.

Comment 1

[Roland Mutter Michael (:rmutter)]

Looks fine now. https://tools.taskcluster.net/groups/DrSXmJ_hTryoirpwoKyodQ/tasks/TsLmdCHrQiKX6ID0FNJByw/runs/0

Comment 2

[Attila Craciun [:arny]]

Sep 28 04:16:58 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: Job {F165AD69-C30F-11E8-9058-F40343DF34E2} :   This event indicates that failure happens when LCM is processing the configuration. Error Id is 0x1. Error Detail is The SendConfigurationApply function did not succeed.. Resource Id is [Package]MsiInstall_Mercurial and Source Info is C:\windows\TEMP\xDynamicConfig.ps1::385::9::Package. Error Message is PowerShell DSC resource MSFT_PackageResource  failed to execute Set-TargetResource functionality with error message: The specified Name (Mercurial 4.7.1 (x64)) and IdentifyingNumber ({68FA175A-0F28-49AF-9E65-30CE6B60AE1C}) do not match Name (Mercurial 4.5.3 (x64)) and IdentifyingNumber ({3F975FA0-6C28-4A37-882B-2AB806D67736}) in the MSI file .#015
Sep 28 04:16:58 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: Job {F165AD69-C30F-11E8-9058-F40343DF34E2} :   MIResult: 1  Error Message: PowerShell DSC resource MSFT_PackageResource  failed to execute Set-TargetResource functionality with error message: The specified Name (Mercurial 4.7.1 (x64)) and IdentifyingNumber ({68FA175A-0F28-49AF-9E65-30CE6B60AE1C}) do not match Name (Mercurial 4.5.3 (x64)) and IdentifyingNumber ({3F975FA0-6C28-4A37-882B-2AB806D67736}) in the MSI file   Message ID: ProviderOperationExecutionFailure  Error Category: 7  Error Code: 1  Error Type: MI#015
Sep 28 04:18:12 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: Job {F165AD69-C30F-11E8-9058-F40343DF34E2} :   Job runs under the following LCM setting.   ConfigurationMode: ApplyAndMonitor   ConfigurationModeFrequencyMins: 15   RefreshMode: PUSH   RefreshFrequencyMins: 30   RebootNodeIfNeeded: NONE   DebugMode: False#015
Sep 28 04:18:12 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: RESULT 1 [NXLOG@14506 Keywords="4611686018427387904" EventType="ERROR" EventID="4252" ProviderGuid="{50DF9E12-A8C4-4939-B281-47E1325BA63E}" Version="0" Task="0" OpcodeValue="0" RecordNumber="41358" ActivityID="{32D1D87B-5718-0007-500D-D2321857D401}" ThreadID="3776" Channel="Microsoft-Windows-DSC/Operational" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="S-1-5-18" AccountType="User" Opcode="Info" JobId="{F165AD69-C30F-11E8-9058-F40343DF34E2}" MIResult="1" ErrorMessage="The SendConfigurationApply function did not succeed." ErrorCategory="0" ErrorCode="1" ErrorType="MI" EventReceivedTime="2018-09-28 11:18:12" SourceModuleName="eventlog" SourceModuleType="im_msvistalog"] Job {F165AD69-C30F-11E8-9058-F40343DF34E2} :   MIResult: 1  Error Message: The SendConfigurationApply function did not succeed.  Message ID: MI RESULT 1  Error Category: 0  Error Code: 1  Error Type: MI#015
Sep 28 04:18:12 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: Job {F165AD69-C30F-11E8-9058-F40343DF34E2} : Details logging completed for C:\windows\System32\Configuration\ConfigurationStatus\{F165AD69-C30F-11E8-9058-F40343DF34E2}-0.details.json.#015
Sep 28 04:18:12 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: Job DscTimerConsistencyOperationResult :   DSC Engine Error :   #011 Error Message: NULL   #011Error Code : 1 #015
Sep 28 04:18:57 T-W1064-MS-119.mdc1.mozilla.com Microsoft-Windows-DSC: The local configuration manager was shut down.#015

Comment 3

[Mark Cornmesser [:markco]]

This sounds like an issue with the MSI itself. 

Rob: Any ideas?

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

this error resembles what we would get with a mismatched sha512 hash. if so, the problem is caused by the sha512 hash for mercurial in the manifest belonging to hg 4.5.3 rather than 4.7.1.

the sha512 hash for hg 4.7.1 should be:
> b3c957860855840c54d7407e13eb129c408c6ee9d5bf6547b5fb598321bc016c53d991cccbc5a63a78963057738aef43efd1da358b2ec41f66389d6be2fdc1cf
eg: https://github.com/mozilla-releng/OpenCloudConfig/blob/e2af1665de35d75554f7d6ff7d710234ab437faa/userdata/Manifest/gecko-t-win10-64-hw.json#L533

perhaps the instance showing this error is taking its manifest from a repo that is not https://github.com/mozilla-releng/OpenCloudConfig ?

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

actually, after checking the code in xDynamicConfig, i think there's actually a bug in there. if we're not using a separate repository for the workers showing this error, you can ignore my earlier comment.

when we download files as part of an MsiInstall (or ExeInstall or ZipInstall) we don't check that the file we want to download doesn't already exist on the filesystem. on ec2 this would rarely cause a problem but on hardware, where the filesystem can outlive some product upgrades, it's a bug.

i'll need to write a patch that takes updates or version numbers into account for msi, exe and zip files.

Comment 6

[Danut Labici [:dlabici]]

Machine is back and running jobs:
https://tools.taskcluster.net/groups/UJajuL3yQ9i59ru9h5A1gQ/tasks/aHD_BpW-SqCgj-Rt3DAiJw/runs/0