1409070 - Windows builds are failing with "You do not have sufficient privilege to perform this operation."

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Greg Arndt [:garndt]]

New workers were rolled out today and afterwards, windows builds started breaking. 

https://treeherder.mozilla.org/logviewer.html#?job_id=137258008&repo=mozilla-inbound&lineNumber=47

Comment 1

[Greg Arndt [:garndt]]

Trees closed

Comment 2

[Pete Moore [:pmoore][:pete]]

This should normally be handled by https://github.com/mozilla-releng/OpenCloudConfig/blob/dce483418bb28f1154f9248c48157a434ba4591a/userdata/Manifest/gecko-3-b-win2012.json#L1265-L1281 - I am not sure why this stopped working. :/

Comment 3

[Greg Arndt [:garndt]]

Assigning to rob as he's generating new images.

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

the problem was:

- symlink permissions are applied using a powershell module called carbon.
- carbon is installed by hg cloning it from it's source repository (we should change this to a simple tooltool zip artifact)
- the hg 4.3.3 patch relied on the new hg msi being in tooltool but we missed that step during the upgrade so hg was missing on the builders. even if the symlink error hadn't occured, the hg clone would have failed in the task steps.

the solution was to upload hg 4.3.3 to tooltool and redeploy all the amis.
- gecko-3-b-win2012 redeploy (with manual intervention) succeeded here: https://tools.taskcluster.net/groups/dG-95fSKStit_cxMCLOBzw
- win7 (x86) redeploy succeeded here: https://tools.taskcluster.net/groups/StbKNTiRSPm980Pjggs2iw
- gecko-2-b-win2012 redeploy initially failed due to load on our ec2 account (too many requests) and is redeploying again here: https://tools.taskcluster.net/task-group-inspector/#/XJdOhKw5Qtm6Xscx2pEGvg
- everything else is here: https://tools.taskcluster.net/groups/GShMs-wJQH2vohqkam7ubQ

it should be safe to reopen trees now. i believe the hg upgrade to 4.3.3 is corrected now (except gecko-2-b-win2012, which has very little work to do and i expect to be fixed within the hour)

Comment 5

[Intermittent Failures Robot]

118 failures in 129 pushes (0.915 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 76
* autoland: 42

Platform breakdown:
* windows2012-32: 46
* windows2012-64: 40
* windows2012-64-rusttests: 8
* windows2012-64-noopt: 8
* windows2012-32-rusttests: 8
* windows2012-32-noopt: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1409070&startday=2017-10-16&endday=2017-10-16&tree=all

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

keeping this as infra so we can track infra vs test vs product, maybe we could consider [stockwell fixed:infra], but we haven't done that to date.

Comment 7

[Intermittent Failures Robot]

118 failures in 864 pushes (0.137 failures/push) were associated with this bug in the last 7 days. 

This is the #4 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 76
* autoland: 42

Platform breakdown:
* windows2012-32: 46
* windows2012-64: 40
* windows2012-64-rusttests: 8
* windows2012-64-noopt: 8
* windows2012-32-rusttests: 8
* windows2012-32-noopt: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1409070&startday=2017-10-16&endday=2017-10-22&tree=all