1851027 - Perma talos exception with malformed payload [taskcluster:error] UAC is disabled on this worker type (gecko-t/win11-64-2009) - therefore runAsAdministrator property not allowed in task payload

Status: RESOLVED FIXED

(Infrastructure & Operations :: Relops: Azure, defect)

Description

[Natalia Csoregi [:nataliaCs]]

• Push where exceptions occurred: https://treeherder.mozilla.org/jobs?repo=mozilla-beta&revision=87b953f9d23637a6fa6ff27c3e67832db704f61b&searchStr=windows%2C11%2Cx64%2C22h2%2Cwebrender%2Cshippable%2Ctalos

• Task: https://firefox-ci-tc.services.mozilla.com/tasks/FWCHQVGvTPq5WyiXiFptaA

[taskcluster 2023-08-31T19:20:39.810Z] Task ID: FWCHQVGvTPq5WyiXiFptaA
[taskcluster 2023-08-31T19:20:39.810Z] === Task Starting ===
[taskcluster 2023-08-31T19:20:40.632Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/northeurope.vm-anwecupgttwv6fp50dsiqxblmyuwwsyk9af.60099/log/KXcv7LxOTjqGxRiw4Iz-1Q with mime type "text/plain; charset=utf-8" and expiry 2023-08-31T20:35:40.440Z
[taskcluster 2023-08-31T19:20:41.028Z] [mounts] Downloading task c_yW2Q38R7WgFqozNNY27g artifact public/build/mozharness.zip to Z:\downloads\Fl_b-R6cSC6sfkCBcITxkQ
[taskcluster 2023-08-31T19:20:41.662Z] [mounts] Downloaded 3479223 bytes with SHA256 c6c2a319f3eb1f5b25004c7496da67c7fd3c2889c7c10167427891f2291c54ca from task c_yW2Q38R7WgFqozNNY27g artifact public/build/mozharness.zip to Z:\downloads\Fl_b-R6cSC6sfkCBcITxkQ
[taskcluster:warn 2023-08-31T19:20:41.662Z] [mounts] Download Z:\downloads\Fl_b-R6cSC6sfkCBcITxkQ of task c_yW2Q38R7WgFqozNNY27g artifact public/build/mozharness.zip has SHA256 c6c2a319f3eb1f5b25004c7496da67c7fd3c2889c7c10167427891f2291c54ca but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2023-08-31T19:20:41.662Z] [mounts] Creating directory Z:\task_169350926803113\mozharness with permissions 0700
[taskcluster 2023-08-31T19:20:41.662Z] [mounts] Extracting zip file Z:\downloads\Fl_b-R6cSC6sfkCBcITxkQ to 'Z:\task_169350926803113\mozharness'
[taskcluster 2023-08-31T19:20:41.936Z] [mounts] Granting task_169350926803113 full control of directory 'Z:\task_169350926803113\mozharness'
[taskcluster 2023-08-31T19:20:41.989Z] [mounts] Downloading url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/run-task to Z:\downloads\Tn3lxBncQtK5f6JRfzRAng
[taskcluster 2023-08-31T19:20:42.316Z] [mounts] Downloaded 37925 bytes with SHA256 c52fd4fedd061ea8b8e343e5b5aea0218f33d8a0258c34a7a3a3ba02c0d14a32 from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/run-task to Z:\downloads\Tn3lxBncQtK5f6JRfzRAng
[taskcluster:warn 2023-08-31T19:20:42.317Z] [mounts] Download Z:\downloads\Tn3lxBncQtK5f6JRfzRAng of url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/run-task has SHA256 c52fd4fedd061ea8b8e343e5b5aea0218f33d8a0258c34a7a3a3ba02c0d14a32 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2023-08-31T19:20:42.317Z] [mounts] Creating directory Z:\task_169350926803113 with permissions 0700
[taskcluster 2023-08-31T19:20:42.317Z] [mounts] Copying Z:\downloads\Tn3lxBncQtK5f6JRfzRAng to Z:\task_169350926803113\run-task
[taskcluster 2023-08-31T19:20:42.331Z] [mounts] Granting task_169350926803113 full control of file 'Z:\task_169350926803113\run-task'
[taskcluster 2023-08-31T19:20:42.350Z] [mounts] Downloading url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/fetch-content to Z:\downloads\fqAOfNbdREWHB4b-b1L75A
[taskcluster 2023-08-31T19:20:42.668Z] [mounts] Downloaded 29494 bytes with SHA256 f9bcf967dd15d0501e179a6692b791f03b60417ea726bc4504db8e7833e52405 from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/fetch-content to Z:\downloads\fqAOfNbdREWHB4b-b1L75A
[taskcluster:warn 2023-08-31T19:20:42.668Z] [mounts] Download Z:\downloads\fqAOfNbdREWHB4b-b1L75A of url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Dh0DVbm9SFmGkz_f36ar3w/artifacts/public/fetch-content has SHA256 f9bcf967dd15d0501e179a6692b791f03b60417ea726bc4504db8e7833e52405 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2023-08-31T19:20:42.668Z] [mounts] Creating directory Z:\task_169350926803113 with permissions 0700
[taskcluster 2023-08-31T19:20:42.668Z] [mounts] Copying Z:\downloads\fqAOfNbdREWHB4b-b1L75A to Z:\task_169350926803113\fetch-content
[taskcluster 2023-08-31T19:20:42.672Z] [mounts] Granting task_169350926803113 full control of file 'Z:\task_169350926803113\fetch-content'
[taskcluster 2023-08-31T19:20:42.913Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2023-11-29T16:26:52.807Z
[taskcluster:error] UAC is disabled on this worker type (gecko-t/win11-64-2009) - therefore runAsAdministrator property not allowed in task payload

Comment 1

[Natalia Csoregi [:nataliaCs]]

Andrew, is there any chance you could help us assign this to someone?
Thank you.

Comment 2

[Andrew Halberstadt [:ahal]]

Mark, could this be fallout from some of your worker upgrades?

Greg, I think your team is responsible for these tasks. Maybe there's a work around here?

Comment 3

[Greg Mierzwinski [:sparky]]

It looks like it's either failing when it tries modifying the fetch-content script, or running the task (which should be the next printed line). I'm not sure there's much we can do on our side. :jmaher, do you know what machines are running these tests?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

oh, we just had a worker upgrade a few hours ago, this could be the issue. Workers + XPERF + Administrator == potential breakage.

Comment 5

[Mark Cornmesser [:markco]]

It looks like this was due to the new image that landed today. Jonathan has a fix for it. he is currently generating a new image.

Comment 6

[Natalia Csoregi [:nataliaCs]]

(In reply to Mark Cornmesser [:markco] from comment #5)

It looks like this was due to the new image that landed today. Jonathan has a fix for it. he is currently generating a new image.

Do you need to terminate the running workers or shall we do it?

Comment 7

[:jmoss]

Attachment: Bug 1851027 - Enable UAC for win11 pools. r?#relsre-reviewers

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/9e6b6d1cf355
Enable UAC for win11 pools. r=relsre-reviewers,markco

Comment 9

[Intermittent Failures Robot]

34 failures in 641 pushes (0.053 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 9
• mozilla-beta: 13
• autoland: 12

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 34
  • opt: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851027&startday=2023-08-31&endday=2023-08-31&tree=all

Comment 10

[Intermittent Failures Robot]

36 failures in 3019 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 13
• mozilla-central: 10
• autoland: 13

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 36
  • opt: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851027&startday=2023-08-28&endday=2023-09-03&tree=all