1596123 - make 'Access denied' errors on Windows fatal, can bust descendants

Status: NEW

(Release Engineering :: Release Automation, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

On Windows 2012 shippable opt, the partials signing job for ast was broken. Reruns also failed:

https://firefox-ci-tc.services.mozilla.com/tasks/cnO8WQYkQNGWRye2Zxp6tA/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FcnO8WQYkQNGWRye2Zxp6tA%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Fchain_of_trust.log
scriptworker.exceptions.CoTError: 'path public/build/ast/target.partial-1.mar not in scriptworker:partials JTzxePKeSkmari1VW_-Y7w chain of trust artifacts!'

Rerunning the successful partials ast job and after that its signing job fixed it. The initial p(ast) run contained:

2019-11-13 15:08:13,793 - WARNING - target.partial-2.mar:
2019-11-13 15:08:13,793 - WARNING - target.partial-2.mar: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2019-11-13 15:08:13,832 - INFO - target.partial-2.mar: metric post releng.releases.partials.s3_cache.miss 1 --type gauge --no_host --tags='branch:mozilla-central,platform:win32'
2019-11-13 15:08:13,934 - WARNING - target.partial-4.mar:
2019-11-13 15:08:13,934 - WARNING - target.partial-4.mar: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2019-11-13 15:08:13,972 - INFO - target.partial-4.mar: metric post releng.releases.partials.s3_cache.miss 1 --type gauge --no_host --tags='branch:mozilla-central,platform:win32'
2019-11-13 15:08:14,125 - WARNING - target.partial-3.mar:
2019-11-13 15:08:14,125 - WARNING - target.partial-3.mar: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2019-11-13 15:08:14,188 - INFO - target.partial-3.mar: metric post releng.releases.partials.s3_cache.miss 1 --type gauge --no_host --tags='branch:mozilla-central,platform:win32'
2019-11-13 15:08:14,376 - WARNING - target.partial-1.mar:
2019-11-13 15:08:14,376 - WARNING - target.partial-1.mar: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

https://firefoxci.taskcluster-artifacts.net/JTzxePKeSkmari1VW_-Y7w/1/public/logs/live_backing.log

For that reason, these failures should be fatal.