1770352 - Perma OS X Cross Compiled ERROR - Hit ScriptWorkerException: CoTError('Error while rebuilding signing:parent QuLkwHldRnyoFB4bY2RqFw task definition!\nUnknown repo for source url ... )

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/jobs?repo=mozilla-esr102&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=e249c0dc507f77b1637efb6fcf12838add374cac&selectedTaskRun=PlItjSmjS7CUGXvD1rJXxA.0

Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/PlItjSmjS7CUGXvD1rJXxA/runs/1/logs/public/logs/chain_of_trust.log

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/builds/dep2/virtualenv/lib/python3.8/site-packages/scriptworker-42.0.0-py3.8.egg/scriptworker/cot/verify.py", line 1674, in verify_parent_task
    await verify_parent_task_definition(chain, link)
  File "/builds/dep2/virtualenv/lib/python3.8/site-packages/scriptworker-42.0.0-py3.8.egg/scriptworker/cot/verify.py", line 1546, in verify_parent_task_definition
    raise CoTError(msg + "\n{}".format(str(e)))
scriptworker.exceptions.CoTError: 'Error while rebuilding signing:parent QuLkwHldRnyoFB4bY2RqFw task definition!\nUnknown repo for source url https://hg.mozilla.org/releases/mozilla-esr102/raw-file/e249c0dc507f77b1637efb6fcf12838add374cac/.taskcluster.yml!'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/builds/dep2/virtualenv/lib/python3.8/site-packages/scriptworker-42.0.0-py3.8.egg/scriptworker/cot/verify.py", line 1992, in verify_chain_of_trust
    await verify_task_types(chain)
  File "/builds/dep2/virtualenv/lib/python3.8/site-packages/scriptworker-42.0.0-py3.8.egg/scriptworker/cot/verify.py", line 1750, in verify_task_types
    await valid_task_types[task_type](chain, obj)
  File "/builds/dep2/virtualenv/lib/python3.8/site-packages/scriptworker-42.0.0-py3.8.egg/scriptworker/cot/verify.py", line 1676, in verify_parent_task
    raise CoTError(e)
scriptworker.exceptions.CoTError: CoTError('Error while rebuilding signing:parent QuLkwHldRnyoFB4bY2RqFw task definition!\nUnknown repo for source url https://hg.mozilla.org/releases/mozilla-esr102/raw-file/e249c0dc507f77b1637efb6fcf12838add374cac/.taskcluster.yml!')
2022-05-20T07:55:20    ERROR - Hit ScriptWorkerException: CoTError('Error while rebuilding signing:parent QuLkwHldRnyoFB4bY2RqFw task definition!\nUnknown repo for source url https://hg.mozilla.org/releases/mozilla-esr102/raw-file/e249c0dc507f77b1637efb6fcf12838add374cac/.taskcluster.yml!')
2022-05-20T07:55:20    DEBUG -   "/builds/dep2/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2022-05-20T07:55:20     INFO - "/builds/dep2/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

This occurs only on mozilla-esr102 currently. Aki, can you please take a look?

Comment 1

[Julien Cristau [:jcristau]]

per aki, we need to deploy https://github.com/mozilla-platform-ops/ronin_puppet/pull/385 to fix this.

Comment 2

[Andrew Halberstadt [:ahal]]

This is on the production-mac-signing branch:
https://github.com/mozilla-platform-ops/ronin_puppet/tree/production-mac-signing

Does that not mean it's deployed? Did something fail during deployment?

Comment 3

[Andrew Halberstadt [:ahal]]

That PR was for adding a new CoT key, but it didn't invalidate the old one. So I don't think it will fix the issue. This looks like CoT doesn't consider esr102 to be a trusted repo, so guessing a new fix + release will be needed.

Comment 4

[Andrew Halberstadt [:ahal]]

Hm, although the logs do say scriptworker v42 is being used, which is several versions behind. Not sure why these aren't using the version on production-mac-signing. For context scriptworker 42.0.0 was released Dec 1st, 2021.

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

That PR was for adding a new CoT key, but it didn't invalidate the old one.

esr102 changes were in scriptworker 43.0.0. I was about to deploy that when 43.1.0 came out, so I abandoned my 43.0.0 efforts.

Comment 6

[Intermittent Failures Robot]

3 failures in 2694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 3

Platform and build breakdown:

• osx-cross: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770352&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 7

[Geoff Brown [:gbrown]]

Working now, after what seemed like a very tricky redeployment to pick up scriptworker 43.1.0:

https://treeherder.mozilla.org/logviewer?job_id=379170161&repo=mozilla-esr102

Thanks :ahal and :hneiva!