1773274 - Perma exception scriptworker.exceptions.CoTError: 'path public/build/target.normandy.json not in scriptworker:build SvDq8JQwTtqkPR8vtFxlkg chain of trust artifacts!'

Status: RESOLVED FIXED

(Firefox :: Normandy Client, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&selectedTaskRun=UictFZdwQkGS1FyIj5ee9g.1&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel

Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/UictFZdwQkGS1FyIj5ee9g/runs/2/logs/public/logs/chain_of_trust.log

2022-06-08T11:39:53    DEBUG -  /app/workdir/cot/B8BbpDYRSWiAFCPX7bB9QQ/public/build/target.complete.mar matches the expected sha256 dfec184896cd73cae7cd0d03d2d64817a253746d7bdbcc0bc9ed29e88ee68eda
2022-06-08T11:39:53 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker/cot/verify.py", line 1990, in verify_chain_of_trust
    await download_cot_artifacts(chain)
  File "/app/lib/python3.9/site-packages/scriptworker/cot/verify.py", line 764, in download_cot_artifacts
    mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 409, in raise_future_exceptions
    succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 443, in _process_future_exceptions
    raise exc
  File "/app/lib/python3.9/site-packages/scriptworker/cot/verify.py", line 716, in download_cot_artifact
    raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/target.normandy.json not in scriptworker:build SvDq8JQwTtqkPR8vtFxlkg chain of trust artifacts!'
2022-06-08T11:39:53    ERROR - Hit ScriptWorkerException: 'path public/build/target.normandy.json not in scriptworker:build SvDq8JQwTtqkPR8vtFxlkg chain of trust artifacts!'
2022-06-08T11:39:53    DEBUG -  "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2022-06-08T11:39:53     INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Andreea Pavel [:apavel]]

Aki can you please take a look?

Comment 2

[Julien Cristau [:jcristau] (back April 22)]

From the build task's log:

[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> # Generate a file that describes the local Normandy client.
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> env LD_LIBRARY_PATH=":../../dist//bin"
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> ../../dist//bin/xpcshell
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> /builds/worker/checkouts/gecko/toolkit/components/normandy/metadata-script.js ../../dist//target.normandy.json
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> JavaScript error: /builds/worker/checkouts/gecko/toolkit/components/normandy/metadata-script.js, line 21: OperationError: Could not parse path (../../dist//target.normandy.json): NS_ERROR_FILE_UNRECOGNIZED_PATH

That error is somehow ignored in the build task, but then breaks nightly.

Regressed by 8fc495765c2c547544b559635324530127684eae.

Comment 3

[Julien Cristau [:jcristau] (back April 22)]

Fixed by backout:
https://hg.mozilla.org/mozilla-central/rev/d799e63cac8b5750c746e724ef7416398c8cfa3e

Comment 4

[Aki Sasaki (not active)]

(In reply to Julien Cristau [:jcristau] from comment #2)

From the build task's log:

[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> # Generate a file that describes the local Normandy client.
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> env LD_LIBRARY_PATH=":../../dist//bin"
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> ../../dist//bin/xpcshell
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> /builds/worker/checkouts/gecko/toolkit/components/normandy/metadata-script.js ../../dist//target.normandy.json
[task 2022-06-08T11:06:49.235Z] 11:06:49 INFO - package> JavaScript error: /builds/worker/checkouts/gecko/toolkit/components/normandy/metadata-script.js, line 21: OperationError: Could not parse path (../../dist//target.normandy.json): NS_ERROR_FILE_UNRECOGNIZED_PATH

That error is somehow ignored in the build task, but then breaks nightly.

Sounds like bug 1512733 again.

Comment 5

[Intermittent Failures Robot]

8 failures in 3336 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform and build breakdown:

• linux64-shippable: 6
  • opt: 6
• windows2012-64-shippable: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773274&startday=2022-06-06&endday=2022-06-12&tree=all