1499610 - Build signing fails with scriptworker.exceptions.CoTError: 'path public/build/setup-stub.exe not in signing:build PminrMWDTMq4TAhWEdAmYg chain of trust artifacts!'

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=07406e9a0d2ee6389d52b519ea87b6202f24e7cc&group_state=expanded&searchStr=windows%2C2012%2Copt%2Cbuild-signing-win32%2Fopt%2C%28bs%29

Task details: https://tools.taskcluster.net/groups/dKgwbStvT5quZwGvYgkicQ/tasks/LOP2y9gcT2SDQPfAkW_qMw/details

Failure log: https://taskcluster-artifacts.net/LOP2y9gcT2SDQPfAkW_qMw/0/public/logs/chain_of_trust.log

2018-10-16T16:36:11     INFO - Downloading Chain of Trust artifact:
https://queue.taskcluster.net/v1/task/PminrMWDTMq4TAhWEdAmYg/artifacts/public%2Fbuild%2Ftarget.zip
2018-10-16T16:36:11     INFO - Downloading https://queue.taskcluster.net/v1/task/dKgwbStvT5quZwGvYgkicQ/artifacts/public%2Ftask-graph.json
2018-10-16T16:36:11     INFO - Downloading https://queue.taskcluster.net/v1/task/dKgwbStvT5quZwGvYgkicQ/artifacts/public%2Factions.json
2018-10-16T16:36:11     INFO - Downloading https://queue.taskcluster.net/v1/task/dKgwbStvT5quZwGvYgkicQ/artifacts/public%2Fparameters.yml
2018-10-16T16:36:11     INFO - Downloading https://queue.taskcluster.net/v1/task/PminrMWDTMq4TAhWEdAmYg/artifacts/public%2Fbuild%2Fsetup.exe
2018-10-16T16:36:11     INFO - Downloading https://queue.taskcluster.net/v1/task/PminrMWDTMq4TAhWEdAmYg/artifacts/public%2Fbuild%2Ftarget.zip
2018-10-16T16:36:11     INFO - Done
2018-10-16T16:36:11    DEBUG -   /builds/scriptworker/work/cot/dKgwbStvT5quZwGvYgkicQ/public/actions.json matches the expected sha256 a011130e470e8284196015b344886f1ef918480509e6e430ea21060aaffc43d4
2018-10-16T16:36:11    DEBUG -   makedirs(/builds/scriptworker/work/cot/PminrMWDTMq4TAhWEdAmYg/public/build)
2018-10-16T16:36:11     INFO - Done
2018-10-16T16:36:11    DEBUG -   /builds/scriptworker/work/cot/dKgwbStvT5quZwGvYgkicQ/public/parameters.yml matches the expected sha256 7687fb1875252d1e16714f1d40651738165009f6f2494fd3cd5da568da1c73bb
2018-10-16T16:36:11     INFO - Done
2018-10-16T16:36:11    DEBUG -   /builds/scriptworker/work/cot/PminrMWDTMq4TAhWEdAmYg/public/build/setup.exe matches the expected sha256 bac2f452ab30978f7b8e8ad2d4a7e1dbd72bb848d83db5f51205f2fe1ce24d0f
2018-10-16T16:36:11     INFO - Done
2018-10-16T16:36:11    DEBUG -   /builds/scriptworker/work/cot/dKgwbStvT5quZwGvYgkicQ/public/task-graph.json matches the expected sha256 cae287d34f8bc87d56ee82a9814a567f51617cb9753c44b7c52b98177241df35
2018-10-16T16:36:14     INFO - Done
2018-10-16T16:36:14    DEBUG -   /builds/scriptworker/work/cot/PminrMWDTMq4TAhWEdAmYg/public/build/target.zip matches the expected sha256 5973a033644d2d7b16669fbd2a09d4f31799633a34fdfeac944410d90a67152a
2018-10-16T16:36:14 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 1831, in verify_chain_of_trust
    await download_cot_artifacts(chain)
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 707, in download_cot_artifacts
    mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 326, in raise_future_exceptions
    succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 360, in _process_future_exceptions
    raise exc
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 655, in download_cot_artifact
    raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/setup-stub.exe not in signing:build PminrMWDTMq4TAhWEdAmYg chain of trust artifacts!'
2018-10-16T16:36:14    ERROR - Hit ScriptWorkerException: 'path public/build/setup-stub.exe not in signing:build PminrMWDTMq4TAhWEdAmYg chain of trust artifacts!'
2018-10-16T16:36:14    DEBUG -   "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2018-10-16T16:36:14     INFO - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Aki Sasaki (not active)]

Something turned off the stub installer generation in the central->beta merge, so this is a build configuration bug that shows itself when the signing scriptworker tries to download the missing artifact.

Comment 2

[Tom Prince [:tomprince]]

Attachment: Bug 1499610: Don't try to enable stub installer on opt builds; r?aki

These were originally enabled becuase the opt builds on mozilla-beta and
mozilla-release had the update channel set. Since Bug 1498028, only the nightly
(i.e. shippable) builds have the update channel set, the build system no longer
honors the request to enable stubs. This was causing failures in downstream tasks
that expected stub installers to be built.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7fc565646f9f
Don't try to enable stub installer on opt builds; r=aki

Comment 4

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/7fc565646f9f

Comment 5

[Intermittent Failures Robot]

6 failures in 2759 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 6

Platform breakdown:
* windows2012-32: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499610&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 2584 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499610&startday=2018-10-22&endday=2018-10-28&tree=all

Comment 7

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/88daa39a1144