1563971 - Perma android signing nightly COT ScriptWorkerException when ecking array values: Check all menuitems are displayed correctly. - got 1, expected 2 when Gecko 69 merges to Beta on 2019-07-01

Status: VERIFIED FIXED

(Release Engineering :: Release Automation: Signing, defect)

Description

[Andreea Pavel [:apavel]]

Central as Beta simulation

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=255161582&resultStatus=testfailed%2Cbusted%2Cexception&revision=093f3ab614392a2934deeee1536fadf4998e1657

Failure log: https://taskcluster-artifacts.net/OoOMC8sFTN2AVM0MGUl_AA/1/public/logs/chain_of_trust.log

2019-07-07T20:38:43 INFO - Downloading Chain of Trust artifact:
https://queue.taskcluster.net/v1/task/buSYEilYTPe-2x1zOO8zFQ/artifacts/public%2Fbuild%2Ftarget.apk
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/GKWuFZn8RiumUU8npINinQ/artifacts/public%2Factions.json
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/GKWuFZn8RiumUU8npINinQ/artifacts/public%2Fparameters.yml
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/GKWuFZn8RiumUU8npINinQ/artifacts/public%2Ftask-graph.json
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/O1A4if0eRrq2a8X9QBEKAQ/artifacts/public%2Factions.json
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/O1A4if0eRrq2a8X9QBEKAQ/artifacts/public%2Fparameters.yml
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/O1A4if0eRrq2a8X9QBEKAQ/artifacts/public%2Ftask-graph.json
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/buSYEilYTPe-2x1zOO8zFQ/artifacts/public%2Fbuild%2Fen-US%2Ftarget.apk
2019-07-07T20:38:43 INFO - Downloading https://queue.taskcluster.net/v1/task/buSYEilYTPe-2x1zOO8zFQ/artifacts/public%2Fbuild%2Ftarget.apk
2019-07-07T20:38:43 INFO - Done
2019-07-07T20:38:43 DEBUG - /builds/scriptworker/work/cot/O1A4if0eRrq2a8X9QBEKAQ/public/parameters.yml matches the expected sha256 4a2e9147b393f266b3fe2a560a8d18ffdb8dd0c79b645ce85c27c14177392c30
2019-07-07T20:38:43 INFO - Done
2019-07-07T20:38:43 DEBUG - /builds/scriptworker/work/cot/O1A4if0eRrq2a8X9QBEKAQ/public/actions.json matches the expected sha256 5b21144de5d70ce67a31fa272857ff58c7519e20f14d9889b8fc33bdf431a898
2019-07-07T20:38:43 INFO - Done
2019-07-07T20:38:43 DEBUG - /builds/scriptworker/work/cot/GKWuFZn8RiumUU8npINinQ/public/actions.json matches the expected sha256 87ddd7640398873b021171260ca0c87f2cae87773ca8a646df635b936aa91211
2019-07-07T20:38:43 INFO - Done
2019-07-07T20:38:43 DEBUG - /builds/scriptworker/work/cot/GKWuFZn8RiumUU8npINinQ/public/parameters.yml matches the expected sha256 239d4d28a2ac0cddfd88899a6027a4bdcc6083948a2ffa626ed6843e61c8325d
2019-07-07T20:38:44 INFO - Done
2019-07-07T20:38:44 DEBUG - /builds/scriptworker/work/cot/O1A4if0eRrq2a8X9QBEKAQ/public/task-graph.json matches the expected sha256 7b8667c7608a40c4132cbde94258012226fc3dbe3e3bba3812f8c5cd9d11ad62
2019-07-07T20:38:44 INFO - Done
2019-07-07T20:38:44 DEBUG - /builds/scriptworker/work/cot/GKWuFZn8RiumUU8npINinQ/public/task-graph.json matches the expected sha256 52086ecca50b9a802d93ca0678b2f683f197fa6674233b356ee8eef83491ff2e
2019-07-07T20:39:12 DEBUG - makedirs(/builds/scriptworker/work/cot/buSYEilYTPe-2x1zOO8zFQ/public/build/en-US)
2019-07-07T20:39:18 INFO - Done
2019-07-07T20:39:18 DEBUG - /builds/scriptworker/work/cot/buSYEilYTPe-2x1zOO8zFQ/public/build/target.apk matches the expected sha256 48802e01754ba813b6a584f69f5a9e05c4c474aebc839a5793782315324781f0
2019-07-07T20:39:18 INFO - Done
2019-07-07T20:39:18 DEBUG - /builds/scriptworker/work/cot/buSYEilYTPe-2x1zOO8zFQ/public/build/en-US/target.apk matches the expected sha256 5ea1a8acf7d406c5fdfc49b6696eeef81da705c76a2c7334d1a76b492fc84f99
2019-07-07T20:39:18 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 2139, in verify_chain_of_trust
await download_cot_artifacts(chain)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 716, in download_cot_artifacts
mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 327, in raise_future_exceptions
succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/utils.py", line 361, in _process_future_exceptions
raise exc
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/cot/verify.py", line 664, in download_cot_artifact
raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/maven/org/mozilla/geckoview/geckoview-nightly-try-armeabi-v7a/69.0.20190707115014/geckoview-nightly-try-armeabi-v7a-69.0.20190707115014-javadoc.jar not in signing:build buSYEilYTPe-2x1zOO8zFQ chain of trust artifacts!'
2019-07-07T20:39:18 ERROR - Hit ScriptWorkerException: 'path public/build/maven/org/mozilla/geckoview/geckoview-nightly-try-armeabi-v7a/69.0.20190707115014/geckoview-nightly-try-armeabi-v7a-69.0.20190707115014-javadoc.jar not in signing:build buSYEilYTPe-2x1zOO8zFQ chain of trust artifacts!'
2019-07-07T20:39:18 DEBUG - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2019-07-07T20:39:18 INFO - "/builds/scriptworker/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Tom Prince [:tomprince]]

It looks like this should look at the update-channel attribute of the build that is being signed (or beetmoved), rather than having a hardcoded mapping (like here for example).

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1563971 - part 1: Fix promote graphs by passing down jobs instead of tasks r=mtabara

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1563971 - part 2: geckoview, use update-channel instead of hardcoded file name r=mtabara

Depends on D37231

Comment 4

[Johan Lorenzo [:jlorenzo]]

Thanks for the investigation, Tom! I agree using update-channel is the right fix here. I got rid of the hardcoded mapping.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/42ba06cb7ad4
part 1: Fix promote graphs by passing down jobs instead of tasks r=mtabara
https://hg.mozilla.org/integration/autoland/rev/791d8621533f
part 2: geckoview, use update-channel instead of hardcoded file name r=mtabara

Comment 6

[Ryan VanderMeulen [:RyanVM]]

No need to track if this only affects Try simulations.

Comment 7

[Johan Lorenzo [:jlorenzo]]

[Tracking Requested - why for this release]: I'm sorry, I forgot to add some information I found while writing the patch. While the original bug only affects Try simulations, this patch [1] is needed on beta. It's another fallout of bug 1557269. If this patch doesn't make beta, the promote phase won't start because of a Python error.

[1] https://hg.mozilla.org/integration/autoland/rev/42ba06cb7ad4

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Well, in that case...!

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/42ba06cb7ad4
https://hg.mozilla.org/mozilla-central/rev/791d8621533f

Comment 10

[Johan Lorenzo [:jlorenzo]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7f7392be9ddb
https://hg.mozilla.org/releases/mozilla-beta/rev/50c84e8e8e46

Comment 11

[Cosmin Sabou [:CosminS]]

Fix verified in beta-sims: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=77c16030b36ccdd0d575b9dcc644a88d5932875a&searchStr=ns%2Candroid&selectedJob=255710862