1618216 - asan reporter nightly build repackages broken - chain of trust error for target.awsy.tests.tar.gz

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Likely from bug 1606922 according to "Firefox CI" channel on Matrix.

Merge with nightly: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=290550997&searchStr=asan%2Creporter

Log: https://firefox-ci-tc.services.mozilla.com/tasks/CEKqPkDhRj6CvY8A8z4yhg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FCEKqPkDhRj6CvY8A8z4yhg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Fchain_of_trust.log

2020-02-26T12:08:22 DEBUG - /app/workdir/cot/Hhf9j57DQsen5D9RJro9aA/public/build/target.complete.mar matches the expected sha256 fa6ddb71f4eb8b1b2267b14343c3c16278e140ece208731a474e1b072ab0dd87
2020-02-26T12:08:27 INFO - Done
2020-02-26T12:08:28 DEBUG - /app/workdir/cot/Z8DTrjTfR3mr0YH2KAWG1Q/public/build/target.tar.bz2 matches the expected sha256 6f1499451ca053ef686b8c0905eb6d6c0b2f23749097f144079344992309f735
2020-02-26T12:08:28 CRITICAL - Chain of Trust verification error!
Traceback (most recent call last):
File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 1906, in verify_chain_of_trust
await download_cot_artifacts(chain)
File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 719, in download_cot_artifacts
mandatory_artifacts_paths = await raise_future_exceptions(mandatory_artifact_tasks)
File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 398, in raise_future_exceptions
succeeded_results, _ = await _process_future_exceptions(tasks, raise_at_first_error=True)
File "/app/lib/python3.8/site-packages/scriptworker/utils.py", line 432, in _process_future_exceptions
raise exc
File "/app/lib/python3.8/site-packages/scriptworker/cot/verify.py", line 671, in download_cot_artifact
raise CoTError("path {} not in {} {} chain of trust artifacts!".format(path, link.name, link.task_id))
scriptworker.exceptions.CoTError: 'path public/build/target.awsy.tests.tar.gz not in scriptworker:build VoLWXK69QryAfEzgmZrjSw chain of trust artifacts!'
2020-02-26T12:08:28 ERROR - Hit ScriptWorkerException: 'path public/build/target.awsy.tests.tar.gz not in scriptworker:build VoLWXK69QryAfEzgmZrjSw chain of trust artifacts!'
2020-02-26T12:08:28 DEBUG - "/app/artifacts/public/logs/chain_of_trust.log" is encoded with "None" and has mime/type "text/plain"
2020-02-26T12:08:28 INFO - "/app/artifacts/public/logs/chain_of_trust.log" can be gzip'd. Compressing...

Comment 1

[Justin Wood (:Callek)]

I adjusted the in-tree logic to treat nightly in the attributes the same as shippable for the verification, and made exceptions be a list that is all raised at the end...

:mshal do you see any further details that can help us here?

Exception: Build job build-android-x86-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-win64-aarch64-devedition/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-win64-asan-reporter-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-linux-devedition/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-android-aarch64-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-android-x86_64-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-linux64-asan-reporter-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.
Build job build-android-api-16-nightly/opt is nightly and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment.

Comment 2

[Justin Wood (:Callek)]

Attachment: Bug 1618216 - When doing test packaging test for the nightly attribute as well, and validate against it. r=mshal

Future work should hopefully find a way to not assume tests are packaged with beetmover, and to have a solution for w64's emefree design.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd221aeba081
When doing test packaging test for the nightly attribute as well, and validate against it. r=mshal

Comment 4

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/cd221aeba081

Comment 5

[Justin Wood (:Callek)]

Attachment: Bug 1618216 - Followup, android nightlies *do* need test packaging. r=#firefox-build-system-reviewers

This is necessary, at the least because Raptor on Fenix needs the test packages
e.g. https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fKZ-MTqGQo-yMxDn1eCEKQ/artifacts/public/build/en-US/target.test_packages.json

In Fenix that particular task is pulled into the taskgraph via https://github.com/mozilla-mobile/fenix/blob/master/taskcluster/ci/geckoview/kind.yml

• specifically the index at gecko.v2.mozilla-central.nightly.latest.mobile.android-x86_64-opt

That is then referenced in https://github.com/mozilla-mobile/fenix/blob/a0c60c70da2bb648049ddd0f64348fff5509130a/taskcluster/fenix_taskgraph/transforms/raptor.py
Lines 68 and 77

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08cf33acbe67
Followup, android nightlies *do* need test packaging. r=froydnj

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Justin, should the fenix jobs pass now?

Comment 8

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/08cf33acbe67

Comment 9

[Justin Wood (:Callek)]

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #7)

Justin, should the fenix jobs pass now?

Confirmed, next full push of Fenix that triggers raptor jobs should work. Fenix caches the taskid of the last android nightly when its graph is generated, so a rerun of raptor won't grab newer nightlies. But the next run should be fine

Comment 10

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #9)

(In reply to Alexandru Ionescu :alexandrui (needinfo me) from comment #7)

Justin, should the fenix jobs pass now?

Confirmed, next full push of Fenix that triggers raptor jobs should work. Fenix caches the taskid of the last android nightly when its graph is generated, so a rerun of raptor won't grab newer nightlies. But the next run should be fine

...Actually I can just trigger a new set of raptor jobs on the tip of Fenix, I'll do that

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Another error that seems to be related to this bug:
https://treeherder.mozilla.org/#/jobs?repo=fenix&searchStr=rap&fromchange=29d9ba472ce3dc3a72b1e46846b18a4f5e7fd4ce&selectedJob=291735058
HTTPError: 400 Client Error: Bad Request for url: http://taskcluster/queue/v1/task/I0r11TqXQ3-P2Vl_ZVBTYQ