1601197 - Make sure the condprofclient picks mozilla-central artifacts by default

Status: RESOLVED FIXED

(Testing :: Condprofile, defect, P1)

Description

[Tarek Ziadé (:tarek)]

The conditioned profile is supposed to be triggered by a cron (or manually during tests)
I am still seeing "cold" profiles generated and published at https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.try.latest.firefox/condprof-macosx64

this task for instance, triggered it https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3769c6e5fa881310f2e2c343f329011338bfcfd9

Comment 1

[Tarek Ziadé (:tarek)]

we need to filter out the condprof task

https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#99

but this won't be caught in mach try fuzzy

jlorenzo suggested to poke at https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#161 to try to filter
out the task when try fuzzy is called. (digging)

Comment 2

[Tarek Ziadé (:tarek)]

To summarize: it's hard to prevent try runs, but we're storing artifacts on mozilla-central too, and those are "clean" and the ones we should pick by default in the client.

Comment 3

[Tarek Ziadé (:tarek)]

Attachment: Bug 1601197 - Make sure the condprofclient picks mozilla-central artifacts by default

The client will pick condprofs from mozilla-central by default

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26f4282834a3
Make sure the condprofclient picks mozilla-central artifacts by default r=stephendonner

Comment 5

[Alexandru Michis [:malexandru]]

Backed out changeset 26f4282834a3 for causing condprof failures in condprof/tests/test_client.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/3c7dfe06b8492c73c322044fe7a2146414e9c0e5

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&collapsedPushes=598800&resultStatus=testfailed%2Cbusted%2Cexception&tochange=3c7dfe06b8492c73c322044fe7a2146414e9c0e5&fromchange=26f4282834a3ef9bfd8ae3a388aef7f2248bb451

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=279749198&repo=autoland&lineNumber=288

[task 2019-12-05T10:20:27.445Z] Virtualenv location: /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-QUm8IANb-2
[task 2019-12-05T10:20:27.466Z] Creating a Pipfile for this project…
[task 2019-12-05T10:20:28.353Z] running build_ext
[task 2019-12-05T10:20:28.354Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-12-05T10:20:28.354Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-12-05T10:20:28.354Z]
[task 2019-12-05T10:20:28.354Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-12-05T10:20:35.263Z] Test configuration changed. Regenerating backend.
[task 2019-12-05T10:20:35.263Z] No build detected, test metadata may be incomplete.
[task 2019-12-05T10:20:35.263Z] 0:23.04 /builds/worker/checkouts/gecko/testing/condprofile/condprof/tests/test_client.py
[task 2019-12-05T10:20:35.264Z] 0:23.04 TEST-UNEXPECTED-FAIL | /builds/worker/checkouts/gecko/testing/condprofile/condprof/tests/test_client.py | TestClient.test_cache, line 3: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-05T10:20:35.264Z] 0:23.04 ERROR: test_cache (main.TestClient)
[task 2019-12-05T10:20:35.264Z] 0:23.04 Traceback (most recent call last):
[task 2019-12-05T10:20:35.264Z] 0:23.04 File "<string>", line 3, in wrapper
[task 2019-12-05T10:20:35.264Z] 0:23.04 File "/builds/worker/checkouts/gecko/testing/condprofile/condprof/tests/test_client.py", line 50, in test_cache
[task 2019-12-05T10:20:35.264Z] 0:23.04 get_profile(self.target, "win64", "settled", "default")
[task 2019-12-05T10:20:35.264Z] 0:23.04 File "/builds/worker/checkouts/gecko/testing/condprofile/condprof/client.py", line 77, in get_profile
[task 2019-12-05T10:20:35.264Z] 0:23.04 raise ProfileNotFoundError(url)
[task 2019-12-05T10:20:35.264Z] 0:23.04 ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-05T10:20:35.265Z] 0:23.04
[task 2019-12-05T10:20:35.265Z] 0:23.04 Getting https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-05T10:20:35.265Z] 0:23.04 [2019-12-05T10:20:35.211891] Cannot find 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz'
[task 2019-12-05T10:20:35.265Z] 0:23.04 Setting retcode to 1 from /builds/worker/checkouts/gecko/testing/condprofile/condprof/tests/test_client.py
[task 2019-12-05T10:20:35.266Z] 0:23.04 Return code from mach python-test: 1
[taskcluster 2019-12-05 10:20:35.815Z] === Task Finished ===
[taskcluster 2019-12-05 10:20:35.815Z] Unsuccessful task run with exit code: 1 completed in 354.557 seconds

Comment 6

[Tarek Ziadé (:tarek)]

I forgot to append the tests change in the commit. oops.. done now.

new try run for the tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=99673cb02c31afddb9e9d05b44542351288ddaf2

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1bee09baeac8
Make sure the condprofclient picks mozilla-central artifacts by default r=stephendonner

Comment 8

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/1bee09baeac8

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Does this need uplift to beta?

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Liz Henry (:lizzard) from comment #9)

Does this need uplift to beta?

Yes.

Comment 11

[Andreea Pavel [:apavel]]

(In reply to Stephen Donner [:stephend] from comment #10)

(In reply to Liz Henry (:lizzard) from comment #9)

Does this need uplift to beta?

Yes.

Can you request beta approval or should we wait for Tarek to come back from pto?

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Andreea Pavel [:apavel] from comment #11)

Can you request beta approval or should we wait for Tarek to come back from pto?

I'd prefer Tarek or Tom (sorry to yet-again tag you, Tom, but you already have a better handle) to also weigh in, please, as there might be additional changes needed/dependencies. I'll dig in again first thing tomorrow.

Comment 13

[Andreea Pavel [:apavel]]

Sounds good, thank you.

Comment 14

[Liz Henry (:lizzard) (relman/hg->git project)]

I'll keep an eye on this over the weekend and I'm sure will be checking back in Sunday night so we have a chance to get this landed before the merge and the RC build on Monday.

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

This will miss the 72 release but we can keep tracking for now.