1596612 - browsertime android fenix and refbrow jobs created for central when they should be "try" only

Status: RESOLVED WORKSFORME

(Testing :: Raptor, defect, P1)

Description

[Robert Wood [:rwood]]

I think there might be an issue in the raptor taskcluster transform. In browsertime-mobile.yml if you set the default so the tests on fenix and refbrow will run on 'try' only, like this:

run-on-projects:
by-app:
fenix: []
geckoview: ["mozilla-central", "try"]
refbrow: []

In reality the tests run on all three apps on "mozilla-central" and "try" even though fenix and refbrow should only run on "try". In the browsertime-tp6m-1 test definition in the same config we use this:

apps: ["fenix", "geckoview", "refbrow"]

Perhaps the transform building the jobs out for each app is adding jobs for each app on all branches instead of "try" only for fenix and refbrow? Just guessing.

Comment 1

[Robert Wood [:rwood]]

I grabbed a 'parameters.yml' from a D decision task on central then ran (with existing configs/transforms):

./mach taskgraph full -p ../Downloads/parameters.yml —json > rw-taskgraph-before.json

Then I modified the browsertime-tp6m transform and added 'fenix' to the apps list here:
https://searchfox.org/mozilla-central/rev/c61720a7d0c094d772059f9d6a7844eb7619f107/taskcluster/ci/test/browsertime-mobile.yml#57

Then ran taskgraph again:

./mach taskgraph full -p ../Downloads/parameters.yml —json > rw-taskgraph-after.json

And diffed the two .json files and I can see the browsertime android fenix jobs added as expected, and in each one I see:

"run_on_projects": [],

Which is correct! So I don't see how these new jobs would run on central, my understanding is '[]' means try only...

Comment 2

[Robert Wood [:rwood]]

In the patch in Bug 1592063 I added "fenix" to the apps list for browsertime-tp6m (but have by-app: fenix: []) so after that merges I'll see if those jobs are unexpectedly running on central or if just enabled for try only as expected. Hopefully it's fine and this bug can be closed out.

Comment 3

[Robert Wood [:rwood]]

Good news, it's working fine, with Fenix now included in the 'apps' list in the browsertime-tp6m taskcluster config entry, it is not running on central, only on try as expected. Must have been a one time thing when I first spotted it, or else I was misunderstanding and looking at the wrong jobs.

Comment 4

[Robert Wood [:rwood]]

Meant to add a link to latest central:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=btime&revision=e928d60013446484acd63757ac883626b722f85d