1578723 - Raptor-speedometer-cpu-memory-power is running on central when it shouldn't

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P3)

Description

[Greg Mierzwinski [:sparky]]

I noticed that the raptor-speedometer-cpu-memory-power has been running on mozilla-central when it shouldn't since we've set the run-on-projects flag to []: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/raptor-gve.yml#178

I don't see anything obvious that could be causing this issue. Dustin, would you know what might be causing this? If not, could you or someone else from the TaskCluster team look into this?

Edit, here's a link to a task running this test (and many others) when it shouldn't: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=power&revision=174361d152923aea2b24226a57cde3099509ed6a

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I don't know much about how the taskgraph stuff works these days. At a guess, does something else depend on it? Otherwise, I'd suggest checking with others in the taskgraph module

Comment 2

[Greg Mierzwinski [:sparky]]

Ok, thanks :dustin! I can't find anything that depends on it.

:ahal, would you know what might be causing this issue (see first comment)?

Comment 3

[Andrew Halberstadt [:ahal]]

It could be getting set in a transform somewhere. Sorry, I don't know offhand. Maybe Rob would have an idea, or if not you might need to dig through all the transforms and see where it might be set.

Somewhat related but not helpful to this bug.. The raptor configs are getting a bit out of hand. It might be time for us to try and figure out how we can move more logic into transforms and reduce the .yml configs (e.g ideally we would set platform information via a list key + transforms rather than have separate .yml files for each platform). But this is more to prevent these kinds of things from happening in the future than anything actionable now.

Comment 4

[Robert Wood [:rwood]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

It could be getting set in a transform somewhere. Sorry, I don't know offhand. Maybe Rob would have an idea, or if not you might need to dig through all the transforms and see where it might be set.

No I don't know offhand, sorry.

Somewhat related but not helpful to this bug.. The raptor configs are getting a bit out of hand. It might be time for us to try and figure out how we can move more logic into transforms and reduce the .yml configs (e.g ideally we would set platform information via a list key + transforms rather than have separate .yml files for each platform). But this is more to prevent these kinds of things from happening in the future than anything actionable now.

Yes absolutely, this is something that we are going to look at and solve as we transition from Raptor to Browsertime. For the Browsertime MVP we are using the existing Raptor test jobs so we can easily compare data; however once we move to Browsertime in production we definitely need to solve this. I have filed Bug 1578749 to make sure we look at this. Thanks Andrew!

Comment 5

[Greg Mierzwinski [:sparky]]

I found the issue. It seems like the merge_to function is merging the job-defaults with the setting in the test's task which should be overridden (unless by-test-platform is specified).

Here's where this is called: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/loader/transform.py#38
And this is where the error is coming from: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/util/templates.py#39

dest is the run-on-projects current default setting from job-defaults, and source is the task's setting. In this case, they are merged together. They shouldn't be merged here since that defeats the purpose of having a default that can be overridden.

If I remove 'mozilla-central' from the job-defaults section in raptor-gve.yml, we get an empty 'run-on-projects' entry and the power tests are no longer scheduled.

Comment 6

[Greg Mierzwinski [:sparky]]

Attachment: gve_patch.patch

Pushing patch like this because phabricator is broken for me at the moment.

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1578723 - Make geckoview tests use their task-specific run-on-projects setting. r?#perftest

This patch prevents the job-defaults run-on-projects setting from overriding the task-specific run-on-projects setting. It also fixes a minor error in raptor-fennec.yml.

Comment 8

[Greg Mierzwinski [:sparky]]

Push to try with this patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=16f16d3b53e0ed72cd257ced8018a464ecf42ee8

Comment 9

[Andrew Halberstadt [:ahal]]

Yeah, it's not great that there's no way to "replace" a list or dict from job-defaults. I can't really think of any elegant way of solving this though.. I guess for edge cases like this there are always transforms.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Agreed -- job-defaults was intended as a "bridge" from "I have one task, here it is" to "I have three tasks and they're nearly identical", but wasn't expected to reach the far shore of "I have a whole bunch of tasks with complex differences between them."

Comment 11

[Greg Mierzwinski [:sparky]]

I'm thinking that we could solve this if we had a job-defaults section for setting default values that are used when a task doesn't have those values set, and a job-additionals section which are added/appended to a task's settings. Having this job-additionals section could also make our task definitions nicer (like the mozharness.config entries in raptor definitions).

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4aed8e10318f
Make geckoview tests use their task-specific run-on-projects setting. r=perftest-reviewers,stephendonner

Comment 13

[Greg Mierzwinski [:sparky]]

I'm going to leave this bug open after the patch lands so we know to come back and fix this hack once bug 1578804 is resolved.

Comment 14

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/4aed8e10318f

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:sparky, maybe it's time to close this bug?