Closed Bug 1433075 Opened 2 years ago Closed 2 years ago

Surprisingly, h1 and h2 started to run on Windows

Categories

(Testing :: Talos, enhancement)

enhancement
Not set

Tracking

(firefox59 fixed, firefox60 fixed)

RESOLVED FIXED
mozilla60
Tracking Status
firefox59 --- fixed
firefox60 --- fixed

People

(Reporter: igoldan, Assigned: jmaher)

References

Details

Attachments

(1 file, 2 obsolete files)

Due to frequent timeout errors, we turned off h1 and h2 Talos tests from autoland and inbound. All went fine until backout from [1] caused h1 to run on Windows again.

Investigate how the backout re-enabled them, then shut them down again.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1431533#c47
odd, the definition to limit on windows and linux to try only is here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml#296

and the backout doesn't have anything related to taskcluster.

I see some action tasks (after the decision task ran we scheduled other tasks):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cc87ad81ff86109c7ea0187424fa9a8ed3b4af6c&filter-searchStr=decision

this is:
* run all talos tests
* run all missing tests

so I suspect that we need to fix the action tasks to respect what branches to run on.
It's a matter of definition, really -- an action can run a test if it was in the full task graph generated by the decision task.  If that task is omitted due to target task selection (which is where run-on-branches is seen) or optimization, that doesn't affect the full-task-graph that the actions are based on.  The whole idea of these actions is to override task-graph selection and optimization.

So, if we have tasks that should *never* run on windows, they shouldn't be in the full-task-graph at all, which means a change to the test-sets, rather than run-on-branches.
thanks Dustin- that helps clarify things.  We often have tests we want to run on try only until they are stable- is there a better way to support this?  I would like to explore other options before removing these tests from a test-set.yml definition, although in a patch to test it we could always add it back.
You could tag the affected tasks with some kind of not_part_of_all_talos_tests=True and not_part_of_missing_tests=True or something like that, and then modify the action implementation to skip tasks with that tag.  It does seem like a bit of a hack, though.  I assume these are tier-2 or tier-3?  If so, won't they just not appear on treeherder anyway?
these are tier-1 and they consume too many cpu hours on our physical workers- so they cannot run by default. I think we should just remove them from the test_sets.yml, that will be easier to 'debug'.
Attached patch remove h1/h2 from test_sets.yml (obsolete) — Splinter Review
:rwood, let me know if this looks like a reasonable solution.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8945496 - Flags: review?(rwood)
Sounds good.  I think the tools available here are tiers, run-on-projects, and test-sets, and it's simplest to use the tool that fits the needs most closely, rather than creating a new tool.
Attached patch remove h1/h2 from test_sets.yml (obsolete) — Splinter Review
Attachment #8945496 - Attachment is obsolete: true
Attachment #8945496 - Flags: review?(rwood)
Attachment #8945523 - Flags: review?(rwood)
Comment on attachment 8945523 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945523 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/test-sets.yml
@@ -270,5 @@
>      - talos-tp6
>      - talos-tp6-stylo-threads
>      - talos-speedometer
> -    - talos-h1
> -    - talos-h2

This removes it from mac, we want to leave it on mac (correct?) and remove it from windows and linux
Attachment #8945523 - Flags: review?(rwood) → review-
I thought this left the mac ones alone, apologies if I overlooked that
this is for linux (including -qr) and windows- we only run on macosx now.
Attachment #8945523 - Attachment is obsolete: true
Attachment #8945807 - Flags: review?(rwood)
Comment on attachment 8945807 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945807 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you sir!
Attachment #8945807 - Flags: review?(rwood) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7ff05f538453
removed h1/h2 from test-sets so they are not accidentally run. r=rwood
https://hg.mozilla.org/mozilla-central/rev/7ff05f538453
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
You need to log in before you can comment on or make changes to this bug.