Closed Bug 1433075 Opened 4 years ago Closed 4 years ago

Surprisingly, h1 and h2 started to run on Windows


(Testing :: Talos, enhancement)

Not set


(firefox59 fixed, firefox60 fixed)

Tracking Status
firefox59 --- fixed
firefox60 --- fixed


(Reporter: igoldan, Assigned: jmaher)




(1 file, 2 obsolete files)

Due to frequent timeout errors, we turned off h1 and h2 Talos tests from autoland and inbound. All went fine until backout from [1] caused h1 to run on Windows again.

Investigate how the backout re-enabled them, then shut them down again.

odd, the definition to limit on windows and linux to try only is here:

and the backout doesn't have anything related to taskcluster.

I see some action tasks (after the decision task ran we scheduled other tasks):

this is:
* run all talos tests
* run all missing tests

so I suspect that we need to fix the action tasks to respect what branches to run on.
It's a matter of definition, really -- an action can run a test if it was in the full task graph generated by the decision task.  If that task is omitted due to target task selection (which is where run-on-branches is seen) or optimization, that doesn't affect the full-task-graph that the actions are based on.  The whole idea of these actions is to override task-graph selection and optimization.

So, if we have tasks that should *never* run on windows, they shouldn't be in the full-task-graph at all, which means a change to the test-sets, rather than run-on-branches.
thanks Dustin- that helps clarify things.  We often have tests we want to run on try only until they are stable- is there a better way to support this?  I would like to explore other options before removing these tests from a test-set.yml definition, although in a patch to test it we could always add it back.
You could tag the affected tasks with some kind of not_part_of_all_talos_tests=True and not_part_of_missing_tests=True or something like that, and then modify the action implementation to skip tasks with that tag.  It does seem like a bit of a hack, though.  I assume these are tier-2 or tier-3?  If so, won't they just not appear on treeherder anyway?
these are tier-1 and they consume too many cpu hours on our physical workers- so they cannot run by default. I think we should just remove them from the test_sets.yml, that will be easier to 'debug'.
Attached patch remove h1/h2 from test_sets.yml (obsolete) — Splinter Review
:rwood, let me know if this looks like a reasonable solution.
Assignee: nobody → jmaher
Attachment #8945496 - Flags: review?(rwood)
Sounds good.  I think the tools available here are tiers, run-on-projects, and test-sets, and it's simplest to use the tool that fits the needs most closely, rather than creating a new tool.
Attached patch remove h1/h2 from test_sets.yml (obsolete) — Splinter Review
Attachment #8945496 - Attachment is obsolete: true
Attachment #8945496 - Flags: review?(rwood)
Attachment #8945523 - Flags: review?(rwood)
Comment on attachment 8945523 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945523 [details] [diff] [review]:

::: taskcluster/ci/test/test-sets.yml
@@ -270,5 @@
>      - talos-tp6
>      - talos-tp6-stylo-threads
>      - talos-speedometer
> -    - talos-h1
> -    - talos-h2

This removes it from mac, we want to leave it on mac (correct?) and remove it from windows and linux
Attachment #8945523 - Flags: review?(rwood) → review-
I thought this left the mac ones alone, apologies if I overlooked that
this is for linux (including -qr) and windows- we only run on macosx now.
Attachment #8945523 - Attachment is obsolete: true
Attachment #8945807 - Flags: review?(rwood)
Comment on attachment 8945807 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945807 [details] [diff] [review]:

Thank you sir!
Attachment #8945807 - Flags: review?(rwood) → review+
Pushed by
removed h1/h2 from test-sets so they are not accidentally run. r=rwood
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
You need to log in before you can comment on or make changes to this bug.