1433075 - Surprisingly, h1 and h2 started to run on Windows

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Ionuț Goldan [:igoldan], Performance Sheriff]

Due to frequent timeout errors, we turned off h1 and h2 Talos tests from autoland and inbound. All went fine until backout from [1] caused h1 to run on Windows again.

Investigate how the backout re-enabled them, then shut them down again.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1431533#c47

Comment 1

[Ionuț Goldan [:igoldan], Performance Sheriff]

These are the associated jobs: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=windows%20t-e10s&fromchange=cc87ad81ff86109c7ea0187424fa9a8ed3b4af6c&tochange=cc87ad81ff86109c7ea0187424fa9a8ed3b4af6c

Comment 2

[Joel Maher ( :jmaher ) (UTC-4)]

odd, the definition to limit on windows and linux to try only is here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml#296

and the backout doesn't have anything related to taskcluster.

I see some action tasks (after the decision task ran we scheduled other tasks):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cc87ad81ff86109c7ea0187424fa9a8ed3b4af6c&filter-searchStr=decision

this is:
* run all talos tests
* run all missing tests

so I suspect that we need to fix the action tasks to respect what branches to run on.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It's a matter of definition, really -- an action can run a test if it was in the full task graph generated by the decision task.  If that task is omitted due to target task selection (which is where run-on-branches is seen) or optimization, that doesn't affect the full-task-graph that the actions are based on.  The whole idea of these actions is to override task-graph selection and optimization.

So, if we have tasks that should *never* run on windows, they shouldn't be in the full-task-graph at all, which means a change to the test-sets, rather than run-on-branches.

Comment 4

[Joel Maher ( :jmaher ) (UTC-4)]

thanks Dustin- that helps clarify things.  We often have tests we want to run on try only until they are stable- is there a better way to support this?  I would like to explore other options before removing these tests from a test-set.yml definition, although in a patch to test it we could always add it back.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

You could tag the affected tasks with some kind of not_part_of_all_talos_tests=True and not_part_of_missing_tests=True or something like that, and then modify the action implementation to skip tasks with that tag.  It does seem like a bit of a hack, though.  I assume these are tier-2 or tier-3?  If so, won't they just not appear on treeherder anyway?

Comment 6

[Joel Maher ( :jmaher ) (UTC-4)]

these are tier-1 and they consume too many cpu hours on our physical workers- so they cannot run by default. I think we should just remove them from the test_sets.yml, that will be easier to 'debug'.

Comment 7

[Joel Maher ( :jmaher ) (UTC-4)]

Attachment: remove h1/h2 from test_sets.yml

:rwood, let me know if this looks like a reasonable solution.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Sounds good.  I think the tools available here are tiers, run-on-projects, and test-sets, and it's simplest to use the tool that fits the needs most closely, rather than creating a new tool.

Comment 9

[Joel Maher ( :jmaher ) (UTC-4)]

Attachment: remove h1/h2 from test_sets.yml

Comment 10

[Robert Wood [:rwood]]

Comment on attachment 8945523 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945523 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/test-sets.yml
@@ -270,5 @@
>      - talos-tp6
>      - talos-tp6-stylo-threads
>      - talos-speedometer
> -    - talos-h1
> -    - talos-h2

This removes it from mac, we want to leave it on mac (correct?) and remove it from windows and linux

Comment 11

[Joel Maher ( :jmaher ) (UTC-4)]

I thought this left the mac ones alone, apologies if I overlooked that

Comment 12

[Joel Maher ( :jmaher ) (UTC-4)]

Attachment: remove h1/h2 from test_sets.yml

this is for linux (including -qr) and windows- we only run on macosx now.

Comment 13

[Robert Wood [:rwood]]

Comment on attachment 8945807 [details] [diff] [review]
remove h1/h2 from test_sets.yml

Review of attachment 8945807 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you sir!

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7ff05f538453
removed h1/h2 from test-sets so they are not accidentally run. r=rwood

Comment 15

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/7ff05f538453

Comment 16

[Intermittent Failures Robot]

8 failures in 701 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 8

Platform breakdown:
* windows7-32: 4
* windows10-64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433075&startday=2018-01-22&endday=2018-01-28&tree=all

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/977314b2679d