1432872 - Do not run test-verify on beta

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown] (less available until Feb 5)]

As raised in bug 1431154: Some test types (screenshots) don't normally run on beta, so it seems wrong to verify such tests on beta. Also, when we promote changes to beta quickly, a single verification failure may repeat on multiple branches, triggering stockwell triage with minimal value.

Let's skip test-verify outside of [inbound/autoland/central].

Comment 1

[Geoff Brown [:gbrown] (less available until Feb 5)]

Attachment: run test-verify only on central, inbound, autoland, and try

Does that seem reasonable?

Comment 2

[Ryan VanderMeulen [:RyanVM][PTO Jan 25 - Feb 2]]

Comment on attachment 8945182 [details] [diff] [review]
run test-verify only on central, inbound, autoland, and try

Review of attachment 8945182 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/misc.yml
@@ +70,5 @@
>          by-test-platform:
>              # do not run on ccov; see also the enable_code_coverage transform
>              linux64-ccov/.*: []
> +            # do not run on beta or release: usually just confirms earlier results
> +            default: ['mozilla-central', 'autoland', 'mozilla-inbound', 'try']

Couldn't this be ['trunk', 'try'] instead? Anyway, I'm fine in principle with this change.

::: taskcluster/ci/test/web-platform.yml
@@ +137,5 @@
>          by-test-platform:
>              # do not run on ccov; see also the enable_code_coverage transform
>              linux64-ccov/.*: []
> +            # do not run on beta or release: usually just confirms earlier results
> +            default: ['mozilla-central', 'autoland', 'mozilla-inbound', 'try']  

Likewise.

Comment 3

[Geoff Brown [:gbrown] (less available until Feb 5)]

Thanks! Will change to 'trunk'...

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/95cfbed0308f
Stop running test-verify on beta and release; r=ryanvm

Comment 5

[Ryan VanderMeulen [:RyanVM][PTO Jan 25 - Feb 2]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e6bdb360e084

Comment 6

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/95cfbed0308f