1523492 - Make Web Replay tests tier 2

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, enhancement)

Description

[Brian Hackett [Laid off!]]

Attachment: patch

After bug 1517837 web replay tests are effectively tier 2 (they should be disabled in preference to backing out other changes), but this is not an official thing and patches can still be backed out due to breaking the tests, e.g. bug 1521189. Making the tests tier 2 should avoid future backouts due to these tests failing, and also reduce patch traffic enabling/disabling the tests. I think the attached patch will make this change, but I'm waiting on try server results before adding an r?.

Comment 1

[Brian Hackett [Laid off!]]

This looks good on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e2f187203cbe12247934cc676c5ccb7239730a9

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9039698 [details] [diff] [review]
patch

Review of attachment 9039698 [details] [diff] [review]:
-----------------------------------------------------------------

pretty close, this will be nice to see as tier2

::: taskcluster/ci/test/mochitest.yml
@@ +236,5 @@
> +    suite: mochitest/mochitest-devtools-webreplay
> +    treeherder-symbol: M(dt-wr)
> +    loopback-video: true
> +    tier: 2
> +    max-run-time: 5400

do we need 5400 for max run time?  I assume these run faster.

@@ +240,5 @@
> +    max-run-time: 5400
> +    mozharness:
> +        mochitest-flavor: chrome
> +    allow-software-gl-layers: false
> +

we should add in run-on-projects, similar to this:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#44

Then we can do something similar to:
run-on-project:
  by-test-plaform:
    macosx64-opt.*: [...]
    default: []

::: taskcluster/ci/test/test-sets.yml
@@ +294,5 @@
>      - web-platform-tests-wdspec
>      - xpcshell
>  
> +macosx64-opt-tests:
> +    - mochitest-devtools-webreplay

if we do run-on-projects we don't need to define macosx64-opt-tests.

Comment 3

[Brian Hackett [Laid off!]]

Attachment: patch

Updated patch. I used 900 as a max run time --- the tests took 7 minutes (420 seconds) on the last try run but we'd like to have room to add more tests in the future. This also turns the tests back on, as they were disabled by bug 1521189 and the fix (bug 1523364) has landed.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9039842 [details] [diff] [review]
patch

Review of attachment 9039842 [details] [diff] [review]:
-----------------------------------------------------------------

hard to know 100% this is right, but from looking at what is currently running, this should do the trick.

Comment 5

[Brian Hackett [Laid off!]]

Attachment: addendum

Sorry about the churn, but this addendum is needed to get the tests to run on mac platforms. The run-on-projects setting from the earlier patch does keep the tests from running on debug builds:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf8acbf5cccd

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9039897 [details] [diff] [review]
addendum

Review of attachment 9039897 [details] [diff] [review]:
-----------------------------------------------------------------

apologies for not catching this, good find.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/91f5131e326a
Make web replay tests tier 2, r=jmaher.

Comment 8

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/91f5131e326a