Closed Bug 1523492 Opened 5 years ago Closed 5 years ago

Make Web Replay tests tier 2


(Core Graveyard :: Web Replay, enhancement)

Not set


(firefox67 fixed)

Tracking Status
firefox67 --- fixed


(Reporter: bhackett1024, Assigned: bhackett1024)



(2 files, 1 obsolete file)

Attached patch patch (obsolete) — Splinter Review

After bug 1517837 web replay tests are effectively tier 2 (they should be disabled in preference to backing out other changes), but this is not an official thing and patches can still be backed out due to breaking the tests, e.g. bug 1521189. Making the tests tier 2 should avoid future backouts due to these tests failing, and also reduce patch traffic enabling/disabling the tests. I think the attached patch will make this change, but I'm waiting on try server results before adding an r?.

Assignee: nobody → bhackett1024
Attachment #9039698 - Flags: review?(jmaher)
Comment on attachment 9039698 [details] [diff] [review]

Review of attachment 9039698 [details] [diff] [review]:

pretty close, this will be nice to see as tier2

::: taskcluster/ci/test/mochitest.yml
@@ +236,5 @@
> +    suite: mochitest/mochitest-devtools-webreplay
> +    treeherder-symbol: M(dt-wr)
> +    loopback-video: true
> +    tier: 2
> +    max-run-time: 5400

do we need 5400 for max run time?  I assume these run faster.

@@ +240,5 @@
> +    max-run-time: 5400
> +    mozharness:
> +        mochitest-flavor: chrome
> +    allow-software-gl-layers: false
> +

we should add in run-on-projects, similar to this:

Then we can do something similar to:
    macosx64-opt.*: [...]
    default: []

::: taskcluster/ci/test/test-sets.yml
@@ +294,5 @@
>      - web-platform-tests-wdspec
>      - xpcshell
> +macosx64-opt-tests:
> +    - mochitest-devtools-webreplay

if we do run-on-projects we don't need to define macosx64-opt-tests.
Attachment #9039698 - Flags: review?(jmaher) → review-
Attached patch patchSplinter Review

Updated patch. I used 900 as a max run time --- the tests took 7 minutes (420 seconds) on the last try run but we'd like to have room to add more tests in the future. This also turns the tests back on, as they were disabled by bug 1521189 and the fix (bug 1523364) has landed.

Attachment #9039698 - Attachment is obsolete: true
Attachment #9039842 - Flags: review?(jmaher)
Comment on attachment 9039842 [details] [diff] [review]

Review of attachment 9039842 [details] [diff] [review]:

hard to know 100% this is right, but from looking at what is currently running, this should do the trick.
Attachment #9039842 - Flags: review?(jmaher) → review+
Attached patch addendumSplinter Review

Sorry about the churn, but this addendum is needed to get the tests to run on mac platforms. The run-on-projects setting from the earlier patch does keep the tests from running on debug builds:

Attachment #9039897 - Flags: review?(jmaher)
Comment on attachment 9039897 [details] [diff] [review]

Review of attachment 9039897 [details] [diff] [review]:

apologies for not catching this, good find.
Attachment #9039897 - Flags: review?(jmaher) → review+
Pushed by
Make web replay tests tier 2, r=jmaher.
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.