Make Web Replay tests tier 2

RESOLVED FIXED in Firefox 67

Status

()

enhancement
RESOLVED FIXED
5 months ago
5 months ago

People

(Reporter: bhackett, Assigned: bhackett)

Tracking

unspecified
mozilla67
Points:
---

Firefox Tracking Flags

(firefox67 fixed)

Details

Attachments

(2 attachments, 1 obsolete attachment)

Assignee

Description

5 months ago
Posted patch patch (obsolete) — Splinter Review

After bug 1517837 web replay tests are effectively tier 2 (they should be disabled in preference to backing out other changes), but this is not an official thing and patches can still be backed out due to breaking the tests, e.g. bug 1521189. Making the tests tier 2 should avoid future backouts due to these tests failing, and also reduce patch traffic enabling/disabling the tests. I think the attached patch will make this change, but I'm waiting on try server results before adding an r?.

Assignee

Updated

5 months ago
Assignee: nobody → bhackett1024
Assignee

Updated

5 months ago
Attachment #9039698 - Flags: review?(jmaher)
Comment on attachment 9039698 [details] [diff] [review]
patch

Review of attachment 9039698 [details] [diff] [review]:
-----------------------------------------------------------------

pretty close, this will be nice to see as tier2

::: taskcluster/ci/test/mochitest.yml
@@ +236,5 @@
> +    suite: mochitest/mochitest-devtools-webreplay
> +    treeherder-symbol: M(dt-wr)
> +    loopback-video: true
> +    tier: 2
> +    max-run-time: 5400

do we need 5400 for max run time?  I assume these run faster.

@@ +240,5 @@
> +    max-run-time: 5400
> +    mozharness:
> +        mochitest-flavor: chrome
> +    allow-software-gl-layers: false
> +

we should add in run-on-projects, similar to this:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#44

Then we can do something similar to:
run-on-project:
  by-test-plaform:
    macosx64-opt.*: [...]
    default: []

::: taskcluster/ci/test/test-sets.yml
@@ +294,5 @@
>      - web-platform-tests-wdspec
>      - xpcshell
>  
> +macosx64-opt-tests:
> +    - mochitest-devtools-webreplay

if we do run-on-projects we don't need to define macosx64-opt-tests.
Attachment #9039698 - Flags: review?(jmaher) → review-
Assignee

Comment 3

5 months ago
Posted patch patchSplinter Review

Updated patch. I used 900 as a max run time --- the tests took 7 minutes (420 seconds) on the last try run but we'd like to have room to add more tests in the future. This also turns the tests back on, as they were disabled by bug 1521189 and the fix (bug 1523364) has landed.

Attachment #9039698 - Attachment is obsolete: true
Attachment #9039842 - Flags: review?(jmaher)
Comment on attachment 9039842 [details] [diff] [review]
patch

Review of attachment 9039842 [details] [diff] [review]:
-----------------------------------------------------------------

hard to know 100% this is right, but from looking at what is currently running, this should do the trick.
Attachment #9039842 - Flags: review?(jmaher) → review+
Assignee

Comment 5

5 months ago
Posted patch addendumSplinter Review

Sorry about the churn, but this addendum is needed to get the tests to run on mac platforms. The run-on-projects setting from the earlier patch does keep the tests from running on debug builds:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf8acbf5cccd

Attachment #9039897 - Flags: review?(jmaher)
Comment on attachment 9039897 [details] [diff] [review]
addendum

Review of attachment 9039897 [details] [diff] [review]:
-----------------------------------------------------------------

apologies for not catching this, good find.
Attachment #9039897 - Flags: review?(jmaher) → review+

Comment 7

5 months ago
Pushed by bhackett@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/91f5131e326a
Make web replay tests tier 2, r=jmaher.

Comment 8

5 months ago
bugherder
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.