stylo: Retire stylo-failures.md and enable more mochitests which don't fail a lot

RESOLVED FIXED in Firefox 57

Status

Testing
Mochitest
P1
normal
RESOLVED FIXED
3 months ago
2 months ago

People

(Reporter: xidorn, Assigned: xidorn)

Tracking

(Blocks: 1 bug)

Trunk
mozilla57
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox57 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

We want to enable all mochitests for stylo sooner than later, so that we can catch improvement and regression on a larger range. At the same time, we would need to retire stylo-failures.md and use fails-if instead for style system tests as well.

stylo-failures.md was used because many tests in layout/style/test could actually contain lots of different failures, so a coarse-grained "fails-if" isn't good enough to catch improvement and regression. But now, majority of failures in stylo-failures.md have been fixed.
Depends on: 1374872, 1383980, 1356087
Assignee: nobody → xidorn+moz
Depends on: 1380259
Depends on: 1382077
Depends on: 1302513
Priority: -- → P1
Depends on: 1367299
Depends on: 1387080
No longer depends on: 1387080
Summary: stylo: Retire stylo-failures.md and enable all mochitests → stylo: Retire stylo-failures.md and enable more mochitests which don't fail a lot
I'm not going to enable all... at least browser-chrome and a11y still seem to fail a lot. We can try them later.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=57cc1c3e3c32d9742e9b20be7d74e05ee17755e1
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Blocks: 1387993
Blocks: 1387995
Comment on attachment 8894444 [details]
Bug 1383992 part 1 - Disable failing mochitests.

https://reviewboard.mozilla.org/r/165628/#review170698
Attachment #8894444 - Flags: review?(cam) → review+

Comment 6

2 months ago
mozreview-review
Comment on attachment 8894445 [details]
Bug 1383992 part 2 - Enable some more mochitests for stylo.

https://reviewboard.mozilla.org/r/165630/#review170746

::: taskcluster/ci/test/test-sets.yml:95
(Diff revision 1)
> +    - mochitest-browser-screenshots
> +    - mochitest-chrome
> +    - mochitest-clipboard
> +    - mochitest-gpu
> +    - mochitest-media
> +    - mochitest-webgl

my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase?

::: taskcluster/ci/test/tests.yml:677
(Diff revision 1)
>                  extra-options:
>                      - --mochitest-suite=chrome
>      tier:
>          by-test-platform:
>              windows7-32.*: 2
> +            linux64-stylo/debug: 3

will opt be run on as tier1 on all trees?

::: taskcluster/ci/test/tests.yml:690
(Diff revision 1)
>      virtualization: hardware
>      instance-size: xlarge
>      run-on-projects:
>          by-test-platform:
>              windows.*: ['try']
> +            linux64-stylo/debug: ['try']

will opt be run on all trees?

::: taskcluster/ci/test/tests.yml:728
(Diff revision 1)
>                  extra-options:
>                      - --mochitest-suite=plain-clipboard,chrome-clipboard,browser-chrome-clipboard
>      tier:
>          by-test-platform:
>              windows7-32.*: 2
> +            linux64-stylo/debug: 3

this is tier-1 for linux64-stylo/opt?
Attachment #8894445 - Flags: review?(jmaher) → review-
(Assignee)

Comment 7

2 months ago
mozreview-review-reply
Comment on attachment 8894445 [details]
Bug 1383992 part 2 - Enable some more mochitests for stylo.

https://reviewboard.mozilla.org/r/165630/#review170746

> my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase?

Let's drop the whole stylo test set from osx then?

> will opt be run on as tier1 on all trees?

Yes, because we don't have problem on running opt build with these tests. They currently fail on debug build because of bug 1384701. See bug 1387993. If you are concerned about that we are having different test coverage on debug vs. opt, I guess we can universally disable these tests?
(Assignee)

Comment 8

2 months ago
mozreview-review-reply
Comment on attachment 8894445 [details]
Bug 1383992 part 2 - Enable some more mochitests for stylo.

https://reviewboard.mozilla.org/r/165630/#review170746

> Let's drop the whole stylo test set from osx then?

So all mochitests runs for 362 mins in total for debug build, and 188 mins for opt build, from the try push. The replaced tests currently take 24 mins and 9 mins respectively (from a recent m-c commit). That means around 338 mins and 179 mins increment. Probably doesn't sound fine?
Discussed with bholley, if we are concerned about the load on mac platform, probably we can only enable them on m-c for mac? That means we would regress some of our test coverage on mac (since we are retiring mochitest-style and mochitest-chrome-style which currently run on autoland), but that's probably not a big problem.

jmaher, would you be happy with that?

Tomcat, would sheriff be fine with that?
Flags: needinfo?(jmaher)
Flags: needinfo?(cbook)
We went through a bit of trouble to get the current set of Stylo tests running on macOS, so it would be a bit sad to disable them...  Maybe we can pre-seed SETA for macOS, like we are doing for Windows (bug 1386806)?
Comment hidden (mozreview-request)
KWierso answered sheriff's point. It seems as far as a task is tier-2, it is fine to only run on m-c and try.
Flags: needinfo?(cbook)
The updated patch moves all the new mochitests into tier-2 and make them only run on m-c and try for macosx64-stylo platform.
yes, m-c+try is good for osx; other platforms that we have tests running in AWS have better capacity, but talos and macosx* are hardware limited.
Flags: needinfo?(jmaher)

Comment 15

2 months ago
mozreview-review
Comment on attachment 8894445 [details]
Bug 1383992 part 2 - Enable some more mochitests for stylo.

https://reviewboard.mozilla.org/r/165630/#review171194

looks much better
Attachment #8894445 - Flags: review?(jmaher) → review+

Comment 16

2 months ago
mozreview-review
Comment on attachment 8894445 [details]
Bug 1383992 part 2 - Enable some more mochitests for stylo.

https://reviewboard.mozilla.org/r/165630/#review171492

::: testing/mozharness/configs/unittests/linux_unittest.py
(Diff revision 2)
>          "mochitest-devtools-chrome-coverage": ["--flavor=browser", "--subsuite=devtools", "--chunk-by-runtime", "--timeout=1200"],
>          "jetpack-package": ["--flavor=jetpack-package"],
>          "jetpack-package-clipboard": ["--flavor=jetpack-package", "--subsuite=clipboard"],
>          "jetpack-addon": ["--flavor=jetpack-addon"],
>          "a11y": ["--flavor=a11y"],
> -        "plain-style": ["--failure-pattern-file=stylo-failures.md", "layout/style/test", "dom/smil/test", "dom/animation/test"],

Please also remove the same entries from `mac_unittest.py` and `win_taskcluster_unittest.py`.
Depends on: 1385027
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 19

2 months ago
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s c12dd8a82fb9 -d c95150c0fe9d: rebasing 412513:c12dd8a82fb9 "Bug 1383992 part 1 - Disable failing mochitests. r=heycam"
merging dom/base/test/mochitest.ini
rebasing 412514:1a79deb8189a "Bug 1383992 part 2 - Enable some more mochitests for stylo. r=jmaher" (tip)
merging taskcluster/ci/test/test-sets.yml
merging taskcluster/ci/test/tests.yml
merging testing/mozharness/configs/unittests/linux_unittest.py
warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 20

2 months ago
Pushed by xquan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/76dafa6645cb
part 1 - Disable failing mochitests. r=heycam
https://hg.mozilla.org/integration/autoland/rev/baf282c4df0c
part 2 - Enable some more mochitests for stylo. r=jmaher
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ffeb51629a2668bcd32b4b701647521a6ae27277

Comment 22

2 months ago
Pushed by xquan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d0e338649b01
followup - Fix tests.yml.

Comment 23

2 months ago
Pushed by xquan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b886db98d2bc
followup - Update mochitest expectation.

Comment 24

2 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/76dafa6645cb
https://hg.mozilla.org/mozilla-central/rev/baf282c4df0c
https://hg.mozilla.org/mozilla-central/rev/d0e338649b01
https://hg.mozilla.org/mozilla-central/rev/b886db98d2bc
Status: NEW → RESOLVED
Last Resolved: 2 months ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.