Closed Bug 1383992 Opened 7 years ago Closed 7 years ago

stylo: Retire stylo-failures.md and enable more mochitests which don't fail a lot

Categories

(Testing :: Mochitest, defect, P1)

Product:
Testing
Component:
Mochitest
Type:
defect

Tracking

(firefox57 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla57
Tracking Flags:
Tracking Status
firefox57 --- fixed

People

(Reporter: xidorn, Assigned: xidorn)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Bug 1383992 part 1 - Disable failing mochitests.
59 bytes, text/x-review-board-request
heycam
: review+
Details
Bug 1383992 part 2 - Enable some more mochitests for stylo.
59 bytes, text/x-review-board-request
jmaher
: review+
Details
We want to enable all mochitests for stylo sooner than later, so that we can catch improvement and regression on a larger range. At the same time, we would need to retire stylo-failures.md and use fails-if instead for style system tests as well. stylo-failures.md was used because many tests in layout/style/test could actually contain lots of different failures, so a coarse-grained "fails-if" isn't good enough to catch improvement and regression. But now, majority of failures in stylo-failures.md have been fixed.
Depends on: 1374872, 1383980, 1356087
Assignee: nobody → xidorn+moz
Depends on: 1380259
Depends on: 1382077
Depends on: 1302513
Priority: -- → P1
Depends on: 1367299
Depends on: 1387080
No longer depends on: 1387080
Summary: stylo: Retire stylo-failures.md and enable all mochitests → stylo: Retire stylo-failures.md and enable more mochitests which don't fail a lot
I'm not going to enable all... at least browser-chrome and a11y still seem to fail a lot. We can try them later.
Blocks: 1387993
Blocks: 1387995
Comment on attachment 8894444 [details] Bug 1383992 part 1 - Disable failing mochitests. https://reviewboard.mozilla.org/r/165628/#review170698
Attachment #8894444 - Flags: review?(cam) → review+
Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 ::: taskcluster/ci/test/test-sets.yml:95 (Diff revision 1) > + - mochitest-browser-screenshots > + - mochitest-chrome > + - mochitest-clipboard > + - mochitest-gpu > + - mochitest-media > + - mochitest-webgl my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase? ::: taskcluster/ci/test/tests.yml:677 (Diff revision 1) > extra-options: > - --mochitest-suite=chrome > tier: > by-test-platform: > windows7-32.*: 2 > + linux64-stylo/debug: 3 will opt be run on as tier1 on all trees? ::: taskcluster/ci/test/tests.yml:690 (Diff revision 1) > virtualization: hardware > instance-size: xlarge > run-on-projects: > by-test-platform: > windows.*: ['try'] > + linux64-stylo/debug: ['try'] will opt be run on all trees? ::: taskcluster/ci/test/tests.yml:728 (Diff revision 1) > extra-options: > - --mochitest-suite=plain-clipboard,chrome-clipboard,browser-chrome-clipboard > tier: > by-test-platform: > windows7-32.*: 2 > + linux64-stylo/debug: 3 this is tier-1 for linux64-stylo/opt?
Attachment #8894445 - Flags: review?(jmaher) → review-
Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 > my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase? Let's drop the whole stylo test set from osx then? > will opt be run on as tier1 on all trees? Yes, because we don't have problem on running opt build with these tests. They currently fail on debug build because of bug 1384701. See bug 1387993. If you are concerned about that we are having different test coverage on debug vs. opt, I guess we can universally disable these tests?
Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 > Let's drop the whole stylo test set from osx then? So all mochitests runs for 362 mins in total for debug build, and 188 mins for opt build, from the try push. The replaced tests currently take 24 mins and 9 mins respectively (from a recent m-c commit). That means around 338 mins and 179 mins increment. Probably doesn't sound fine?
Discussed with bholley, if we are concerned about the load on mac platform, probably we can only enable them on m-c for mac? That means we would regress some of our test coverage on mac (since we are retiring mochitest-style and mochitest-chrome-style which currently run on autoland), but that's probably not a big problem. jmaher, would you be happy with that? Tomcat, would sheriff be fine with that?
Flags: needinfo?(jmaher)
Flags: needinfo?(cbook)
We went through a bit of trouble to get the current set of Stylo tests running on macOS, so it would be a bit sad to disable them... Maybe we can pre-seed SETA for macOS, like we are doing for Windows (bug 1386806)?
KWierso answered sheriff's point. It seems as far as a task is tier-2, it is fine to only run on m-c and try.
Flags: needinfo?(cbook)
The updated patch moves all the new mochitests into tier-2 and make them only run on m-c and try for macosx64-stylo platform.
yes, m-c+try is good for osx; other platforms that we have tests running in AWS have better capacity, but talos and macosx* are hardware limited.
Flags: needinfo?(jmaher)
Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review171194 looks much better
Attachment #8894445 - Flags: review?(jmaher) → review+
Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review171492 ::: testing/mozharness/configs/unittests/linux_unittest.py (Diff revision 2) > "mochitest-devtools-chrome-coverage": ["--flavor=browser", "--subsuite=devtools", "--chunk-by-runtime", "--timeout=1200"], > "jetpack-package": ["--flavor=jetpack-package"], > "jetpack-package-clipboard": ["--flavor=jetpack-package", "--subsuite=clipboard"], > "jetpack-addon": ["--flavor=jetpack-addon"], > "a11y": ["--flavor=a11y"], > - "plain-style": ["--failure-pattern-file=stylo-failures.md", "layout/style/test", "dom/smil/test", "dom/animation/test"], Please also remove the same entries from `mac_unittest.py` and `win_taskcluster_unittest.py`.
Depends on: 1385027
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. hg error in cmd: hg rebase -s c12dd8a82fb9 -d c95150c0fe9d: rebasing 412513:c12dd8a82fb9 "Bug 1383992 part 1 - Disable failing mochitests. r=heycam" merging dom/base/test/mochitest.ini rebasing 412514:1a79deb8189a "Bug 1383992 part 2 - Enable some more mochitests for stylo. r=jmaher" (tip) merging taskcluster/ci/test/test-sets.yml merging taskcluster/ci/test/tests.yml merging testing/mozharness/configs/unittests/linux_unittest.py warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark') unresolved conflicts (see hg resolve, then hg rebase --continue)
Pushed by xquan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/76dafa6645cb part 1 - Disable failing mochitests. r=heycam https://hg.mozilla.org/integration/autoland/rev/baf282c4df0c part 2 - Enable some more mochitests for stylo. r=jmaher
Pushed by xquan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b886db98d2bc followup - Update mochitest expectation.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: