Closed Bug 1425589 Opened 7 years ago Closed 7 years ago

Get web-platform-tests running on linux64-qr builds in automation

Categories

(Core :: Graphics: WebRender, enhancement, P2)

Product:
Core
Component:
Graphics: WebRender
Version:
Other Branch
Platform:
x86_64
Linux
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla61
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: kats, Assigned: kats)

References

Details

(Whiteboard: [gfx-noted][triage][wptsync upstream])

Attachments

(2 files, 1 obsolete file)

Bug 1425589 - Update web-platform-tests annotations for webrender on linux.
59 bytes, text/x-review-board-request
jgraham
: review+
Details
Bug 1425589 - Turn on wpt on linux64-qr.
59 bytes, text/x-review-board-request
jgraham
: review+
Details
We are not running these tests yet in automation on linux64-qr. We should do that. Here's a try push with what it looks like now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4faa2533aae34544691a185c78d102fb88749b39
Comment on attachment 8950969 [details] Bug 1425589: Webcompat tests on linux. After this, we have failures with intermittent bugs filed against them and timeouts. https://reviewboard.mozilla.org/r/220232/#review226106 ::: commit-message-16547:1 (Diff revision 1) > +Bug 1425589: Webcompat tests on linux. After this, we have failures with intermittent bugs filed against them and timeouts. r?kats s/Webcompat tests on linux./Annotation web-platform reftests for WebRender on Linux./ Also we should probably move this to a dependent bug if we're landing this patch right away, since it doesn't actually fix bug 1425589.
Attachment #8950969 - Flags: review?(bugmail) → review+
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcf5cd02189240b977b73906dd79dc697e814979 has a bunch of retriggers. Of the Wr (reftest) jobs the Wr1 chunk is failing at a 100% rate so that still needs additional annotating before we can land that. The Wd (wdspec) and Wpt (mochitest-style tests) jobs also have failures that will need to be addressed, but we can defer turning those on.
From what I can tell, one test fails in Wr1 optimized all the time (object-fit-contain-png-001c.html) and none of them fail all the time in debug.
Yeah, nondeterministic results are hard to deal with. We have only three possible options: 1) Find and fix the nondeterminism 2) Skip the tests (or mark them as random) - not sure if this is possible with web-platform-tests? 3) Don't enable the test suite
Apparently it is possible to disable web platform tests. Here's a try push with more stuff annotated/disabled, hopefully I got everything: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6f8acf2496fdc2dba2266730a8e1f1ccf96d542
Assignee: nobody → bugmail
Comment on attachment 8960758 [details] Bug 1425589 - Turn on wpt on linux64-qr. https://reviewboard.mozilla.org/r/229502/#review235386
Attachment #8960758 - Flags: review?(james) → review+
Comment on attachment 8960757 [details] Bug 1425589 - Update web-platform-tests annotations for webrender on linux. https://reviewboard.mozilla.org/r/229500/#review235388 So in general I'm not sure if these tests are disabled for instability (fine) or for failing (not the recommended approach). Also, in order to update metadata correctly when we import new tests, a change is required in testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py::update_properties to ensure that we consider webrender as a possible varaible when updating metadata (hopefully we can remove stylo there now since we are ending the non-stylo configuration). ::: testing/web-platform/meta/css/css-backgrounds/background-size-027.html.ini:4 (Diff revision 1) > [background-size-027.html] > prefs: [dom.webcomponents.shadowdom.enabled:true] > + disabled: > + if webrender: bug 1425588 So is this disabled for instability? Generally we don't mark web-platform-tests as disabled unless they aren't stable. If the test happens to fail we mark it as expected: FAIL so we can continue to check that it doesn't crash or whatever and notice as soon as soon as it starts to pass.
Attachment #8960757 - Flags: review?(james) → review-
(In reply to James Graham [:jgraham] from comment #15) > Also, in order to update metadata correctly when we import new tests, a > change is required in > testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py:: > update_properties to ensure that we consider webrender as a possible > varaible when updating metadata Is there any documentation, or can you explain, what the function is supposed to return? A tuple containing a [] and a {} with overlapping properties isn't obvious to me. Should webrender go in the [] or the {} or both? > > So is this disabled for instability? Generally we don't mark > web-platform-tests as disabled unless they aren't stable. If the test > happens to fail we mark it as expected: FAIL so we can continue to check > that it doesn't crash or whatever and notice as soon as soon as it starts to > pass. Yes, all the ones I disabled were because of instability. Bug 1438284 which landed earlier took care of most of the stable results (unexpected-fail/unexpected-pass etc.)
Flags: needinfo?(james)
Oh wow yes, that's *surprisingly* non-obvious, isn't it. The first list is all the properties that are considered, and the second is a set of properties that only take boolean values. s/"stylo"/"stylo", "webrender"/ throughout that function should do the right thing.
Flags: needinfo?(james)
Comment on attachment 8960757 [details] Bug 1425589 - Update web-platform-tests annotations for webrender on linux. https://reviewboard.mozilla.org/r/229500/#review235440 Thanks!
Attachment #8960757 - Flags: review?(james) → review+
Thanks for the quick reviews! :)
Pushed by kgupta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6e0605add1f5 Update web-platform-tests annotations for webrender on linux. r=jgraham https://hg.mozilla.org/integration/autoland/rev/08bc853b01ae Turn on wpt on linux64-qr. r=jgraham
https://hg.mozilla.org/mozilla-central/rev/6e0605add1f5 https://hg.mozilla.org/mozilla-central/rev/08bc853b01ae
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Created web-platform-tests PR https://github.com/w3c/web-platform-tests/pull/10307 for changes under testing/web-platform/tests
Whiteboard: [gfx-noted][triage] → [gfx-noted][triage][wptsync upstream]
See Also: → 1489700
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: