Get web-platform-tests running on linux64-qr builds in automation

RESOLVED FIXED in Firefox 61

Status

()

enhancement
P2
normal
RESOLVED FIXED
a year ago
9 months ago

People

(Reporter: kats, Assigned: kats)

Tracking

(Blocks 2 bugs)

Other Branch
mozilla61
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox61 fixed)

Details

(Whiteboard: [gfx-noted][triage][wptsync upstream])

Attachments

(2 attachments, 1 obsolete attachment)

We are not running these tests yet in automation on linux64-qr. We should do that.

Here's a try push with what it looks like now:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4faa2533aae34544691a185c78d102fb88749b39
Depends on: 1433579
Assignee

Comment 5

a year ago
mozreview-review
Comment on attachment 8950969 [details]
Bug 1425589: Webcompat tests on linux.  After this, we have failures with intermittent bugs filed against them and timeouts.

https://reviewboard.mozilla.org/r/220232/#review226106

::: commit-message-16547:1
(Diff revision 1)
> +Bug 1425589: Webcompat tests on linux.  After this, we have failures with intermittent bugs filed against them and timeouts. r?kats

s/Webcompat tests on linux./Annotation web-platform reftests for WebRender on Linux./

Also we should probably move this to a dependent bug if we're landing this patch right away, since it doesn't actually fix bug 1425589.
Attachment #8950969 - Flags: review?(bugmail) → review+
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcf5cd02189240b977b73906dd79dc697e814979 has a bunch of retriggers. Of the Wr (reftest) jobs the Wr1 chunk is failing at a 100% rate so that still needs additional annotating before we can land that. The Wd (wdspec) and Wpt (mochitest-style tests) jobs also have failures that will need to be addressed, but we can defer turning those on.
From what I can tell, one test fails in Wr1 optimized all the time (object-fit-contain-png-001c.html) and none of them fail all the time in debug.
Yeah, nondeterministic results are hard to deal with. We have only three possible options:
1) Find and fix the nondeterminism
2) Skip the tests (or mark them as random) - not sure if this is possible with web-platform-tests?
3) Don't enable the test suite
Apparently it is possible to disable web platform tests. Here's a try push with more stuff annotated/disabled, hopefully I got everything:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6f8acf2496fdc2dba2266730a8e1f1ccf96d542
Assignee: nobody → bugmail

Comment 14

a year ago
mozreview-review
Comment on attachment 8960758 [details]
Bug 1425589 - Turn on wpt on linux64-qr.

https://reviewboard.mozilla.org/r/229502/#review235386
Attachment #8960758 - Flags: review?(james) → review+

Comment 15

a year ago
mozreview-review
Comment on attachment 8960757 [details]
Bug 1425589 - Update web-platform-tests annotations for webrender on linux.

https://reviewboard.mozilla.org/r/229500/#review235388

So in general I'm not sure if these tests are disabled for instability (fine) or for failing (not the recommended approach).

Also, in order to update metadata correctly when we import new tests, a change is  required in testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py::update_properties to ensure that we consider webrender as a possible varaible when updating metadata (hopefully we can remove stylo there now since we are ending the  non-stylo configuration).

::: testing/web-platform/meta/css/css-backgrounds/background-size-027.html.ini:4
(Diff revision 1)
>  [background-size-027.html]
>    prefs: [dom.webcomponents.shadowdom.enabled:true]
> +  disabled:
> +    if webrender: bug 1425588

So is this disabled for instability? Generally we don't mark web-platform-tests as disabled unless they aren't stable. If the test happens to fail we mark it as expected: FAIL so we can continue to check that it doesn't crash or whatever and notice as soon as soon as it starts to pass.
Attachment #8960757 - Flags: review?(james) → review-
(In reply to James Graham [:jgraham] from comment #15)
> Also, in order to update metadata correctly when we import new tests, a
> change is  required in
> testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py::
> update_properties to ensure that we consider webrender as a possible
> varaible when updating metadata

Is there any documentation, or can you explain, what the function is supposed to return? A tuple containing a [] and a {} with overlapping properties isn't obvious to me. Should webrender go in the [] or the {} or both?

> 
> So is this disabled for instability? Generally we don't mark
> web-platform-tests as disabled unless they aren't stable. If the test
> happens to fail we mark it as expected: FAIL so we can continue to check
> that it doesn't crash or whatever and notice as soon as soon as it starts to
> pass.

Yes, all the ones I disabled were because of instability. Bug 1438284 which landed earlier took care of most of the stable results (unexpected-fail/unexpected-pass etc.)
Flags: needinfo?(james)
Oh wow yes, that's *surprisingly* non-obvious, isn't it.

The first list is all the properties that are considered, and the second is a set of properties that only take boolean values. s/"stylo"/"stylo", "webrender"/ throughout that function should do the right thing.
Flags: needinfo?(james)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 20

a year ago
mozreview-review
Comment on attachment 8960757 [details]
Bug 1425589 - Update web-platform-tests annotations for webrender on linux.

https://reviewboard.mozilla.org/r/229500/#review235440

Thanks!
Attachment #8960757 - Flags: review?(james) → review+
Thanks for the quick reviews! :)

Comment 22

a year ago
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6e0605add1f5
Update web-platform-tests annotations for webrender on linux. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/08bc853b01ae
Turn on wpt on linux64-qr. r=jgraham

Comment 23

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/6e0605add1f5
https://hg.mozilla.org/mozilla-central/rev/08bc853b01ae
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Depends on: 1447733
Depends on: 1448086
Created web-platform-tests PR https://github.com/w3c/web-platform-tests/pull/10307 for changes under testing/web-platform/tests
Whiteboard: [gfx-noted][triage] → [gfx-noted][triage][wptsync upstream]

Updated

9 months ago
See Also: → 1489700
You need to log in before you can comment on or make changes to this bug.