1352128 - Web platform test failures for Windows 10 on TaskCluster

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Armen [:armenzg]]

We have a bunch of web platform test failures for Windows 10 and Windows 7 on TaskCluster. We're trying to green them up:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=253b7b9da0f75142f4650d4fbce6d6ca45597e3e

From jmaher:
Most likely we need to update the expectation files for wpt; typically these are as specific as possible to have os, version, flavor, etc.
For example, this is windows 8 specifically: https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/2dcontext/path-objects/2d.path.arc.selfintersect.1.html.ini#5
You can see windows 10 versions: (version == "10.0.10240") but we updated that to 10.0.14393 recently- a search replace should work, maybe 10.0.* would be more appropriate assuming 10.0.* works

Comment 1

[Ryan VanderMeulen [:RyanVM]]

You can also go the more generic route and drop the OS version check and skip on all win64. I've found more often than not that it's what we often want to do. I find wpt's default annotations to be overly-restrictive.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

yes, that is what I will do- bug I expect some tests to pass/fail differently when we get there, so this will be a single chance to wpt vs. a few changes.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update wpt manifests for win10

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9745b4fb9ac4
Update web-platform-tests expected data. r=jmaher, a=testonly

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: enable wpt tests for m-c and try

as per:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=91055aec46ba13b89cc7011dbd3f8729d7a4209f&exclusion_profile=false

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I backed this out for breaking webdriver tests https://hg.mozilla.org/integration/mozilla-inbound/rev/1032d46e20fdf193727be2d3cbd91a71944be35e

https://treeherder.mozilla.org/logviewer.html#?job_id=104642835&repo=mozilla-inbound

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: update wpt manifests for win10

removed the one change to conftest.py.ini and verified on try:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=5492bf1b91c9&newProject=try&newRevision=fd685976b2ac64e987c73609842525e5d4a841ae&framework=1&showOnlyImportant=0

I will land both patches when I am not in |afk mode.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0e098111552f
enabled web-platform-tests for win10 on m-c and try. r=grenade
https://hg.mozilla.org/integration/mozilla-inbound/rev/98c867096a06
Update web-platform-tests expected data. r=jmaher, a=testonly

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0e098111552f
https://hg.mozilla.org/mozilla-central/rev/98c867096a06

Comment 12

[Intermittent Failures Robot]

98 failures in 182 pushes (0.538 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-central: 98

Platform breakdown:
* windows10-64-vm: 98

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352128&startday=2017-06-07&endday=2017-06-07&tree=all

Comment 13

[Intermittent Failures Robot]

123 failures in 172 pushes (0.715 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-central: 123

Platform breakdown:
* windows10-64-vm: 123

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352128&startday=2017-06-08&endday=2017-06-08&tree=all

Comment 14

[Carsten Book [:Tomcat]]

joel since this is perma failing shoudn't we just hide them on central till this is fixed ?

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

oh we should, I haven't been able to get a fix with the toolchain

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

opt is working, but not debug.

Comment 17

[Intermittent Failures Robot]

48 failures in 153 pushes (0.314 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-central: 48

Platform breakdown:
* windows10-64-vm: 48

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352128&startday=2017-06-09&endday=2017-06-09&tree=all

Comment 18

[Intermittent Failures Robot]

270 failures in 864 pushes (0.313 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 270

Platform breakdown:
* windows10-64-vm: 270

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1352128&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: windows 10 manifest changes for web-platform-tests

all changes outside of win10:
$ hg qdiff | grep ^\+ | grep -v \+\+\+ | grep -v 10.0.15063
+
+
+
+
+
+  [contexts.py::test_window_resize]
+    expected:
+      if not debug and not stylo and e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): FAIL
+      if not debug and not stylo and not e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): FAIL
+

Comment 20

[James Graham [:jgraham]]

Comment on attachment 8879669 [details] [diff] [review]
windows 10 manifest changes for web-platform-tests

Review of attachment 8879669 [details] [diff] [review]:
-----------------------------------------------------------------

The one non-Windows 10 change is a known intermittent that ato is working on, please revert.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

got it, done

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89775519af6b
Web platform test failures for Windows 10 debug on TaskCluster. r=jgraham

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

one more failure:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6345c721b3f237bf3b0572c8463c0c776a6bfb87&exclusion_profile=false&filter-searchStr=wpt4%20debug

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/89775519af6b