1435844 - reftests on windows10 and the new moonshot hardware have many failures

Status: RESOLVED FIXED

(Testing :: Reftest, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we currently run windows10 reftests on hardware.  The old hardware is getting replaced and while doing trial runs of the new hardware we find many reftests that are failing.  Most recently I see 66 failures.
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

here is the try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=578fae181887b2927b984d283ff8bb69da041d02

the new hardware has an Intel graphics card vs the old hardware which had a nVidia graphics card.  I suspect the drives and GPUs do different things which could explain some differences- although I am not sure.

In order to move the job we need to get it green, which means we have 3 choices:
1) fixing the tests
2) annotating the manifests for fuzzying
3) some combination of the above 2 

I don't know how to investigate the failures fully- I am happy to fix all the issues by manifest hacking- although only after trying to fix or fully explain the issue.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:jet/:milan- could you two help look at or find the right people to look at the above failures?  If there are suggests to try on future try pushes (like prefs or environment variables or os changes) I am happy to work on that.

Comment 2

[Milan Sreckovic [:milan] (needinfo for best results)]

Some may be fuzzing, but there are some that worry me more - the ones with alpha differences, or 128 vs. 127.  Looking more.

Comment 3

[Milan Sreckovic [:milan] (needinfo for best results)]

Are we getting a different level of acceleration between the two?  What does "about:support" look like for the two configurations (old and new)?  Do we have a way of getting that?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

here is the about:support for the new hardware:
https://pastebin.mozilla.org/9077424

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

and the win10 on IX hardware about:support is here:
https://bug1358306.bmoattachments.org/attachment.cgi?id=8870492

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

Markus, these seven tests are all off by 16 in r and g - we fuzz to a 64 difference for webrender, why is that OK, and is the 16 difference here OK?

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

:milan- the 7 links are all the same url, maybe you could use the test names?

Comment 8

[Milan Sreckovic [:milan] (needinfo for best results)]

Aah.  All the failing seven tests in layout/reftests/w3c-css/submitted/masking/ -

Comment 9

[Markus Stange [:mstange]]

(In reply to Milan Sreckovic [:milan] from comment #6)
> Markus, these seven tests are all off by 16 in r and g - we fuzz to a 64
> difference for webrender, why is that OK, and is the 16 difference here OK?

I don't know, 64 really seems like a bit much. And a difference of exactly 16 sounds ominous. These circles are rasterized by Skia and composited by WebRender - we should at least find out whether the imprecision originates in Skia-land or during the WebRender compositing. There's not really a good reason for WebRender to be imprecise at mask compositing.

Comment 10

[Milan Sreckovic [:milan] (needinfo for best results)]

Right - note that the difference of 16 is without WebRender, just from switching to a different GPU.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add fuzzy-if to get reftests green on 66 tests

you can see the green:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5386bc84baef711e00d92953be0eaf8977c33e9e&selectedJob=161417472

I suspect some of these fuzzy-if statements should be actual fixes (in product, test, or OS)- I am not sure how to determine that though.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I haven't seen much action on this bug in the last week.  My plan if there is no specific tests we are focusing on is to fuzzy-if all 66 failures as per comment 11.  I will take action here next week on Friday February 23rd with all the fuzzy-if statements.  If there are tests I should exclude from that, please let me know!

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I have created individual bugs for each manifest file- I am verifying one more time that the changes I added to each bug still work and I get green- the plan is to land all those changes next friday (1 week from today)

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: migrate windows 10 reftests from buildbot to taskcluster and new hardware

all the failures have r+ fuzzy-if patches, we are ready to go with a small patch to made the official change.

Comment 16

[Robert Wood [:rwood]]

Comment on attachment 8953389 [details] [diff] [review]
migrate windows 10 reftests from buildbot to taskcluster and new hardware

Review of attachment 8953389 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d5118022be0
run windows 10 reftests on new hardware and taskcluster. r=rwood

Comment 18

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/9d5118022be0

Comment 19

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/673b8d253e4d
backout for win10 hardware failures. r=me
https://hg.mozilla.org/integration/mozilla-inbound/rev/1bc5d8dfe84a
backout changes to schedule motionmark and h1 on windows. r=me CLOSED TREE

Comment 20

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/89023d3b5422
backout for win10 hardware failures. r=me a=backout CLOSED TREE
https://hg.mozilla.org/mozilla-central/rev/c50f5f846c2e
backout changes to schedule motionmark and h1 on windows. r=me a=backout CLOSED TREE

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/08c54405586b
reftests on windows10 and the new moonshot hardware. r=rwood

Comment 22

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/08c54405586b

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cfe7adda153d