Closed
Bug 1435844
Opened 6 years ago
Closed 6 years ago
reftests on windows10 and the new moonshot hardware have many failures
Categories
(Testing :: Reftest, enhancement)
Testing
Reftest
Tracking
(firefox60 fixed, firefox61 fixed)
RESOLVED
FIXED
mozilla61
People
(Reporter: jmaher, Assigned: jmaher)
References
(Depends on 15 open bugs)
Details
Attachments
(1 file, 1 obsolete file)
1.16 KB,
patch
|
rwood
:
review+
|
Details | Diff | Splinter Review |
we currently run windows10 reftests on hardware. The old hardware is getting replaced and while doing trial runs of the new hardware we find many reftests that are failing. Most recently I see 66 failures. https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 here is the try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=578fae181887b2927b984d283ff8bb69da041d02 the new hardware has an Intel graphics card vs the old hardware which had a nVidia graphics card. I suspect the drives and GPUs do different things which could explain some differences- although I am not sure. In order to move the job we need to get it green, which means we have 3 choices: 1) fixing the tests 2) annotating the manifests for fuzzying 3) some combination of the above 2 I don't know how to investigate the failures fully- I am happy to fix all the issues by manifest hacking- although only after trying to fix or fully explain the issue.
Assignee | ||
Comment 1•6 years ago
|
||
:jet/:milan- could you two help look at or find the right people to look at the above failures? If there are suggests to try on future try pushes (like prefs or environment variables or os changes) I am happy to work on that.
Flags: needinfo?(milan)
Flags: needinfo?(bugs)
Some may be fuzzing, but there are some that worry me more - the ones with alpha differences, or 128 vs. 127. Looking more.
Flags: needinfo?(milan)
Are we getting a different level of acceleration between the two? What does "about:support" look like for the two configurations (old and new)? Do we have a way of getting that?
Flags: needinfo?(milan)
Assignee | ||
Comment 4•6 years ago
|
||
here is the about:support for the new hardware: https://pastebin.mozilla.org/9077424
Assignee | ||
Comment 5•6 years ago
|
||
and the win10 on IX hardware about:support is here: https://bug1358306.bmoattachments.org/attachment.cgi?id=8870492
Markus, these seven tests are all off by 16 in r and g - we fuzz to a 64 difference for webrender, why is that OK, and is the 16 difference here OK? https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1 https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/V3pDRrN1QfWvTfRzu0XlUg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
Flags: needinfo?(milan) → needinfo?(mstange)
Assignee | ||
Comment 7•6 years ago
|
||
:milan- the 7 links are all the same url, maybe you could use the test names?
Aah. All the failing seven tests in layout/reftests/w3c-css/submitted/masking/ -
Flags: needinfo?(milan)
Comment 9•6 years ago
|
||
(In reply to Milan Sreckovic [:milan] from comment #6) > Markus, these seven tests are all off by 16 in r and g - we fuzz to a 64 > difference for webrender, why is that OK, and is the 16 difference here OK? I don't know, 64 really seems like a bit much. And a difference of exactly 16 sounds ominous. These circles are rasterized by Skia and composited by WebRender - we should at least find out whether the imprecision originates in Skia-land or during the WebRender compositing. There's not really a good reason for WebRender to be imprecise at mask compositing.
Flags: needinfo?(mstange)
Right - note that the difference of 16 is without WebRender, just from switching to a different GPU.
Assignee | ||
Comment 11•6 years ago
|
||
you can see the green: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5386bc84baef711e00d92953be0eaf8977c33e9e&selectedJob=161417472 I suspect some of these fuzzy-if statements should be actual fixes (in product, test, or OS)- I am not sure how to determine that though.
Assignee | ||
Comment 12•6 years ago
|
||
I haven't seen much action on this bug in the last week. My plan if there is no specific tests we are focusing on is to fuzzy-if all 66 failures as per comment 11. I will take action here next week on Friday February 23rd with all the fuzzy-if statements. If there are tests I should exclude from that, please let me know!
Assignee | ||
Comment 13•6 years ago
|
||
I have created individual bugs for each manifest file- I am verifying one more time that the changes I added to each bug still work and I get green- the plan is to land all those changes next friday (1 week from today)
Assignee | ||
Comment 14•6 years ago
|
||
all the failures have r+ fuzzy-if patches, we are ready to go with a small patch to made the official change.
Assignee: nobody → jmaher
Attachment #8949884 -
Attachment is obsolete: true
Status: NEW → ASSIGNED
Flags: needinfo?(milan)
Flags: needinfo?(bugs)
Attachment #8953389 -
Flags: review?(rwood)
Comment 16•6 years ago
|
||
Comment on attachment 8953389 [details] [diff] [review] migrate windows 10 reftests from buildbot to taskcluster and new hardware Review of attachment 8953389 [details] [diff] [review]: ----------------------------------------------------------------- LGTM
Attachment #8953389 -
Flags: review?(rwood) → review+
Comment 17•6 years ago
|
||
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/9d5118022be0 run windows 10 reftests on new hardware and taskcluster. r=rwood
Comment 18•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/9d5118022be0
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox60:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Comment 19•6 years ago
|
||
Backout by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/673b8d253e4d backout for win10 hardware failures. r=me https://hg.mozilla.org/integration/mozilla-inbound/rev/1bc5d8dfe84a backout changes to schedule motionmark and h1 on windows. r=me CLOSED TREE
Comment 20•6 years ago
|
||
Backout by archaeopteryx@coole-files.de: https://hg.mozilla.org/mozilla-central/rev/89023d3b5422 backout for win10 hardware failures. r=me a=backout CLOSED TREE https://hg.mozilla.org/mozilla-central/rev/c50f5f846c2e backout changes to schedule motionmark and h1 on windows. r=me a=backout CLOSED TREE
Updated•6 years ago
|
Status: RESOLVED → REOPENED
status-firefox60:
fixed → ---
Resolution: FIXED → ---
Target Milestone: mozilla60 → ---
Updated•6 years ago
|
Comment 21•6 years ago
|
||
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/08c54405586b reftests on windows10 and the new moonshot hardware. r=rwood
Comment 22•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/08c54405586b
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
status-firefox61:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Assignee | ||
Updated•6 years ago
|
Whiteboard: [checkin-needed-beta]
Comment 23•6 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/cfe7adda153d
status-firefox60:
--- → fixed
Whiteboard: [checkin-needed-beta]
You need to log in
before you can comment on or make changes to this bug.
Description
•