contain-paint-clip-015.html and -016.html need a fuzzy annotation instead of a failure annotation
Categories
(Core :: Layout, task, P3)
Tracking
()
Tracking | Status | |
---|---|---|
firefox70 | --- | fixed |
People
(Reporter: dholbert, Assigned: dholbert)
References
Details
Attachments
(2 files)
The WPT tests contain-paint-clip-015.html and -016.html basically pass, but they've got some fuzzy pixels around the fringe of the green circle.
We have them annotated as failing now, but we should annotate them as fuzzy instead.
(I'm attaching a failure log from a local WPT run, with the failure info, which can be visualized using https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml )
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 1•5 years ago
•
|
||
I think this try run (which removes the failure annotations) will show these fuzzy-failures and let us know the range of fuzziness we need in order to cover all platforms:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ec7224ebb3e39351e49a900442f99804750238e
Assignee | ||
Comment 2•5 years ago
|
||
Looks like the results are pretty consistent. The two tests have the exact same fuzziness (on a given platform), and the observed fuzzy metrics were:
Linux64-qr, Win10-qr: 332 pixels different, maximum difference per channel 98
Linux64, Win7, Win10, Android: 372 pixels different, maximum difference per channel 99
The MacOS builder had a harness timeout, so I don't have results there yet. But at this point it looks like I can just have a fuzzy annotation with two possibilities, one for "if webrender" and the other for "if !webrender".
Assignee | ||
Comment 3•5 years ago
|
||
For some reason in my local environment with Android emulator, I get 373 pixels different, maximum difference per channel 131
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 4•5 years ago
|
||
Assignee | ||
Comment 5•5 years ago
|
||
The MacOS testrun from comment 1 / comment 2 has now finished, and it matches the non-webrender consensus of 372 pixels different, maximum difference per channel 99
for these tests.
I think that means the attached patch has correct annotations. Here's a Try run to be sure:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=315470258782003f3eb09771e52812f170baffd8
Pushed by dholbert@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/c97ef13f37d2 Give contain-paint-clip-015.html & -016.html a fuzzy annotation instead of a failure annotation. (no review, test-annotation-only)
Assignee | ||
Comment 7•5 years ago
|
||
Comment 5's try run showed a bit of variability in the fuzziness, so I loosened the annotations a bit (and merged the non-webrender cases for simplicity):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec7a058bcb69869d9db06cee761ad0dac34d5cdc
That try run looks good, so I went ahead and landed the annotation-change (comment 6).
Comment 8•5 years ago
|
||
Backed out changeset c97ef13f37d2 (Bug 1563059) for BPGO (run) failure.
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=254821272&repo=mozilla-inbound&lineNumber=585
[taskcluster 2019-07-04T19:56:45.831Z] Task Duration: 20m0.2222225s
[taskcluster:error] Uploading error artifact public/build/profdata.tar.xz from file build/src/artifacts/profdata.tar.xz with message "Could not read file 'Z:\\task_1562266407\\build\\src\\artifacts\\profdata.tar.xz'", reason "file-missing-on-worker" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profdata.tar.xz'
[taskcluster 2019-07-04T19:56:46.823Z] Uploading artifact public/build/profile-run-1.log from file build/src/artifacts/profile-run-1.log with content encoding "gzip", mime type "text/plain" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] Uploading error artifact public/build/profile-run-2.log from file build/src/artifacts/profile-run-2.log with message "Could not read file 'Z:\\task_1562266407\\build\\src\\artifacts\\profile-run-2.log'", reason "file-missing-on-worker" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profile-run-2.log'
[taskcluster 2019-07-04T19:56:47.745Z] [mounts] Preserving cache: Moving "Z:\\task_1562266407\\build" to "Z:\\caches\\aTpqSH5mQP-81SIBDMQ_WA"
[taskcluster 2019-07-04T19:58:18.575Z] [mounts] Removing cache gecko-level-3-checkouts from cache table
[taskcluster 2019-07-04T19:58:18.575Z] [mounts] Deleting cache gecko-level-3-checkouts file(s) at Z:\caches\aTpqSH5mQP-81SIBDMQ_WA
[taskcluster 2019-07-04T19:58:39.660Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/UG5SHAHwRbyvOhq5EvjwAA/runs/2/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] Task aborted - max run time exceeded
[taskcluster:error] file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profdata.tar.xz'
[taskcluster:error] file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profile-run-2.log'
[taskcluster:error] goroutine 1 [running]:
[taskcluster:error] runtime/debug.Stack(0xc0428e94e8, 0x92f440, 0xc0429e2360)
[taskcluster:error] /home/travis/.gimme/versions/go1.10.8.src/src/runtime/debug/stack.go:24 +0xae
[taskcluster:error] main.(*TaskRun).Run.func2(0xc042004068, 0xc04207aa00, 0xc042004070)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:896 +0x231
[taskcluster:error] panic(0x92f440, 0xc0429e2360)
[taskcluster:error] /home/travis/.gimme/versions/go1.10.8.src/src/runtime/panic.go:502 +0x237
[taskcluster:error] main.(*WritableDirectoryCache).Unmount(0xc04230e050, 0xc04207aa00, 0x529044, 0xc0420be000)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:517 +0x4a4
[taskcluster:error] main.(*TaskMount).Stop(0xc04215c2a0, 0xc04221e780)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:378 +0xaa
[taskcluster:error] main.(*TaskRun).Run.func3(0xc042004068, 0xa8fac0, 0xc04215c2a0, 0xa90000, 0xd77bc8)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:978 +0xdc
[taskcluster:error] main.(*TaskRun).Run(0xc04207aa00, 0xc04222a690)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1058 +0x1729
[taskcluster:error] main.RunWorker(0x0)
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:446 +0xc7c
[taskcluster:error] main.main()
[taskcluster:error] /home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:151 +0x7ba
[taskcluster:error]
[taskcluster:error] &os.PathError{Op:"remove", Path:"Z:\\caches\\aTpqSH5mQP-81SIBDMQ_WA\\src\\obj-firefox\\dist\\firefox\\api-ms-win-crt-convert-l1-1-0.dll", Err:0x5}
[taskcluster:error] remove Z:\caches\aTpqSH5mQP-81SIBDMQ_WA\src\obj-firefox\dist\firefox\api-ms-win-crt-convert-l1-1-0.dll: Access is denied.
Updated•5 years ago
|
Assignee | ||
Comment 9•5 years ago
|
||
That looks like it was an intermittent infrastructure issue, unrelated to my commit (which just tweaked some test annotations, which are only read in WPT test-run jobs).
Trees are closed today, so I'll reland whenever they reopen.
Comment 10•5 years ago
|
||
Pushed by dholbert@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/064218165cb1 Give contain-paint-clip-015.html & -016.html a fuzzy annotation instead of a failure annotation. (no review, test-annotation-only)
Assignee | ||
Comment 11•5 years ago
|
||
(In reply to Daniel Holbert [:dholbert] from comment #9)
That looks like it was an intermittent infrastructure issue, unrelated to my commit (which just tweaked some test annotations, which are only read in WPT test-run jobs).
Note also that a retrigger (today) of one of those failing tasks ended up green/successful:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=pending%2Crunning%2Cusercancel%2Csuperseded%2Cretry%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=c97ef13f37d297ded34d28e1e4954fedf4851654&searchStr=windows%2Crun&selectedJob=254821277
Perhaps there was some network hiccup or some other temporary issue that caused the failures yesterday. shrug
Trees are closed today, so I'll reland whenever they reopen.
Relanded!
Comment 12•5 years ago
|
||
bugherder |
Description
•