Closed Bug 1563059 Opened 4 months ago Closed 4 months ago

contain-paint-clip-015.html and -016.html need a fuzzy annotation instead of a failure annotation

Categories

(Core :: Layout, task, P3)

task

Tracking

()

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: dholbert, Assigned: dholbert)

References

Details

Attachments

(2 files)

The WPT tests contain-paint-clip-015.html and -016.html basically pass, but they've got some fuzzy pixels around the fringe of the green circle.

We have them annotated as failing now, but we should annotate them as fuzzy instead.

(I'm attaching a failure log from a local WPT run, with the failure info, which can be visualized using https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml )

Summary: contain-paint-clip-015.html and -16.html need a fuzzy annotation → contain-paint-clip-015.html and -16.html need a fuzzy annotation instead of a failure annotation
Assignee: nobody → dholbert
Status: NEW → ASSIGNED
Flags: needinfo?(dholbert)

I think this try run (which removes the failure annotations) will show these fuzzy-failures and let us know the range of fuzziness we need in order to cover all platforms:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ec7224ebb3e39351e49a900442f99804750238e

Depends on: 1563061
Flags: needinfo?(dholbert)

Looks like the results are pretty consistent. The two tests have the exact same fuzziness (on a given platform), and the observed fuzzy metrics were:

Linux64-qr, Win10-qr: 332 pixels different, maximum difference per channel 98
Linux64, Win7, Win10, Android: 372 pixels different, maximum difference per channel 99

The MacOS builder had a harness timeout, so I don't have results there yet. But at this point it looks like I can just have a fuzzy annotation with two possibilities, one for "if webrender" and the other for "if !webrender".

For some reason in my local environment with Android emulator, I get 373 pixels different, maximum difference per channel 131

Summary: contain-paint-clip-015.html and -16.html need a fuzzy annotation instead of a failure annotation → contain-paint-clip-015.html and -016.html need a fuzzy annotation instead of a failure annotation

The MacOS testrun from comment 1 / comment 2 has now finished, and it matches the non-webrender consensus of 372 pixels different, maximum difference per channel 99 for these tests.

I think that means the attached patch has correct annotations. Here's a Try run to be sure:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=315470258782003f3eb09771e52812f170baffd8

Pushed by dholbert@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c97ef13f37d2
Give contain-paint-clip-015.html & -016.html a fuzzy annotation instead of a failure annotation. (no review, test-annotation-only)

Comment 5's try run showed a bit of variability in the fuzziness, so I loosened the annotations a bit (and merged the non-webrender cases for simplicity):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec7a058bcb69869d9db06cee761ad0dac34d5cdc

That try run looks good, so I went ahead and landed the annotation-change (comment 6).

Backed out changeset c97ef13f37d2 (Bug 1563059) for BPGO (run) failure.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=254821272&resultStatus=testfailed%2Cbusted%2Cexception&revision=c97ef13f37d297ded34d28e1e4954fedf4851654

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=254821272&repo=mozilla-inbound&lineNumber=585

Backout link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=254821272&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=d5470ddece8d5a0d85cc55141e3ff473387a9c37

[taskcluster 2019-07-04T19:56:45.831Z] Task Duration: 20m0.2222225s
[taskcluster:error] Uploading error artifact public/build/profdata.tar.xz from file build/src/artifacts/profdata.tar.xz with message "Could not read file 'Z:\\task_1562266407\\build\\src\\artifacts\\profdata.tar.xz'", reason "file-missing-on-worker" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profdata.tar.xz'
[taskcluster 2019-07-04T19:56:46.823Z] Uploading artifact public/build/profile-run-1.log from file build/src/artifacts/profile-run-1.log with content encoding "gzip", mime type "text/plain" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] Uploading error artifact public/build/profile-run-2.log from file build/src/artifacts/profile-run-2.log with message "Could not read file 'Z:\\task_1562266407\\build\\src\\artifacts\\profile-run-2.log'", reason "file-missing-on-worker" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profile-run-2.log'
[taskcluster 2019-07-04T19:56:47.745Z] [mounts] Preserving cache: Moving "Z:\\task_1562266407\\build" to "Z:\\caches\\aTpqSH5mQP-81SIBDMQ_WA"
[taskcluster 2019-07-04T19:58:18.575Z] [mounts] Removing cache gecko-level-3-checkouts from cache table
[taskcluster 2019-07-04T19:58:18.575Z] [mounts] Deleting cache gecko-level-3-checkouts file(s) at Z:\caches\aTpqSH5mQP-81SIBDMQ_WA
[taskcluster 2019-07-04T19:58:39.660Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/UG5SHAHwRbyvOhq5EvjwAA/runs/2/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-07-03T18:02:31.574Z
[taskcluster:error] Task aborted - max run time exceeded
[taskcluster:error] file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profdata.tar.xz'
[taskcluster:error] file-missing-on-worker: Could not read file 'Z:\task_1562266407\build\src\artifacts\profile-run-2.log'
[taskcluster:error] goroutine 1 [running]:
[taskcluster:error] runtime/debug.Stack(0xc0428e94e8, 0x92f440, 0xc0429e2360)
[taskcluster:error] 	/home/travis/.gimme/versions/go1.10.8.src/src/runtime/debug/stack.go:24 +0xae
[taskcluster:error] main.(*TaskRun).Run.func2(0xc042004068, 0xc04207aa00, 0xc042004070)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:896 +0x231
[taskcluster:error] panic(0x92f440, 0xc0429e2360)
[taskcluster:error] 	/home/travis/.gimme/versions/go1.10.8.src/src/runtime/panic.go:502 +0x237
[taskcluster:error] main.(*WritableDirectoryCache).Unmount(0xc04230e050, 0xc04207aa00, 0x529044, 0xc0420be000)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:517 +0x4a4
[taskcluster:error] main.(*TaskMount).Stop(0xc04215c2a0, 0xc04221e780)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/mounts.go:378 +0xaa
[taskcluster:error] main.(*TaskRun).Run.func3(0xc042004068, 0xa8fac0, 0xc04215c2a0, 0xa90000, 0xd77bc8)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:978 +0xdc
[taskcluster:error] main.(*TaskRun).Run(0xc04207aa00, 0xc04222a690)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:1058 +0x1729
[taskcluster:error] main.RunWorker(0x0)
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:446 +0xc7c
[taskcluster:error] main.main()
[taskcluster:error] 	/home/travis/gopath/src/github.com/taskcluster/generic-worker/main.go:151 +0x7ba
[taskcluster:error] 
[taskcluster:error] &os.PathError{Op:"remove", Path:"Z:\\caches\\aTpqSH5mQP-81SIBDMQ_WA\\src\\obj-firefox\\dist\\firefox\\api-ms-win-crt-convert-l1-1-0.dll", Err:0x5}
[taskcluster:error] remove Z:\caches\aTpqSH5mQP-81SIBDMQ_WA\src\obj-firefox\dist\firefox\api-ms-win-crt-convert-l1-1-0.dll: Access is denied.
Flags: needinfo?(dholbert)

That looks like it was an intermittent infrastructure issue, unrelated to my commit (which just tweaked some test annotations, which are only read in WPT test-run jobs).

Trees are closed today, so I'll reland whenever they reopen.

Pushed by dholbert@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/064218165cb1
Give contain-paint-clip-015.html & -016.html a fuzzy annotation instead of a failure annotation. (no review, test-annotation-only)

(In reply to Daniel Holbert [:dholbert] from comment #9)

That looks like it was an intermittent infrastructure issue, unrelated to my commit (which just tweaked some test annotations, which are only read in WPT test-run jobs).

Note also that a retrigger (today) of one of those failing tasks ended up green/successful:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=pending%2Crunning%2Cusercancel%2Csuperseded%2Cretry%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=c97ef13f37d297ded34d28e1e4954fedf4851654&searchStr=windows%2Crun&selectedJob=254821277

Perhaps there was some network hiccup or some other temporary issue that caused the failures yesterday. shrug

Trees are closed today, so I'll reland whenever they reopen.

Relanded!

Flags: needinfo?(dholbert)
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
You need to log in before you can comment on or make changes to this bug.