1584760 - Intermittent TEST-UNEXPECTED-PASS | border-image/svg-as-border-image-4b.html == border-image/svg-as-border-image-4-ref.html | image comparison, max difference: 16, number of differing pixels: 69

Status: RESOLVED FIXED

(Core :: Layout, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=268932713&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/YsJ_g-yTS_qK8_0d3QWz1A/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/YsJ_g-yTS_qK8_0d3QWz1A/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

Comment 1

[Intermittent Failures Robot]

1 failures in 4530 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1584760&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 2

[Lee Salzman [:lsalzman]]

Attachment: Bug 1584760 - adjust fuzz more for svg-as-border-image-4b.html. r?jrmuizel

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24a4aba3f466
adjust fuzz more for svg-as-border-image-4b.html. r=jrmuizel

Comment 4

[Geoff Brown [:gbrown]]

:lsalzman - Thanks for this. You probably didn't notice that some of the most recent unexpected passes have 0 differences:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=269172218&repo=autoland&lineNumber=4320

Expand the range further?

Comment 5

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/24a4aba3f466

Comment 6

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/24a4aba3f466

Comment 7

[Daniel Holbert [:dholbert]]

lsalzman, this change seems to have made some platforms perma-orange due to having no differences:
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=269376115&revision=348fa5ed7407d53fd9e3661cf9f55cb187622fbc&searchStr=Android%2C8.0%2CPixel2%2CQuantumRender%2Copt%2CReftests%2Ctest-android-hw-p2-8-0-arm7-api-16-qr%2Fopt-geckoview-reftest-e10s-3%2CR%28R3%29

Looks like this needs a followup.

Comment 8

[Daniel Holbert [:dholbert]]

Side note that the logs there do show a bit of difference if you view them in reftest-analyzer, even though we're reporting max difference: 0, number of differing pixels: 0. This is because we apparently "round down" fuzz in some cases via this autofuzz clause:
https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/layout/tools/reftest/reftest.jsm#1128-1137

As shown in one log:

INFO - REFTEST TEST-LOAD | http://10.7.205.221:8854/tests/layout/reftests/border-image/svg-as-border-image-4b.html | 306 / 1928 (15%)
INFO - REFTEST INFO | REFTEST wr-on-android dropping fuzz of (1, 36) to (0, 0)
INFO - REFTEST INFO | REFTEST fuzzy test (16, 69) <= (0, 0) <= (16, 150)
INFO - REFTEST TEST-UNEXPECTED-PASS | http://10.7.205.221:8854/tests/layout/reftests/border-image/svg-as-border-image-4b.html == http://10.7.205.221:8854/tests/layout/reftests/border-image/svg-as-border-image-4-ref.html | image comparison, max difference: 0, number of differing pixels: 0

Anyway, even the "real" difference here (1,36) is below the acceptable-fuzz-range that we added in this bug. And since autofuzz bumps it down 0,0, we need to allow 0,0 in our ranges.

Comment 9

[Lee Salzman [:lsalzman]]

Attachment: Bug 1584760 - expand fuzz range more for svg-as-border-image-4b.html. r?jrmuizel

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db0a3abfc428
expand fuzz range more for svg-as-border-image-4b.html. r=jrmuizel

Comment 11

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/db0a3abfc428

Comment 13

[Intermittent Failures Robot]

7 failures in 4292 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4
• autoland: 2
• mozilla-inbound: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16-qr: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1584760&startday=2019-09-30&endday=2019-10-06&tree=all