1796590 - Perma reftest TEST-UNEXPECTED-PASS | css-grid/grid-item-video-stretch-001.html == css-grid/grid-item-video-stretch-001-ref.html | image comparison, max difference: 0, number of differing pixels: 0

Status: RESOLVED FIXED

(Core :: Layout, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=393808246&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EJO9e3dVQna8jQzDZRMwsw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EJO9e3dVQna8jQzDZRMwsw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

There are no obvious differences.

Comment 1

[Cristian Tuns]

And also this: https://treeherder.mozilla.org/logviewer?job_id=393808247&repo=mozilla-central might be caused by the same Bug.
Failure line: REFTEST TEST-UNEXPECTED-PASS | dom/media/test/reftest/color_quads/720p.png.bt709.bt709.pc.yuv420p.vp9.webm == dom/media/test/reftest/color_quads/720p.png.bt709.bt709.pc.yuv420p.av1.webm | image comparison, max difference: 16, number of differing pixels: 1874

Backfill link: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&tochange=67ab7c3f34435806d010d5e06bc20088af73c05b&fromchange=bdf51a01a2e1b343cafff08d4847e869ff76b9c9&searchStr=os%2Cx%2C11%2Cwebrender%2Cshippable%2Copt%2Creftests%2Ctest-macosx1100-64-shippable-qr%2Fopt-reftest%2Cr&selectedTaskRun=e4tHxBFDTAe7OFqih337yA.0

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1792257

:az, since you are the author of the regressor, bug 1792257, could you take a look?

For more information, please visit auto_nag documentation.

Comment 3

[Intermittent Failures Robot]

24 failures in 600 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15
• try: 9

Platform and build breakdown:

• macosx1100-64-shippable-qr: 24
  • opt: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796590&startday=2022-10-20&endday=2022-10-20&tree=all

Comment 4

[Intermittent Failures Robot]

39 failures in 2793 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 24
• try: 13
• autoland: 2

Platform and build breakdown:

• macosx1100-64-shippable-qr: 39
  • opt: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796590&startday=2022-10-17&endday=2022-10-23&tree=all

Comment 5

[az [:az]]

Thanks -- I'm looking into it but haven't been able to repro locally yet. Should have more to report this week.

Comment 6

[Daniel Holbert [:dholbert]]

Sounds like maybe we just need to remove or update the fuzzy annotation, to reflect the fact that this test is always passing (or sometimes-passing, if it's not 100%) on this platform? [EDIT: Looks like this is indeed always-passing, per the bug title and the perma-orange on recent history at https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=macosx11%2Cr6 .]

The test is currently annotated as fuzzy-if(appleSilicon,255-255,167058-167058) which essentially means "Expected to catastrophically fail" (expecting 167,058 pixels to always differ between testcase & reference case). That annotation was added as part of bug 1706716, which was a broad "annotate all the existing failures on this platform, to reflect reality" sort of patch.

Probably those mismatching pixels were just all of the video areas failing to render at all, and probably that was fixed (partly or entirely) by bug 1792257?

Comment 7

[Daniel Holbert [:dholbert]]

Attachment: Bug 1796590: Remove fuzzy-if(appleSilicon) annotation for css-grid/grid-item-video-stretch-001.html. (no review, just updating reftest manifest to reflect reality)

Looks like this test used to catastrophically fail on this platform, per this
annotation; and that failure seems to have been fixed by bug 1792257.

Comment 8

[Daniel Holbert [:dholbert]]

Here's a try run with all macos11 reftest tasks, with the fuzzy annotation removed, just to be sure this test really does pass on all variants of this task on this platform:
https://treeherder.mozilla.org/jobs?repo=try&revision=98518b95349740a84f1270e6a351aaf5c910e4f2

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/780d2d1d736f
Remove fuzzy-if(appleSilicon) annotation for css-grid/grid-item-video-stretch-001.html. (no review, just updating reftest manifest to reflect reality)

Comment 10

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/780d2d1d736f

Comment 11

[Intermittent Failures Robot]

9 failures in 3743 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• try: 4

Platform and build breakdown:

• macosx1100-64-shippable-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796590&startday=2022-10-24&endday=2022-10-30&tree=all