1897042 - Perma [Tier 2] snap test_custom_fonts | AssertionError('Mismatching screenshots for test_custom_fonts_ref.png with extremas: ((0, 12), (0, 24), (0, 65))')

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=458387087&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EP8IVPDxQgifiXeVCJYNtQ/runs/0/artifacts/public/logs/live_backing.log

---

TEST-START | test_custom_fonts
[task 2024-05-16T01:11:26.136Z] Non empty differences bbox: (60, 358, 657, 432)
[task 2024-05-16T01:11:26.180Z] Saving screenshot 'screenshot_fail.png' to '/home/ubuntu/tasks/task_171582096320768/builds/worker/artifacts/screenshot_fail.png'
[task 2024-05-16T01:11:26.238Z] Saving screenshot 'screenshot_fail_parent.png' to '/home/ubuntu/tasks/task_171582096320768/builds/worker/artifacts/screenshot_fail_parent.png'
[task 2024-05-16T01:11:26.354Z] TEST-UNEXPECTED-FAIL | test_custom_fonts | AssertionError('Mismatching screenshots for test_custom_fonts_ref.png with extremas: ((0, 12), (0, 24), (0, 65))')
[task 2024-05-16T01:11:26.355Z] TEST-INFO took 6800ms
[task 2024-05-16T01:11:26.404Z] Traceback (most recent call last):
[task 2024-05-16T01:11:26.405Z]   File "/home/ubuntu/tasks/task_171582096320768/fetches/basic_tests.py", line 78, in __init__
[task 2024-05-16T01:11:26.405Z]     rv = getattr(self, m)(self._expectations[m])
[task 2024-05-16T01:11:26.406Z]   File "/home/ubuntu/tasks/task_171582096320768/fetches/qa_tests.py", line 246, in test_custom_fonts
[task 2024-05-16T01:11:26.406Z]     self.assert_rendering(exp, svg_div)
[task 2024-05-16T01:11:26.406Z]   File "/home/ubuntu/tasks/task_171582096320768/fetches/basic_tests.py", line 227, in assert_rendering
[task 2024-05-16T01:11:26.406Z]     assert (
[task 2024-05-16T01:11:26.406Z] AssertionError: Mismatching screenshots for test_custom_fonts_ref.png with extremas: ((0, 12), (0, 24), (0, 65))
[task 2024-05-16T01:11:30.065Z] Exiting with False
[task 2024-05-16T01:11:30.066Z] SUITE-END | took 28s
[taskcluster 2024-05-16T01:11:30.203Z]    Exit Code: 1
[taskcluster 2024-05-16T01:11:30.203Z]    User Time: 52.516764s
[taskcluster 2024-05-16T01:11:30.203Z]  Kernel Time: 31.260626s
[taskcluster 2024-05-16T01:11:30.203Z]    Wall Time: 3m28.385111779s
[taskcluster 2024-05-16T01:11:30.203Z]       Result: FAILED

Comment 1

[:gerard-majax]

so there's definitively changes, but hard to perceive. I'm still running git bisection against the snap package via try so it's a bit slow, but close to converge:

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [1ea86264e4c550f9c08b521f36bb1b7d61059540] Backed out 2 changesets (bug 1895492, bug 1889299) for causing bug 1896944. a=backout
git bisect bad 1ea86264e4c550f9c08b521f36bb1b7d61059540
# status: waiting for good commit(s), bad commit known
# good: [c3962b9c98f1b743afce8fb292c4c9a008f3c167] Backed out changeset afa417babef9 (bug 1842478) for causing mochitests failures in test_transitions_per_property.html. CLOSED TREE
git bisect good c3962b9c98f1b743afce8fb292c4c9a008f3c167
# good: [bd66d4def4648d70e276ffe2120d2490f420ccce] Bug 1896417 - Do not use CLOSED TREE for android-l10n-sync commits r=releng-reviewers,jcristau
git bisect good bd66d4def4648d70e276ffe2120d2490f420ccce
# bad: [a92ee2a84c498fb9dd0d70b209762eb82ea753de] Bug 1890216 - Source files for Skia m125 update. r=aosmond
git bisect bad a92ee2a84c498fb9dd0d70b209762eb82ea753de
# good: [b6399abcd45368a0106d82a0aa32389ac63c2000] Bug 1706121 - part4 : modify test. r=media-playback-reviewers,padenot
git bisect good b6399abcd45368a0106d82a0aa32389ac63c2000
# good: [8436fcc0cf416e336a5bfd00e710ea953e652101] Merge mozilla-central to autoland on a CLOSED TREE
git bisect good 8436fcc0cf416e336a5bfd00e710ea953e652101
# good: [126cae5afd79a0f120ba61bbae9e26052053c852] Bug 1893054 - test(webgpu): use more chunks for WPT on Linux r=webgpu-reviewers,nical
git bisect good 126cae5afd79a0f120ba61bbae9e26052053c852
# good: [37aae8352294891589856c4bf6629e95b6cdae0d] Bug 1880656 - Add font weight and font type controls in Reader menu. r=reader-mode-reviewers,fluent-reviewers,desktop-theme-reviewers,flod,cmkm,Itiel
git bisect good 37aae8352294891589856c4bf6629e95b6cdae0d
# good: [b126f1a1a79b85c373d440a1dfb27b6b8eb11007] Bug 1896719 - move zoom-out.svg into toolkit/themes/shared/icons. r=desktop-theme-reviewers,reader-mode-reviewers,cmkm,Itiel
git bisect good b126f1a1a79b85c373d440a1dfb27b6b8eb11007

And for posterity, ./mach try fuzzy --push-to-lando --full --env TEST_FILTER=custom_fonts --rebuild 3 -q "'snap 'test 'try 'debug 'qa" is how you can trigger those

Comment 2

[:gerard-majax]

Skia update from bug 1890216 ? I think we can probably just update reference screenshots

Comment 3

[:gerard-majax]

Attachment: Capture d’écran du 2024-05-17 09-39-32.png

Comment 4

[:gerard-majax]

Attachment: Bug 1897042 - Allow per-channel Snap QA tests expectations r?benchatt!

Comment 5

[:gerard-majax]

Attachment: Bug 1897042 - Update references for Snap QA test on Nightly r?benchatt!

Comment 6

[:gerard-majax]

from https://treeherder.mozilla.org/jobs?repo=try&revision=88bec2db23e6e7a03f90a252e6d8df85be6fbf93 we need to update all references

Comment 7

[:gerard-majax]

Lee you might want to be aware of that. Left is new rendering i spotted on my selenium tests on Snap, right is the old one. Bisect pointed me to some skia update, and i like the new one better.

Comment 8

[Lee Salzman [:lsalzman]]

Just update the refs. This is the danger of using images as a test ref without any fuzz, unfortunately.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b3b730a7cf50
Allow per-channel Snap QA tests expectations r=benchatt
https://hg.mozilla.org/integration/autoland/rev/9740ed99a9e5
Update references for Snap QA test on Nightly r=benchatt

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/b3b730a7cf50
https://hg.mozilla.org/mozilla-central/rev/9740ed99a9e5

Comment 11

[Intermittent Failures Robot]

18 failures in 4547 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• linux64-snap-nightly: 17
  • debug: 10
  • opt: 7
• linux64-snap-beta: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1897042&startday=2024-05-13&endday=2024-05-19&tree=all