1768650 - Intermittent svg/smil/anim-marker-orient-02.svg == svg/smil/lime.svg | image comparison, max difference: 255, number of differing pixels: 380

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect)

Description

[Treeherder Bug Filer]

Filed by: ccozmuta [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=377536848&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z4Ku1TfYQ6qu0Av-7mmSIg/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z4Ku1TfYQ6qu0Av-7mmSIg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2022-05-10T11:57:08.822Z] 11:57:08     INFO - REFTEST TEST-START | layout/reftests/svg/smil/anim-marker-orient-02.svg == layout/reftests/svg/smil/lime.svg
[task 2022-05-10T11:57:08.824Z] 11:57:08     INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/svg/smil/anim-marker-orient-02.svg | 46 / 161 (28%)
[task 2022-05-10T11:57:09.300Z] 11:57:09     INFO - REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/svg/smil/anim-marker-orient-02.svg == layout/reftests/svg/smil/lime.svg | image comparison, max difference: 255, number of differing pixels: 380
<...>
[task 2022-05-10T11:57:09.312Z] 11:57:09     INFO - REFTEST INFO | Saved log: START file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/svg/smil/anim-marker-orient-02.svg
[task 2022-05-10T11:57:09.314Z] 11:57:09     INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
[task 2022-05-10T11:57:09.316Z] 11:57:09     INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2022-05-10T11:57:09.319Z] 11:57:09     INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
[task 2022-05-10T11:57:09.320Z] 11:57:09     INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
[task 2022-05-10T11:57:09.321Z] 11:57:09     INFO - REFTEST INFO | Saved log: RecordResult fired
[task 2022-05-10T11:57:09.322Z] 11:57:09     INFO - REFTEST INFO | Saved log: RecordResult fired
[task 2022-05-10T11:57:09.323Z] 11:57:09     INFO - REFTEST TEST-END | layout/reftests/svg/smil/anim-marker-orient-02.svg == layout/reftests/svg/smil/lime.svg
[task 2022-05-10T11:57:09.324Z] 11:57:09     INFO - REFTEST TEST-START | layout/reftests/svg/smil/anim-polygon-points-01.svg == layout/reftests/svg/smil/anim-polygon-points-01-ref.svg

'''INFO'''
There is a 'red square' that appears on the first image - the test one, and is missing from the second image.
'''INFO'''

Comment 1

[Noemi Erli[:noemi_erli]]

R+B range: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=Linux%2C18.04%2Cx64%2CWebRender%2Casan%2Copt%2CReftests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64-asan-qr%2Fopt-reftest-fis-e10s%2CR3&tochange=2d6aaf0a4a5a45914de14486061760fdecdaf777&fromchange=40fdd5b8adbaf51ac0903d60e60f6c002b76e1fe
Could this be related to Bug 1686654?

Comment 2

[Nicolas Silva [:nical]]

Could be, sort of. This is an animation test and only intermitten with an asan build, I suspect we have a timing/race issue that we were unlikely to hit before and bug 1686654 changed the timing in a way that is exarbated by asan overhead and somehow made it more likely to happen.

There shouldn't be non-determinism introduced in bug 1686654 at least. We should probably just mark it as random in that configuration since it's only intermittent with asan.

Comment 3

[Iulian Moraru]

Update:
There have been 35 failures within the last 3 days:
• 3 failures on Windows 10 x64 2004 WebRender debug
• 7 failures on Windows 10 x64 2004 asan WebRender opt
• 1 failures on Windows 10 x86 2004 WebRender debug
• 1 failures on Linux 18.04 x64 WebRender tsan opt
• 3 failures on Linux 18.04 x64 WebRender Shippable opt
• 2 failures on Linux 18.04 x64 WebRender opt
• 8 failures on Linux 18.04 x64 WebRender debug
• 7 failures on Linux 18.04 x64 WebRender asan opt
• 1 failures on Android 7.0 x86-64 WebRender opt
• 2 failures on Android 7.0 x86-64 Lite WebRender opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=378019030&repo=autoland&lineNumber=5843

Comment 4

[Iulian Moraru]

Hi Jonathan! As the owner of this component, could you help us assign this to someone?
Thank you!

Comment 5

[Intermittent Failures Robot]

48 failures in 3317 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 32
• mozilla-central: 11
• try: 5

Platform and build breakdown:

• linux1804-64-asan-qr: 10
  • opt: 10
• linux1804-64-qr: 12
  • debug: 10
  • opt: 2
• windows10-64-2004-asan-qr: 9
  • opt: 9
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• windows10-64-2004-qr: 4
  • debug: 3
  • opt: 1
• linux1804-64-shippable-qr: 3
  • opt: 3
• linux1804-64-tsan-qr: 2
  • opt: 2
• windows10-32-2004-qr: 1
  • debug: 1
• android-em-7-0-x86_64-lite-qr: 3
  • opt: 3
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 6

[Daniel Holbert [:dholbert]]

Just dropping a few notes here, since I happened to run across this on a Try run and took a quick look...

• Direct link to test: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/reftests/svg/smil/anim-marker-orient-02.svg

• The initial rendering of the test has a small red square, which is 20px to the right of the top-left corner. The test has logic to advance the animation timeline a bit later in the animation (after the red square is supposed to disappear) and takes a snapshot. (note that this logic doesn't run in the above-linked hg.m.o-served version of the test, because the CSP blocks the script load)

• When the test fails (at least in the failure log that I looked at), it looks like that red square is still there, which means we probably failed to invalidate/repaint properly.

• This test was at one point annotated as random-if(webrender) but that random annotation was removed in bug 1436084 (4 years ago) when it was observed that it was no-longer-random.

So: my tentative guess would be that this might be some version of that same random-if-webrender issue coming back.

Comment 7

[Daniel Holbert [:dholbert]]

Aha, now I see comment 1 - 2. It looks indeed like a good bet that this would be a regression from that change. Marking as such, and updating component.

(In reply to Nicolas Silva [:nical] from comment #2)

Could be, sort of. This is an animation test and only intermitten with an asan build

Unfortunately it's intermittent in more sorts of builds now; see comment 3 and comment 5. So I'm not sure we can just mark this as random on certain platforms, since it looks like it's getting hit on at least some "regular" builds ("plain" opt or debug) on every platform.

nical, what do you think we should do here? If this pref is live, maybe it'd be worth preffing-it-off, just for this test, as a stopgap and to validate the theory that this is indeed what's causing the issue? (Hopefully we wouldn't just leave things in that state indefinitely; maybe we could then try to capture in rr or something and see about fixing?)

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1686654

Comment 9

[Intermittent Failures Robot]

52 failures in 2694 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #21 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• try: 10
• mozilla-central: 12
• autoland: 30

Platform and build breakdown:

• windows10-64-2004-asan-qr: 9
  • opt: 9
• linux1804-64-asan-qr: 9
  • opt: 9
• linux1804-64-shippable-qr: 3
  • opt: 3
• windows10-32-2004-qr: 3
  • debug: 3
• linux1804-64-qr: 15
  • opt: 5
  • debug: 10
• android-em-7-0-x86_64-lite-qr: 2
  • opt: 2
• windows10-64-2004-mingwclang-qr: 1
  • debug: 1
• windows10-64-2004-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 3
  • opt: 3
• linux1804-64-tsan-qr: 2
  • opt: 2
• windows7-32-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 10

[Nicolas Silva [:nical]]

Attachment: Bug 1768650 - Turn off gfx.webrender.svg-images for a test. r=dholbert

Comment 11

[Nicolas Silva [:nical]]

nical, what do you think we should do here? If this pref is live, maybe it'd be worth preffing-it-off, just for this test, as a stopgap

Sounds good to me.

Comment 12

[Intermittent Failures Robot]

48 failures in 2624 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #14 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 31
• mozilla-central: 14
• try: 3

Platform and build breakdown:

• linux1804-64-qr: 14
  • debug: 10
  • opt: 4
• windows10-64-2004-mingwclang-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 2
  • opt: 2
• windows10-64-2004-asan-qr: 6
  • opt: 6
• windows10-32-2004-qr: 3
  • debug: 3
• windows7-32-qr: 2
  • opt: 2
• linux1804-64-asan-qr: 10
  • opt: 10
• windows10-64-2004-qr: 3
  • debug: 1
  • opt: 2
• android-em-7-0-x86_64-lite-qr: 2
  • opt: 2
• linux1804-64-tsan-qr: 3
  • opt: 3
• macosx1015-64-qr: 1
  • debug: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-05-23&endday=2022-05-29&tree=all

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1686654

Comment 14

[Andreea Pavel [:apavel]]

Nicolas can the patch here get landed or are there any changes planned?

Comment 15

[Intermittent Failures Robot]

57 failures in 2917 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 37
• mozilla-beta: 5
• mozilla-central: 8
• try: 7

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 2
  • opt: 2
• linux1804-64-asan-qr: 11
  • opt: 11
• windows10-64-2004-asan-qr: 12
  • opt: 12
• macosx1015-64-qr: 1
  • debug: 1
• linux1804-64-qr: 14
  • opt: 5
  • debug: 9
• windows10-32-2004-qr: 1
  • debug: 1
• windows7-32-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 1
  • opt: 1
• windows10-64-2004-qr: 7
  • debug: 5
  • opt: 2
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 1
  • opt: 1
• android-em-7-0-x86_64-lite-qr: 2
  • opt: 2
• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1
• windows10-64-2004-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 16

[Intermittent Failures Robot]

15 failures in 569 pushes (0.026 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 6
• autoland: 9

Platform and build breakdown:

• windows10-64-2004-qr: 2
  • debug: 2
• macosx1015-64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1
• windows7-32-qr: 2
  • opt: 2
• windows10-64-2004-asan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-lite-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • debug: 1
• linux1804-64-qr: 1
  • debug: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-06-06&endday=2022-06-06&tree=all

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cce9e3eb2cc1
Turn off gfx.webrender.svg-images for a test. r=dholbert

Comment 18

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/cce9e3eb2cc1

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:nical, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 20

[Daniel Holbert [:dholbert]]

RE the beta102 wontfix: I disagree -- we should just uplift the (trivial) patch to beta. Otherwise sheriffs will have to waste time starring instances of this intermittent failure on beta (and eventually release). (They are already doing so; comment 15 shows this cropping up on beta.)

nical, if you've got a beta tree around, would you mind uplifting? Otherwise feel free to ni me back and I can uplift.

(We don't need to bother with beta approval, since this is just a test manifest change and has zero user impact.)

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Agreed, we don't want to live with this on ESR102 for the next year+. I'll take care of it.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/ce140164b266

Comment 23

[Intermittent Failures Robot]

21 failures in 3336 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• autoland: 11
• mozilla-esr102: 1
• mozilla-beta: 2

Platform and build breakdown:

• windows10-64-2004-qr: 3
  • debug: 3
• macosx1015-64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1
• windows7-32-qr: 2
  • opt: 2
• windows10-64-2004-asan-qr: 2
  • opt: 2
• android-em-7-0-x86_64-lite-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 1
  • opt: 1
• windows10-32-2004-qr: 1
  • debug: 1
• linux1804-64-qr: 2
  • debug: 2
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-64-2004-mingwclang-qr: 1
  • debug: 1
• windows7-32-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 24

[Intermittent Failures Robot]

2 failures in 3856 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1
• windows10-64-2004-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1768650&startday=2023-02-13&endday=2023-02-19&tree=all