Closed Bug 1489327 Opened 6 years ago Closed 6 years ago

Intermittent css-animations/continuation-opacity.html == css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Categories

(Core :: Graphics: WebRender, defect)

Product:
Core
Component:
Graphics: WebRender
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla64
Tracking Flags:
Tracking Status
firefox64 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: hiro)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disable-recommended])

Attachments

(1 file)

Bug 1489327 - Start animation once after a MozReftestInvalidate event is received in continuation-opacity.html. r=kats
46 bytes, text/x-phabricator-request
Details | Review 
Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=197925697&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1&only_show_unexpected=1

21:53:41     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888
See also bug 1415046 that happened only on Windows 7 though.
See Also: → 1415046
Component: Layout → CSS Transitions and Animations
There is a total of 25 failures in the last 7 days, all on windows10-64-qr.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=200889081&repo=mozilla-central&lineNumber=34595

23:21:34     INFO - REFTEST TEST-START | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html
23:21:34     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html | 60 / 66 (90%)
23:23:14     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | 60 / 66 (90%)
23:23:15     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Hiro, are you working on this? are there any updates?
Flags: needinfo?(hikezoe)
No, I haven't.

CCing Jeff, this may be a signal of a regression in WebRender.

Note that the failure started from this commit https://hg.mozilla.org/integration/mozilla-inbound/rev/d359982b8ded87c0d69f6032fd38a453b02a001e .

I tried to retrigger R1 jobs on earlier commits, but it seems we are not able to retrigger jobs on inbound any more?
OK, now I understand what's going on there.

(In reply to Andreea Pavel [:apavel] from comment #4)
> css-animations/continuation-opacity-ref.html
> 23:21:34     INFO - REFTEST TEST-LOAD |
> file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/
> css-animations/continuation-opacity.html | 60 / 66 (90%)
> 23:23:14     INFO - REFTEST TEST-LOAD |

Here, the test took 1 min 40 sec, that is 100 sec, which is the animation duration in the test.  So, we didn't get MozReftestInvalidate event.  This is related to bug 1445570 (I think bug 1445570 caused this failure). 

I am going to take a look on next Tuesday.
URL: 1445570
Flags: needinfo?(hikezoe)
It seems the failure got infrequent now.
(In reply to Hiroyuki Ikezoe (:hiro) (PTO on Sep 26) from comment #8)
> It seems the failure got infrequent now.

The average is still ~25 failures in 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-09-18&endday=2018-09-25&tree=all&bug=1489327

Thanks for looking into it.
Something has been changed on Sep. 27 or 28.

Moving to WebRender to be seen this in their radar, gfx team should look into this.  If no one has a chance to look into, please send NI to me.
Component: CSS Transitions and Animations → Graphics: WebRender
Most likely a regression from bug 1494206 which landed immediately before the first occurrence of the high-frequency failures. I'm doing some retriggers to verify.
And from the changes in that WR update, servo/webrender#3092 seems most likely. /cc nical
Wait, that's not right. There are a couple of these failure before that landed.
No longer blocks: 1494206
I think the couple of failures on the previous push are a fluke (this test was failing intermittently before too, just at a really low volume). I don't see any other failures on retriggers prior to that one, and that push is a ES lint warning fix so it's not likely to be an actual culprit. Putting back the finger on bug 1494206.
Blocks: 1494206
hiro: Hi, according to Comment 13, can you please have a look into this? Thank you!
Flags: needinfo?(hikezoe)
Blocks: 1459233
Though I think kats already has started looking into this, I am going to look into this in detail from animation point of view.  Assigning to myself, FWIW.
Assignee: nobody → hikezoe
Status: NEW → ASSIGNED
Flags: needinfo?(hikezoe)
I'm not actually investigating the failure; I just tried to figure out what caused it to start happening.
I can't reproduce this failure locally yet (on 10 thousands runs), so I did push a try run with outputting timestamps.  Here is a part of the log;

00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: waiting for MozAfterPaint
00:20:21 GMT+0000 (Greenwich Mean Time): AfterPaintListener in file:///C:/Users/task_1538611839/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html
00:20:22 GMT+0000 (Greenwich Mean Time): Webrender enabled, sending update whole canvas for invalidation

This shows us that the reftest got stuck at the initial state that we wait for the state where there is no more pending paints.  The fact implies we keep painting even if the opacity animation in the reftest is running on the compositor (or the opacity animation failed to run on the compositor in the first place).

Anyway I am pretty sure this failure caused by bug 1445570 and hopefully will be fixed by bug 1490117. 

That's said, I will figure out a way to work around bug 1490117 here.
URL: 1445570
See Also: → 1490117
A workaround seems to work on a try;

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3736fca12276e64d5a3e14eab035678c65c0501b&selectedJob=203270433

There are a bunch of oranges, but all of them are bug 1490117.
The animation in this reftests runs on the compositor.  In the mean time,
reftest harness waits for the state where there is no pending paint in the
initial phase (STATE_WAITING_TO_FIRE_INVALIDATE_EVENT, i.e. before sending
a MozReftestInvalidate event).  So if the animation starts running on the
compositor before a MozReftestInvalidate event is received, it means that
the reftest harness has to wait for the 'no pending paint' state until the
animation finishes because the reftest harness keeps flushing styles in the
initial phase which means the animation causes a paint on every flush.

To avoid above situation, we start the animation in question after we get a
MozReftestInvalidate event.
Patch in Bug 1496324 also seems to address the problem.
Pushed by hikezoe@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a2ecbcdcec39
Start animation once after a MozReftestInvalidate event is received in continuation-opacity.html. r=kats
https://hg.mozilla.org/mozilla-central/rev/a2ecbcdcec39
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox64: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: