Intermittent css-animations/continuation-opacity.html == css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

RESOLVED FIXED in Firefox 64

Status

()

defect
RESOLVED FIXED
10 months ago
8 months ago

People

(Reporter: intermittent-bug-filer, Assigned: hiro)

Tracking

({intermittent-failure})

unspecified
mozilla64
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 fixed)

Details

(Whiteboard: [stockwell disable-recommended])

Attachments

(1 attachment)

Reporter

Description

10 months ago
treeherder
Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=197925697&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1&only_show_unexpected=1

21:53:41     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888
Assignee

Comment 1

10 months ago
See also bug 1415046 that happened only on Windows 7 though.
See Also: → 1415046
Comment hidden (Intermittent Failures Robot)
Component: Layout → CSS Transitions and Animations
Comment hidden (Intermittent Failures Robot)
There is a total of 25 failures in the last 7 days, all on windows10-64-qr.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=200889081&repo=mozilla-central&lineNumber=34595

23:21:34     INFO - REFTEST TEST-START | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html
23:21:34     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html | 60 / 66 (90%)
23:23:14     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | 60 / 66 (90%)
23:23:15     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Hiro, are you working on this? are there any updates?
Flags: needinfo?(hikezoe)
Assignee

Comment 5

9 months ago
No, I haven't.

CCing Jeff, this may be a signal of a regression in WebRender.

Note that the failure started from this commit https://hg.mozilla.org/integration/mozilla-inbound/rev/d359982b8ded87c0d69f6032fd38a453b02a001e .

I tried to retrigger R1 jobs on earlier commits, but it seems we are not able to retrigger jobs on inbound any more?
Assignee

Comment 6

9 months ago
OK, now I understand what's going on there.

(In reply to Andreea Pavel [:apavel] from comment #4)
> css-animations/continuation-opacity-ref.html
> 23:21:34     INFO - REFTEST TEST-LOAD |
> file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/
> css-animations/continuation-opacity.html | 60 / 66 (90%)
> 23:23:14     INFO - REFTEST TEST-LOAD |

Here, the test took 1 min 40 sec, that is 100 sec, which is the animation duration in the test.  So, we didn't get MozReftestInvalidate event.  This is related to bug 1445570 (I think bug 1445570 caused this failure). 

I am going to take a look on next Tuesday.
URL: 1445570
Flags: needinfo?(hikezoe)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 8

9 months ago
It seems the failure got infrequent now.
(In reply to Hiroyuki Ikezoe (:hiro) (PTO on Sep 26) from comment #8)
> It seems the failure got infrequent now.

The average is still ~25 failures in 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-09-18&endday=2018-09-25&tree=all&bug=1489327

Thanks for looking into it.
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 13

9 months ago
Something has been changed on Sep. 27 or 28.

Moving to WebRender to be seen this in their radar, gfx team should look into this.  If no one has a chance to look into, please send NI to me.
Component: CSS Transitions and Animations → Graphics: WebRender
Most likely a regression from bug 1494206 which landed immediately before the first occurrence of the high-frequency failures. I'm doing some retriggers to verify.
And from the changes in that WR update, servo/webrender#3092 seems most likely. /cc nical
Wait, that's not right. There are a couple of these failure before that landed.
No longer blocks: 1494206
I think the couple of failures on the previous push are a fluke (this test was failing intermittently before too, just at a really low volume). I don't see any other failures on retriggers prior to that one, and that push is a ES lint warning fix so it's not likely to be an actual culprit. Putting back the finger on bug 1494206.
Blocks: 1494206
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
hiro: Hi, according to Comment 13, can you please have a look into this? Thank you!
Flags: needinfo?(hikezoe)
Blocks: 1459233
Assignee

Comment 22

9 months ago
Though I think kats already has started looking into this, I am going to look into this in detail from animation point of view.  Assigning to myself, FWIW.
Assignee: nobody → hikezoe
Status: NEW → ASSIGNED
Flags: needinfo?(hikezoe)
I'm not actually investigating the failure; I just tried to figure out what caused it to start happening.
Assignee

Comment 24

9 months ago
I can't reproduce this failure locally yet (on 10 thousands runs), so I did push a try run with outputting timestamps.  Here is a part of the log;

00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: waiting for MozAfterPaint
00:20:21 GMT+0000 (Greenwich Mean Time): AfterPaintListener in file:///C:/Users/task_1538611839/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html
00:20:22 GMT+0000 (Greenwich Mean Time): Webrender enabled, sending update whole canvas for invalidation

This shows us that the reftest got stuck at the initial state that we wait for the state where there is no more pending paints.  The fact implies we keep painting even if the opacity animation in the reftest is running on the compositor (or the opacity animation failed to run on the compositor in the first place).

Anyway I am pretty sure this failure caused by bug 1445570 and hopefully will be fixed by bug 1490117. 

That's said, I will figure out a way to work around bug 1490117 here.
URL: 1445570
See Also: → 1490117
Assignee

Comment 25

9 months ago
A workaround seems to work on a try;

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3736fca12276e64d5a3e14eab035678c65c0501b&selectedJob=203270433

There are a bunch of oranges, but all of them are bug 1490117.
Assignee

Comment 26

9 months ago
The animation in this reftests runs on the compositor.  In the mean time,
reftest harness waits for the state where there is no pending paint in the
initial phase (STATE_WAITING_TO_FIRE_INVALIDATE_EVENT, i.e. before sending
a MozReftestInvalidate event).  So if the animation starts running on the
compositor before a MozReftestInvalidate event is received, it means that
the reftest harness has to wait for the 'no pending paint' state until the
animation finishes because the reftest harness keeps flushing styles in the
initial phase which means the animation causes a paint on every flush.

To avoid above situation, we start the animation in question after we get a
MozReftestInvalidate event.
Comment hidden (Intermittent Failures Robot)
Patch in Bug 1496324 also seems to address the problem.

Comment 29

9 months ago
Pushed by hikezoe@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a2ecbcdcec39
Start animation once after a MozReftestInvalidate event is received in continuation-opacity.html. r=kats

Comment 30

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a2ecbcdcec39
Status: ASSIGNED → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
You need to log in before you can comment on or make changes to this bug.