1489327 - Intermittent css-animations/continuation-opacity.html == css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect)

Description

[Treeherder Bug Filer]

Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=197925697&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Wea24kXvSbmKA-VPcfMj_Q/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1&only_show_unexpected=1

21:53:41     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1536269937/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Comment 1

[Hiroyuki Ikezoe (:hiro)]

See also bug 1415046 that happened only on Windows 7 though.

Comment 2

[Intermittent Failures Robot]

6 failures in 2667 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* autoland: 2

Platform breakdown:
* windows10-64-qr: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 3

[Intermittent Failures Robot]

9 failures in 2684 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 7
* mozilla-inbound: 2

Platform breakdown:
* windows10-64-qr: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 4

[Andreea Pavel [:apavel]]

There is a total of 25 failures in the last 7 days, all on windows10-64-qr.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=200889081&repo=mozilla-central&lineNumber=34595

23:21:34     INFO - REFTEST TEST-START | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html
23:21:34     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html | 60 / 66 (90%)
23:23:14     INFO - REFTEST TEST-LOAD | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | 60 / 66 (90%)
23:23:15     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html == file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity-ref.html | image comparison, max difference: 255, number of differing pixels: 15888

Hiro, are you working on this? are there any updates?

Comment 5

[Hiroyuki Ikezoe (:hiro)]

No, I haven't.

CCing Jeff, this may be a signal of a regression in WebRender.

Note that the failure started from this commit https://hg.mozilla.org/integration/mozilla-inbound/rev/d359982b8ded87c0d69f6032fd38a453b02a001e .

I tried to retrigger R1 jobs on earlier commits, but it seems we are not able to retrigger jobs on inbound any more?

Comment 6

[Hiroyuki Ikezoe (:hiro)]

OK, now I understand what's going on there.

(In reply to Andreea Pavel [:apavel] from comment #4)
> css-animations/continuation-opacity-ref.html
> 23:21:34     INFO - REFTEST TEST-LOAD |
> file:///C:/Users/task_1537571069/build/tests/reftest/tests/layout/reftests/
> css-animations/continuation-opacity.html | 60 / 66 (90%)
> 23:23:14     INFO - REFTEST TEST-LOAD |

Here, the test took 1 min 40 sec, that is 100 sec, which is the animation duration in the test.  So, we didn't get MozReftestInvalidate event.  This is related to bug 1445570 (I think bug 1445570 caused this failure). 

I am going to take a look on next Tuesday.

Comment 7

[Intermittent Failures Robot]

24 failures in 2902 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 14
* mozilla-inbound: 8
* mozilla-central: 2

Platform breakdown:
* windows10-64-qr: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 8

[Hiroyuki Ikezoe (:hiro)]

It seems the failure got infrequent now.

Comment 9

[Andreea Pavel [:apavel]]

(In reply to Hiroyuki Ikezoe (:hiro) (PTO on Sep 26) from comment #8)
> It seems the failure got infrequent now.

The average is still ~25 failures in 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2018-09-18&endday=2018-09-25&tree=all&bug=1489327

Thanks for looking into it.

Comment 10

[Intermittent Failures Robot]

39 failures in 490 pushes (0.08 failures/push) were associated with this bug yesterday.

Repository breakdown:
* try: 1
* mozilla-central: 4
* mozilla-inbound: 20
* autoland: 14

Platform breakdown:
* windows10-64-qr: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-28&endday=2018-09-28&tree=all

Comment 11

[Intermittent Failures Robot]

23 failures in 146 pushes (0.158 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* try: 4
* mozilla-central: 2

Platform breakdown:
* windows10-64-qr: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-29&endday=2018-09-29&tree=all

Comment 12

[Intermittent Failures Robot]

82 failures in 2773 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 37
* mozilla-inbound: 30
* try: 6
* mozilla-central: 9

Platform breakdown:
* windows10-64-qr: 82

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-09-24&endday=2018-09-30&tree=all

Comment 13

[Hiroyuki Ikezoe (:hiro)]

Something has been changed on Sep. 27 or 28.

Moving to WebRender to be seen this in their radar, gfx team should look into this.  If no one has a chance to look into, please send NI to me.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

Most likely a regression from bug 1494206 which landed immediately before the first occurrence of the high-frequency failures. I'm doing some retriggers to verify.

Comment 15

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&searchStr=win,qr,reftest-e10s-1&tochange=43ace919d4d6cca870a9d614d10b52ebd658eb86&group_state=expanded&fromchange=359c5286645408a2f07461a97566dd9448fb46c0

Comment 16

[Kartikaya Gupta (email:[e-mail])]

And from the changes in that WR update, servo/webrender#3092 seems most likely. /cc nical

Comment 17

[Kartikaya Gupta (email:[e-mail])]

Wait, that's not right. There are a couple of these failure before that landed.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

I think the couple of failures on the previous push are a fluke (this test was failing intermittently before too, just at a really low volume). I don't see any other failures on retriggers prior to that one, and that push is a ES lint warning fix so it's not likely to be an actual culprit. Putting back the finger on bug 1494206.

Comment 19

[Intermittent Failures Robot]

31 failures in 485 pushes (0.064 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 13
* mozilla-central: 6
* try: 3
* mozilla-inbound: 9

Platform breakdown:
* windows10-64-qr: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-10-01&endday=2018-10-01&tree=all

Comment 20

[Intermittent Failures Robot]

57 failures in 561 pushes (0.102 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 15
* autoland: 34
* mozilla-central: 6
* try: 2

Platform breakdown:
* windows10-64-qr: 57

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-10-02&endday=2018-10-02&tree=all

Comment 21

[Eliza Balazs [:ebalazs_]]

hiro: Hi, according to Comment 13, can you please have a look into this? Thank you!

Comment 22

[Hiroyuki Ikezoe (:hiro)]

Though I think kats already has started looking into this, I am going to look into this in detail from animation point of view.  Assigning to myself, FWIW.

Comment 23

[Kartikaya Gupta (email:[e-mail])]

I'm not actually investigating the failure; I just tried to figure out what caused it to start happening.

Comment 24

[Hiroyuki Ikezoe (:hiro)]

I can't reproduce this failure locally yet (on 10 thousands runs), so I did push a try run with outputting timestamps.  Here is a part of the log;

00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
00:18:41 GMT+0000 (Greenwich Mean Time): MakeProgress: waiting for MozAfterPaint
00:20:21 GMT+0000 (Greenwich Mean Time): AfterPaintListener in file:///C:/Users/task_1538611839/build/tests/reftest/tests/layout/reftests/css-animations/continuation-opacity.html
00:20:22 GMT+0000 (Greenwich Mean Time): Webrender enabled, sending update whole canvas for invalidation

This shows us that the reftest got stuck at the initial state that we wait for the state where there is no more pending paints.  The fact implies we keep painting even if the opacity animation in the reftest is running on the compositor (or the opacity animation failed to run on the compositor in the first place).

Anyway I am pretty sure this failure caused by bug 1445570 and hopefully will be fixed by bug 1490117. 

That's said, I will figure out a way to work around bug 1490117 here.

Comment 25

[Hiroyuki Ikezoe (:hiro)]

A workaround seems to work on a try;

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3736fca12276e64d5a3e14eab035678c65c0501b&selectedJob=203270433

There are a bunch of oranges, but all of them are bug 1490117.

Comment 26

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1489327 - Start animation once after a MozReftestInvalidate event is received in continuation-opacity.html. r=kats

The animation in this reftests runs on the compositor.  In the mean time,
reftest harness waits for the state where there is no pending paint in the
initial phase (STATE_WAITING_TO_FIRE_INVALIDATE_EVENT, i.e. before sending
a MozReftestInvalidate event).  So if the animation starts running on the
compositor before a MozReftestInvalidate event is received, it means that
the reftest harness has to wait for the 'no pending paint' state until the
animation finishes because the reftest harness keeps flushing styles in the
initial phase which means the animation causes a paint on every flush.

To avoid above situation, we start the animation in question after we get a
MozReftestInvalidate event.

Comment 27

[Intermittent Failures Robot]

60 failures in 465 pushes (0.129 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 24
* autoland: 18
* mozilla-central: 11
* try: 7

Platform breakdown:
* windows10-64-qr: 60

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-10-03&endday=2018-10-03&tree=all

Comment 28

[Sotaro Ikeda [:sotaro]]

Patch in Bug 1496324 also seems to address the problem.

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a2ecbcdcec39
Start animation once after a MozReftestInvalidate event is received in continuation-opacity.html. r=kats

Comment 30

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/a2ecbcdcec39

Comment 31

[Intermittent Failures Robot]

30 failures in 553 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 1
* try: 2
* mozilla-inbound: 7
* autoland: 20

Platform breakdown:
* windows10-64-qr: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-10-04&endday=2018-10-04&tree=all

Comment 32

[Intermittent Failures Robot]

178 failures in 2708 pushes (0.066 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 85
* mozilla-inbound: 55
* try: 14
* mozilla-central: 24

Platform breakdown:
* windows10-64-qr: 178

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1489327&startday=2018-10-01&endday=2018-10-07&tree=all