975254 - reftest fails - different scroll bar

Status: RESOLVED DUPLICATE of bug 986404

(Testing :: Reftest, defect)

Description

[Vincent Chen [:vichen]]

Failed test:
bugs/933264-1.html

Full log:
Bug 974239 https://bugzilla.mozilla.org/attachment.cgi?id=8378879

Comment 1

[Vincent Chen [:vichen]]

Add similar failed test: (https://bugzilla.mozilla.org/show_bug.cgi?id=974241#c2)

bugs/933264-1.html
scrolling/iframe-scrolling-attr-2.html

Comment 2

[Vincent Chen [:vichen]]

Add similar failed test: (https://bugzilla.mozilla.org/show_bug.cgi?id=974243#c4)

bugs/933264-1.html
scrolling/iframe-scrolling-attr-2.html
reftest-sanity/656041-1.html

Comment 3

[Vincent Chen [:vichen]]

Attachment: Simplified-reftest.list

Comment 4

[Vincent Chen [:vichen]]

Attachment: oop-enable-1.log

Comment 5

[Vincent Chen [:vichen]]

ahal,

Please check attach oop-enabled-1.log, the different part is on the scrollbar. For other fail, we have the same pattern that the different is at scroll bar.

As we have animation on scroll bar of B2G, it should be timing issue that we capture wrong frame and make reftests fail. To clarify this, 
1. The attached Simplified-reftest.list contains two reftests, one is reftest-sanity/656041-1.html and itself and the second one is reftest-sanity/656041-1.html and reftest-sanity/656041-1-ref.html.
2. This CSS is to set animation of the scroll bar: http://mxr.mozilla.org/mozilla-central/source/b2g/chrome/content/content.css#52 If I comment this, reftests will pass the both. If keep animation, reftest will fail.

I also dump from code, it show we capture different frame.

Do you know any method we can do to make sure the scrollbar finish animation before do snapshot?

Comment 6

[Andrew Halberstadt [:ahal]]

No, I don't. Maybe there's a pref we can set . Or possibly we should be waiting for transitionend or something in the reftest harness (though this seems like there's potential to drastically increase test run times). Let's ask around about it.

One idea that came to mind would be to create a userChrome.css/userContent.css which completely disables all transitions, but there might be an easier way.

Comment 7

[Andrew Halberstadt [:ahal]]

Or if it's a small number of tests, we could just add this to the test files:
.notransition {
  -webkit-transition: none !important;
  -moz-transition: none !important;
  -o-transition: none !important;
  -ms-transition: none !important;
  transition: none !important;
}

Though it would be better if we could fix this problem in the general case. 
Hiding scrollbars completely might be another solution, vingtetun pointed me at:
http://mxr.mozilla.org/mozilla-central/source/b2g/app/b2g.js#616

Comment 8

[Vincent Chen [:vichen]]

dbaron,

Need your suggestion for this bug.

The reason for some reftest fail are due to animation in scrollbar, and we should have many reftests fail comes from scrollbar animation, here are another example: bug 975287.

We can use this method to hide scrollbar during reftests:
pref(ui.useOverlayScrollbars,0) == reftest-sanity/656041-1.html reftest-sanity/656041-1-ref.html

Do you suggest we use this method to pass reftests? Is there any method we can do to make sure the scrollbar animation is end before doing snapshot?

Comment 9

[Andrew Halberstadt [:ahal]]

We could also set that pref globally in the harness if we want to disable all scrollbars.

Comment 10

[Vincent Chen [:vichen]]

Attachment: Failed tests passed after hide scrollbar

Comment 11

[Vincent Chen [:vichen]]

Attachment: Passed tests failed after hide scroll bar

Comment 12

[Vincent Chen [:vichen]]

Reftests result:
Condition:
1. B2G/InProcess
2. Ignore skip-if(), that is, do all reftests
3. Do two tests with scroll bar show/hide by set pref in b2g.js
pref("ui.useOverlayScrollbars", 1); // 1: show, 0: hide

Results:
1. Some failed test pass after hide scroll bar: https://bugzilla.mozilla.org/attachment.cgi?id=8388315
2. But some passed test failed after hide scroll bar: https://bugzilla.mozilla.org/attachment.cgi?id=8388317
For 2, the abnormal fail seems not to be able to capture test result. Need to check more.

Comment 15

[Vincent Chen [:vichen]]

dbaron,

While doing reftests, there is one strange failure: reftests only capture one page but miss the other page. This kind of failure happen randomly both in OOP and non-OOP. I test single failed reftest and it can pass, so it's not problem of the single reftest. It looks like loading of OS have impact to result of reftests.

Do you have any idea what we can do?
(dump memory usage? cpu usage? or something like that)

Comment 16

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This should be fixed by bug 986404.  (Sorry, I'd forgotten about this bug when mstange told me about the pref, so I filed a new one.)