1111568 - Test failure 'Number of visible rows should equal 6' in /testAwesomeBar/testVisibleItemsMax.js

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[Mihaela Velimiroviciu (:mihaelav)]

Module:    testVisibleItemsMax
Test:      /testAwesomeBar/testVisibleItemsMax.js
Failure:   Number of visible rows should equal 6
Branches:  Nightly (37.0a1), Aurora (36.0a2)
Platforms: Ubuntu 14.04

Reports:
http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2014-12-08&to=2014-12-15&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax

Comment 1

[Cosmin Malutan, [:cosmin-malutan]]

This failed for 18 times for with the latest beta spin on Ubuntu 14.04 nodes, it should be skipped.

Comment 2

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: skip-patch beta

Here is the skip patch for beta.

Comment 3

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8542536 [details] [diff] [review]
skip-patch beta

Review of attachment 8542536 [details] [diff] [review]:
-----------------------------------------------------------------

We need to figure out what's different for 14.04 machines.

http://hg.mozilla.org/qa/mozmill-tests/rev/05d3a0ce4f5d (beta)

Comment 4

[Teodor Druta]

We had a few failures on Nightly 31.3.0esr
http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=31&platform=All&from=2014-12-29&to=&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax

Comment 5

[Kamil Jozwiak [:kjozwiak]]

Had a failure with fx 31.4.0 RC Build1 under Ubuntu 14.04:

* http://mozmill-release.blargon7.com/#/functional/report/5c7e7a22f9984b46554bc3b9fc6fbfd6
* Build being used: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/31.4.0esr-candidates/build1/

Comment 6

[Mihaela Velimiroviciu (:mihaelav)]

Many failures these days across all branches:
* http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2015-01-06&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax
* http://mozmill-release.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2015-01-06&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax

Comment 7

[Teodor Druta]

Attachment: skipvsblitems.patch

We decided to skip this test on mozilla-beta, because of multiple failures. Attached is the skip patch.

Comment 8

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8548151 [details] [diff] [review]
skipvsblitems.patch

Review of attachment 8548151 [details] [diff] [review]:
-----------------------------------------------------------------

http://hg.mozilla.org/qa/mozmill-tests/rev/1cf1e0c36b25 (beta)

Comment 9

[Andreea Matei [:AndreeaMatei]]

Haven't reproduced this on a local Ubuntu 14.04 machine with or locale in over 50 runs. I'll try on a staging machine.

Comment 10

[Daniela Domnici]

Failed 14 times over the weekend with Nightly, Aurora and ESR31 on Ubuntu 14.04 and Ubuntu 14.10.

http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2015-01-23&to=2015-01-26&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax

Comment 11

[Mihaela Velimiroviciu (:mihaelav)]

I reproduced this intermittently on staging and locally, but only if the tests from other folders (remote, addons) are ran before it. I'm working on finding a smaller amount of tests which help reproduce this when they run.

Comment 12

[Daniela Domnici]

Failed again over the weekend with Nightly and Aurora.

Reports: http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2015-02-07&to=2015-02-09&test=%2FtestAwesomeBar%2FtestVisibleItemsMax.js&func=testVisibleItemsMax

Comment 13

[Mihaela Velimiroviciu (:mihaelav)]

The failure isn't reproducing when running the testVisibleItemsMax.js alone. It reproduces intermittently by running the testSwitchToTab.js test before. 
The problem is that, for some reason, the autocomplete results popup list doesn't have the expected height the time: sometimes, it shows only 1 or 2 entries, the rest being visible only by dragging the scrollbar. 

I'm not sure where else to look on our side. Maybe it's a Firefox bug eventually...?

Comment 14

[Teodor Druta]

Attachment: fix_visible_items.patch

This was really hard to track, but I finally figured it out. The issue we have here is related to the fact that the autocomple richlistbox element's height is computed programmatically, this is the code at fault: http://hg.mozilla.org/mozilla-central/rev/0fc382c36b4c. I actually have past front-end development experience, and I'm acquainted with this approach, and from my experience something like this will not always work when it's automated, and I will take the liberty to say that this is not related to firefox, but to all major browsers.
What happens here: 1st. we run the testSwitchToTab.js, that opens/closes multiple times in a sequence so fast that the code which computes the auto complete popup height fails to finish, this is a matter of milliseconds, this results in the auto complete popup becoming kinda 'broken' with an incorrect height value, we are testing the auto complete popup height in testVisibleItemsMax.js, this is why this test fails only when it's run together with the previous one.

One thing we can do here is to place testVisibleItemsMax.js before testSwitchToTab.js => imho bad approach which will just mean that we are running from the problem instead of fixing it.

What I decided to do, was investigating some of the firefox code to see if there is something we miss in toolbars.js, so I discovered that there are some event listeners that have not been implemented in toolbars.js. I treated those resulting in a big drop in the failure rate, but this still didn't fixed fully the problem, I had no choice but to add a small sleep of 300ms in order to fix all the failures. One of my ideas besides this approach was to wait for the height attribute to change for the autocomplete richlistbox element, this is impossible to implement because, between one instance of autocomplete being open and the next one this value may be the same, so we don't have a way to predict if opening an automplete instance after another will result in a height change.

So my final conclusion: this is not a Firefox bug, imo this is impossible to reproduce manually, unless you're a super saiyan with too much time on your hands and the only thing you like to do with your computer and web browser is to open/close the autocomplete popup multiple times.

Comment 15

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8563468 [details] [diff] [review]
fix_visible_items.patch

Review of attachment 8563468 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 16

[Teodor Druta]

Well, the fix was provided everything was explained in comment #14. :(

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sadly this test was not skipped on default and aurora yet. So I transplanted the skip patch now.

https://hg.mozilla.org/qa/mozmill-tests/rev/724ddaa47716 (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/69146a64c745 (aurora)

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8563468 [details] [diff] [review]
fix_visible_items.patch

Review of attachment 8563468 [details] [diff] [review]:
-----------------------------------------------------------------

We are most likely not going to get this fix landed. Instead we want to make sure all works fine for the Marionette tests directly. We simply want to reduce possible test failures here.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I will keep this bug open until we have a Firefox UI test replacement.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill tests have been superseded by Marionette tests.