998253 - Test failure "The tab with index '1' has been selected" in "/testSearch/testSearchSelection.js"

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[Cosmin Malutan, [:cosmin-malutan]]

This failed today again, with Beta pl on Windows 7 x86.
>http://mozmill-release.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-08&to=&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab
>http://mozmill-daily.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-08&to=&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab
It failed 4 times since the fix from bug 880135 was landed. I tried to reproduce the failure with the testcase from bug 880135, but with that fix it passes, so I think the underlying issue is different here, and I filed this bug for tracking.
I couldn't reproduce this.

Comment 1

[Cosmin Malutan, [:cosmin-malutan]]

I ran the testcase for 2000 times and it didn't fail.

Comment 2

[Andrei Eftimie]

Failed again.

We have a total of 6 instances in 2 months.
4 on release runs:
http://mozmill-release.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-18&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab
and 2 on daily runs:
http://mozmill-daily.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-18&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab


All are on Windows. 4 on Win7 and 2 on Win8

Comment 3

[Daniel Gherasim]

We see the same failure in another test: 
functional/testSearch/testSearchSelection.js

This failed on Nightly (33.0a1 en-US) on mm-osx-108-1.
http://mozmill-daily.blargon7.com/#/functional/report/6959f9a82369610967633890833ab388

This is the first failure until now:
http://mozmill-release.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-05-01&to=&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu

Comment 4

[Daniel Gherasim]

Also failures on daily with Aurora:

http://mozmill-daily.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-05-01&to=&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu

Looks like 40 failures this month.

Given that those failure are in another test , should we increase the proirity on this or file a new bug?

Comment 5

[Andrei Eftimie]

Since this is library-related we should figure out all of them here (they might very well have the same cause).

Comment 6

[Andrei Eftimie]

Hmm actually the initial reports were on Windows while the recent failures are all on OSX. They might indeed have different causes.
However the initial failure hasn't been seen in 2 months now, we might as well hijack this bug for the new failures.

Comment 7

[Daniel Gherasim]

Failed 7 times today OSX nodes, different locales.

Comment 8

[Daniel Gherasim]

In the last 10 days we have between 3 and 8 failures a day because of this.
We may want to skip this (at least on Mac) before we have time to investigate.

Comment 9

[Andreea Matei [:AndreeaMatei]]

We definitely want to fix this, failed 21 times over the weekend, I'll try to reproduce this one as well.

Comment 10

[Andreea Matei [:AndreeaMatei]]

This didn't reproduced locally on my OS X. I will look more on a remote machine next week.

Comment 11

[Andrei Eftimie]

We're now averaging 5 failures per day.
If we don't find a fix fast, we might skip this test for now.

Comment 12

[Daniel Gherasim]

This started to fail since 29 May 2014 & only on Mac OSX.

Here is a pushlog from the day before & when the failure started:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e017c15325ae&tochange=1e712b724d17

It failed almost every day since then.

Comment 13

[Daniel Gherasim]

Attachment: skip.patch

Continues to fail a lot, let's unskip it.

Run the test and:

200 runs locally on a 10.8 - no failure
100 runs on mm-osx-109-4 - 60% failure rate.

I will try to find what causes this.

Comment 14

[Andrei Eftimie]

Comment on attachment 8456130 [details] [diff] [review]
skip.patch

Review of attachment 8456130 [details] [diff] [review]:
-----------------------------------------------------------------

Seems to only affect OSX, I haven't seen failures on other platforms.
Please update the patch to only disable the test on the affected platform.

Comment 15

[Daniel Gherasim]

Attachment: skip.patch

Comment 16

[Daniel Gherasim]

Attachment: skip.patch

Comment 17

[Daniel Gherasim]

If we don't use the open in background option the test runs fine.
e.g. changing true to false here:
https://hg.mozilla.org/qa/mozmill-tests/file/b8637d05a1d3/firefox/tests/functional/testSearch/testSearchSelection.js#l59


I'll continue the investigation.

Comment 18

[Andrei Eftimie]

Comment on attachment 8456148 [details] [diff] [review]
skip.patch

Review of attachment 8456148 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled on OSX:
https://hg.mozilla.org/qa/mozmill-tests/rev/34028f725fea (default)

Comment 19

[Daniel Gherasim]

This needs to pe landed on aurora as well.

Comment 20

[Henrik Skupin (:whimboo) [⌚️UTC+1]]

Is there no way to add a description for skip-if? As of right now no-one will know why this test is skipped! Even a comment would have worked. :(

Comment 21

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) [away 07/19 - 08/01] from comment #20)
> Is there no way to add a description for skip-if? As of right now no-one
> will know why this test is skipped! Even a comment would have worked. :(

We can add a comment but won't be shown in the console I guess, like:
[test1.js]
# Bug XXXXXXX - Failure ...
skip-if = os = 'mac'

--
An update here:
This is the line that fails:
https://hg.mozilla.org/qa/mozmill-tests/file/b8637d05a1d3/firefox/tests/functional/testSearch/testSearchSelection.js#l107

With a 'sleep(100);' above this line the test always pass, so might be that we don't properly wait for the tab to be opened so we can click on it.

Comment 22

[Andreea Matei [:AndreeaMatei]]

This is how we can show it in the console:
skip-if = os == 'mac' = Bug 123 - description

Example:
TEST-SKIPPED | test1.js | skip-if: os == 'linux' = Bug 1018161 - Disable test because of failures
TEST-SKIPPED | test2.js | skip-if: os == 'linux' = Bug 1018161 - Disable test because of failures
RESULTS | Passed: 0
RESULTS | Failed: 0
RESULTS | Skipped: 2

Comment 23

[Daniel Gherasim]

Thanks Andreea for that tip,
--
Back on the failure now:

Looks like a race condition that's happening only on the CI machines, not surely why only on those.

But this is what it happens:

1. Open a tab in background
2. Wait for it to be opened
3. Click on the tab to focus the page
- Here the test fails, we are not waiting properly for the tab animation to finish.
- I guess the busy attribute we are waiting for here is not all we want
- To check this I printed the coordinates where we click & here are some of the printed lines:
------
aOffsetX : 23.383331298828125, aOffsetY : 15.5
* Not the correct offsetX but the test PASSES because we click on a part of the tab that's animated

aOffsetX: 22.650001525878906, aOffsetY : 15.5
* Test failed, we clicked on the 'New Tab' button so another unexpected tab is opened

aOffsetX: 0.0500030517578125, aOffsetY : 15.5
* Abviously failed

aOffsetX: 105, aOffsetY : 15.5
* Test passed!
* This arre the correct offsets we should get all the time!

Comment 24

[Daniel Gherasim]

Great, waiting for the same transitions as in the newTab() method fixes this.
With bug 1036825 we can use openTab with a callback & this failures are gone.

Comment 25

[Daniel Gherasim]

Once with the merge the test is now skipped on aurora & default. 
We need to skip this on beta, as it's also affected.

Comment 26

[Andrei Eftimie]

Transplanted skip on Beta:
https://hg.mozilla.org/qa/mozmill-tests/rev/5854ccfa7e1f (mozilla-beta)

Comment 27

[Daniel Gherasim]

Attachment: fix-998253.patch

Due to bug 1036825 we can now give a callback to openTab method.
This can be landed *only on default* until we make backports on the dependency bug.

I run 20 tests without this patch & it failed 7 times on mm-osx-109-4.
With the patch no failure in 100 runs.

The test still remain skipped due to bug 1041420, but this one we can fix.

Comment 28

[Andrei Eftimie]

Comment on attachment 8461389 [details] [diff] [review]
fix-998253.patch

Review of attachment 8461389 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me, but let's wait for bug 1041420 to be fixed (should be today or tomorrow) and as you said we'll need bug 1036825 landed across branches.

Comment 29

[Daniel Gherasim]

No longer blocked, this can be landed on default as bug 141420 was already fixed on that branch.

Comment 30

[Daniel Gherasim]

Sorry, I mean bug 1041420.

Comment 31

[Daniel Gherasim]

Attachment: fix-998253_with-unskip.patch

All dependency are now fixed.
This applies cleanly on default & aurora.

Comment 32

[Daniel Gherasim]

Actually applies on beta too.

Comment 33

[Andrei Eftimie]

Comment on attachment 8463966 [details] [diff] [review]
fix-998253_with-unskip.patch

Review of attachment 8463966 [details] [diff] [review]:
-----------------------------------------------------------------

Nice and easy fix.
Ran some testruns against Aurora and Beta and everything works fine.

https://hg.mozilla.org/qa/mozmill-tests/rev/01111ecbe85d (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/3b038ec35171 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/db79fb0489b2 (mozilla-beta)

(I did change the commit message to include what we changed, not the failure message)

Comment 34

[Andrei Eftimie]

Seems our fix wasn't complete:

We still have some failures on 32 OSX:
http://mozmill-release.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2014-07-30&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu

We haven't seen any failures on newer branches, and while all above mentioned failures are on localised versions, I tried reproducing the failure on some of them without success.

This is an intermittent failure with a relative low occurrence rate, only affecting OSX.

Comment 35

[Daniel Gherasim]

This didn't fail in a long time (more than 1 month).
It's not reproducible at all and given that we now run more locales on beta and we got no failure at all, I would consider this as WFM.

Comment 36

[Andreea Matei [:AndreeaMatei]]

I would still call this fixed as it had a fix landed and we only had some intermittent fails that might not be related and didn't happened again.