Closed Bug 998253 Opened 8 years ago Closed 8 years ago

Test failure "The tab with index '1' has been selected" in "/testSearch/testSearchSelection.js"

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

All
macOS
defect

Tracking

(firefox32 fixed, firefox33 fixed, firefox34 fixed, firefox-esr24 unaffected, firefox-esr31 unaffected)

RESOLVED FIXED
Tracking Status
firefox32 --- fixed
firefox33 --- fixed
firefox34 --- fixed
firefox-esr24 --- unaffected
firefox-esr31 --- unaffected

People

(Reporter: cosmin-malutan, Assigned: danisielm)

References

()

Details

(Keywords: intermittent-failure, Whiteboard: [mozmill-test-failure][sprint])

Attachments

(2 files, 3 obsolete files)

This failed today again, with Beta pl on Windows 7 x86.
>http://mozmill-release.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-08&to=&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab
>http://mozmill-daily.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-03-08&to=&test=%2FtestTabbedBrowsing%2FtestOpenInBackground.js&func=testOpenInBackgroundTab
It failed 4 times since the fix from bug 880135 was landed. I tried to reproduce the failure with the testcase from bug 880135, but with that fix it passes, so I think the underlying issue is different here, and I filed this bug for tracking.
I couldn't reproduce this.
I ran the testcase for 2000 times and it didn't fail.
We see the same failure in another test: 
functional/testSearch/testSearchSelection.js

This failed on Nightly (33.0a1 en-US) on mm-osx-108-1.
http://mozmill-daily.blargon7.com/#/functional/report/6959f9a82369610967633890833ab388

This is the first failure until now:
http://mozmill-release.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-05-01&to=&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu
Also failures on daily with Aurora:

http://mozmill-daily.blargon7.com/#/functional/failure?app=All&branch=All&platform=All&from=2014-05-01&to=&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu

Looks like 40 failures this month.

Given that those failure are in another test , should we increase the proirity on this or file a new bug?
Since this is library-related we should figure out all of them here (they might very well have the same cause).
Priority: P4 → P2
Hmm actually the initial reports were on Windows while the recent failures are all on OSX. They might indeed have different causes.
However the initial failure hasn't been seen in 2 months now, we might as well hijack this bug for the new failures.
Failed 7 times today OSX nodes, different locales.
Summary: Test failure "The tab with index '1' has been selected" → Test failure "The tab with index '1' has been selected" in "/testSearch/testSearchSelection.js"
In the last 10 days we have between 3 and 8 failures a day because of this.
We may want to skip this (at least on Mac) before we have time to investigate.
OS: Windows 7 → Mac OS X
We definitely want to fix this, failed 21 times over the weekend, I'll try to reproduce this one as well.
This didn't reproduced locally on my OS X. I will look more on a remote machine next week.
We're now averaging 5 failures per day.
If we don't find a fix fast, we might skip this test for now.
This started to fail since 29 May 2014 & only on Mac OSX.

Here is a pushlog from the day before & when the failure started:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e017c15325ae&tochange=1e712b724d17

It failed almost every day since then.
Attached patch skip.patch (obsolete) — Splinter Review
Continues to fail a lot, let's unskip it.

Run the test and:

200 runs locally on a 10.8 - no failure
100 runs on mm-osx-109-4 - 60% failure rate.

I will try to find what causes this.
Assignee: nobody → daniel.gherasim
Status: NEW → ASSIGNED
Attachment #8456130 - Flags: review?(andrei.eftimie)
Attachment #8456130 - Flags: review?(andreea.matei)
Comment on attachment 8456130 [details] [diff] [review]
skip.patch

Review of attachment 8456130 [details] [diff] [review]:
-----------------------------------------------------------------

Seems to only affect OSX, I haven't seen failures on other platforms.
Please update the patch to only disable the test on the affected platform.
Attachment #8456130 - Flags: review?(andrei.eftimie)
Attachment #8456130 - Flags: review?(andreea.matei)
Attachment #8456130 - Flags: review-
Attached patch skip.patch (obsolete) — Splinter Review
Attachment #8456130 - Attachment is obsolete: true
Attachment #8456146 - Flags: review?(andrei.eftimie)
Attachment #8456146 - Flags: review?(andreea.matei)
Attached patch skip.patchSplinter Review
Attachment #8456146 - Attachment is obsolete: true
Attachment #8456146 - Flags: review?(andrei.eftimie)
Attachment #8456146 - Flags: review?(andreea.matei)
Attachment #8456148 - Flags: review?(andrei.eftimie)
If we don't use the open in background option the test runs fine.
e.g. changing true to false here:
https://hg.mozilla.org/qa/mozmill-tests/file/b8637d05a1d3/firefox/tests/functional/testSearch/testSearchSelection.js#l59


I'll continue the investigation.
Priority: P2 → P1
Comment on attachment 8456148 [details] [diff] [review]
skip.patch

Review of attachment 8456148 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled on OSX:
https://hg.mozilla.org/qa/mozmill-tests/rev/34028f725fea (default)
Attachment #8456148 - Flags: review?(andrei.eftimie)
Attachment #8456148 - Flags: review+
Attachment #8456148 - Flags: checkin+
This needs to pe landed on aurora as well.
Is there no way to add a description for skip-if? As of right now no-one will know why this test is skipped! Even a comment would have worked. :(
(In reply to Henrik Skupin (:whimboo) [away 07/19 - 08/01] from comment #20)
> Is there no way to add a description for skip-if? As of right now no-one
> will know why this test is skipped! Even a comment would have worked. :(

We can add a comment but won't be shown in the console I guess, like:
[test1.js]
# Bug XXXXXXX - Failure ...
skip-if = os = 'mac'

--
An update here:
This is the line that fails:
https://hg.mozilla.org/qa/mozmill-tests/file/b8637d05a1d3/firefox/tests/functional/testSearch/testSearchSelection.js#l107

With a 'sleep(100);' above this line the test always pass, so might be that we don't properly wait for the tab to be opened so we can click on it.
This is how we can show it in the console:
skip-if = os == 'mac' = Bug 123 - description

Example:
TEST-SKIPPED | test1.js | skip-if: os == 'linux' = Bug 1018161 - Disable test because of failures
TEST-SKIPPED | test2.js | skip-if: os == 'linux' = Bug 1018161 - Disable test because of failures
RESULTS | Passed: 0
RESULTS | Failed: 0
RESULTS | Skipped: 2
Thanks Andreea for that tip,
--
Back on the failure now:

Looks like a race condition that's happening only on the CI machines, not surely why only on those.

But this is what it happens:

1. Open a tab in background
2. Wait for it to be opened
3. Click on the tab to focus the page
- Here the test fails, we are not waiting properly for the tab animation to finish.
- I guess the busy attribute we are waiting for here is not all we want
- To check this I printed the coordinates where we click & here are some of the printed lines:
------
aOffsetX : 23.383331298828125, aOffsetY : 15.5
* Not the correct offsetX but the test PASSES because we click on a part of the tab that's animated

aOffsetX: 22.650001525878906, aOffsetY : 15.5
* Test failed, we clicked on the 'New Tab' button so another unexpected tab is opened

aOffsetX: 0.0500030517578125, aOffsetY : 15.5
* Abviously failed

aOffsetX: 105, aOffsetY : 15.5
* Test passed!
* This arre the correct offsets we should get all the time!
Great, waiting for the same transitions as in the newTab() method fixes this.
With bug 1036825 we can use openTab with a callback & this failures are gone.
Depends on: 1036825
Whiteboard: [mozmill-test-failure] → [mozmill-test-failure][Blocked by 1036825]
Once with the merge the test is now skipped on aurora & default. 
We need to skip this on beta, as it's also affected.
Attached patch fix-998253.patch (obsolete) — Splinter Review
Due to bug 1036825 we can now give a callback to openTab method.
This can be landed *only on default* until we make backports on the dependency bug.

I run 20 tests without this patch & it failed 7 times on mm-osx-109-4.
With the patch no failure in 100 runs.

The test still remain skipped due to bug 1041420, but this one we can fix.
Attachment #8461389 - Flags: review?(andrei.eftimie)
Comment on attachment 8461389 [details] [diff] [review]
fix-998253.patch

Review of attachment 8461389 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me, but let's wait for bug 1041420 to be fixed (should be today or tomorrow) and as you said we'll need bug 1036825 landed across branches.
Attachment #8461389 - Flags: review?(andrei.eftimie) → review+
No longer blocked, this can be landed on default as bug 141420 was already fixed on that branch.
Whiteboard: [mozmill-test-failure][Blocked by 1036825] → [mozmill-test-failure]
Sorry, I mean bug 1041420.
All dependency are now fixed.
This applies cleanly on default & aurora.
Attachment #8461389 - Attachment is obsolete: true
Actually applies on beta too.
Attachment #8463966 - Flags: review?(andrei.eftimie)
Comment on attachment 8463966 [details] [diff] [review]
fix-998253_with-unskip.patch

Review of attachment 8463966 [details] [diff] [review]:
-----------------------------------------------------------------

Nice and easy fix.
Ran some testruns against Aurora and Beta and everything works fine.

https://hg.mozilla.org/qa/mozmill-tests/rev/01111ecbe85d (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/3b038ec35171 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/db79fb0489b2 (mozilla-beta)

(I did change the commit message to include what we changed, not the failure message)
Attachment #8463966 - Flags: review?(andrei.eftimie)
Attachment #8463966 - Flags: review+
Attachment #8463966 - Flags: checkin+
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Seems our fix wasn't complete:

We still have some failures on 32 OSX:
http://mozmill-release.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2014-07-30&test=%2FtestSearch%2FtestSearchSelection.js&func=testSearchSelectionViaContextMenu

We haven't seen any failures on newer branches, and while all above mentioned failures are on localised versions, I tried reproducing the failure on some of them without success.

This is an intermittent failure with a relative low occurrence rate, only affecting OSX.
Status: RESOLVED → REOPENED
Priority: P1 → P2
Resolution: FIXED → ---
This didn't fail in a long time (more than 1 month).
It's not reproducible at all and given that we now run more locales on beta and we got no failure at all, I would consider this as WFM.
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → WORKSFORME
Whiteboard: [mozmill-test-failure] → [mozmill-test-failure][sprint]
I would still call this fixed as it had a fix landed and we only had some intermittent fails that might not be related and didn't happened again.
Resolution: WORKSFORME → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.