701908 - restartTests/testAddons_installMultipleExtensions causing a hang in testruns

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[u279076]

Attachment: Dialog on Mac

MODULE: /remote/restartTests/testDiscoveryPane_installAddonWithEULA/test1.js

I've been noticing a lot recently that the above test is hanging Firefox on the Add-on Install dialog. This reproduces reliably when testing the recent 9.0b1 and Dave ran into it when investigating endurance testruns on Nightly.

It seems to only occur on Linux and Mac.

The general behaviour is the Firefox hangs on the install dialog. When you Force Quit, Mozmill resumes the testrun but fails to report it to brasstacks and does not spit out an error in console.

My recommendation is that we disable this test ASAP as it is causing huge problems in the reliability of our test results.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Were software update dialogs open from OSX or Ubuntu itself? If yes, were those in the foreground? Whenever another window gets the focus the countdown timer will stop and the install button will not be enabled. Not sure why Mozrunner doesn't force a kill of Firefox in such a case after the jsbridge timeout happend.

Comment 2

[u279076]

The dialogs appear to be in the foreground. However, I have not caught it at time of incident. I typically catch it after it has been hung for a few minutes.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I would propose we raise an MozQA/infrastructure bug so Al can deactivate those update dialogs from appearing. IMO they are highly involved in the broken behavior we have here. I still would see it fixed in Mozmill 1.5.x but there is likely no chance to, and we have to wait for our upgrade to Mozmill 2.x.

Comment 4

[u279076]

Henrik, those "Software Installation" dialogs are not software updates. They are the add-on installation dialogs. I don't think the OS Software Update dialogs are getting in the way in the case of this failure.

Regardless, this test needs to be disabled for further investigation as is blocks us from being able to even run our testruns for releases.

Comment 5

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #4)
> Henrik, those "Software Installation" dialogs are not software updates. They
> are the add-on installation dialogs. I don't think the OS Software Update
> dialogs are getting in the way in the case of this failure.
> 
> Regardless, this test needs to be disabled for further investigation as is
> blocks us from being able to even run our testruns for releases.

Based on your screenshot i don't think this test is the problem 

The installEula test is using the 'Echofon' add-on from amo production to test on

Furthermore, the EULA test is ran only on release branch, and it is being skipped on the other branches due to add-on compatibility issues. 
as we can see clearly from the test code 

http://hg.mozilla.org/qa/mozmill-tests/rev/c32846cb4d40

Is the screenshot the right one? Or maybe another test is causing the issue. 

Note: in the screenshot the add-on is empty.xpi, from local data folder

Comment 6

[u279076]

It is possible that another test is causing this hang, but it definitely one of the recently added Addons tests. I only assumed it was EULA because it was the last test I saw in terminal upon failure.

Comment 7

[Maniac Vlad Florin (:vladmaniac)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #6)
> It is possible that another test is causing this hang, but it definitely one
> of the recently added Addons tests. I only assumed it was EULA because it
> was the last test I saw in terminal upon failure.

No problem - we'll just have to see which test, but having the empty.xpi extension install dialog there it should not be a problem.

Comment 8

[Alex Lakatos[:AlexLakatos]]

The only test using empty.xpi is restartTests/testAddons_installMultipleExtensions, so changing the Summary to show that.
I'll skip that test for now

Comment 9

[Alex Lakatos[:AlexLakatos]]

Attachment: skip patch v1.0 [checked-in]

Skipping the test due to upcoming release testing. Will investigate further tomorrow

Comment 10

[u279076]

Comment on attachment 574314 [details] [diff] [review]
skip patch v1.0 [checked-in]

># HG changeset patch
># User Alex Lakatos <[e-mail]>
># Date 1321288500 -7200
># Node ID 4ba612d968c92a0e036daea438ca6e29afb399ba
># Parent  0cc6f214904cf95cbf7856f89d6d22856af00db6
>imported patch skipMultiple

One nit, please use a more detailed commit message. I will change this on check-in but please use something more descriptive in the future.

"Bug 701908 - disabled testAddons_installMultipleExtensions due to hang"

Comment 11

[u279076]

Comment on attachment 574314 [details] [diff] [review]
skip patch v1.0 [checked-in]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/291b480d1cba (default)
http://hg.mozilla.org/qa/mozmill-tests/rev/d728657e8fca (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/53f2950b2f8a (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/d17ed98def90 (mozilla-release)

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Alex, any update here? I kinda would like to see this test re-enabled.

Comment 13

[Andreea Matei [:AndreeaMatei]]

I have investigated this bug today by running testruns with clean repo (where test installMultipleExtensions is skipped) and also with the test unskipped.
What I have noticed is that the following tests are feezing for 1 minute, either single runned or in testruns, with installMultipleExtensions skipped or unskipped:
* testAddons_enableDisableExtension - test3.js and test4.js
* testAddons_uninstallExtension - test3.js and test4.js
* testAddons_changeTheme - test2.js and test3.js

What these tests have in common is the use of restartLink, all other restart tests are restarting just by switching to the next test.

Despite this fact, all tests are passing.

installMultipleExtensions skipped:
* http://mozmill-crowd.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee609168f

installMultipleExtensions unkipped:
* http://mozmill-crowd.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee60a35b0

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andreea Matei [:AndreeaMatei] from comment #13)
> What I have noticed is that the following tests are feezing for 1 minute,
> either single runned or in testruns, with installMultipleExtensions skipped
> or unskipped:
> * testAddons_enableDisableExtension - test3.js and test4.js
> * testAddons_uninstallExtension - test3.js and test4.js
> * testAddons_changeTheme - test2.js and test3.js

We should file this as a separate bug for investigation. Sounds like it is major enough to fix through we do not see any failure. I haven't noticed that myself yet, but it's great if you can constantly reproduce. So any help would be appreciated.

> http://mozmill-crowd.blargon7.com/#/functional/report/
> d87d47fd1034f072b9bece6ee609168f
> http://mozmill-crowd.blargon7.com/#/functional/report/
> d87d47fd1034f072b9bece6ee60a35b0

You missed to test on Mac. That's certainly helpful to do too.

Comment 15

[Andreea Matei [:AndreeaMatei]]

Mac results:
installMultipleExtensions skipped:
* http://mozmill-crowd.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee62e64b6

installMultipleExtensions unskipped:
* http://mozmill-crowd.blargon7.com/#/functional/report/d87d47fd1034f072b9bece6ee62e8887

The freezing doesn't happen on Mac.
In Ubuntu, after each test freezed I get:
Timeout: bridge.describe("3d659dd2-e5e2-11e1-9eec-0017315c6909", bridge.registry["{95f2b261-e7bf-4efe-9b0e-1c7e70a5a2a5}"])
I will file a separate bug.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so lets try to re-enable the test now. Please come up with a patch which also removes the disabled key from the manifest.

Comment 17

[Andreea Matei [:AndreeaMatei]]

Attachment: enable patch v1

Enabling the test and edit manifest.ini file.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

http://hg.mozilla.org/qa/mozmill-tests/rev/4eca052da816

Comment 19

[Andreea Matei [:AndreeaMatei]]

Attachment: [ESR10] enable patch v1

Patch that enables the test on esr10.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This failure has only been fixed in default before which is now also available in aurora. It still misses to be checked into beta, release, and esr10. So before I want to go ahead with the esr10 patch please check beta and release first.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 661716 [details] [diff] [review]
[ESR10] enable patch v1

Please ask for review again once beta and release are done.

Comment 22

[Andreea Matei [:AndreeaMatei]]

Attachment: [beta, release] enable patch v1

Enables test patch for beta and release branches.

Comment 23

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 661716 [details] [diff] [review]
[ESR10] enable patch v1

Enables test patch for ESR10.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

http://hg.mozilla.org/qa/mozmill-tests/rev/f23143437512 (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/38d66f18912b (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/847463ebfcc6 (esr10)

Andreea, can you please make sure to update the Litmus test for Firefox 10?

Comment 25

[Andreea Matei [:AndreeaMatei]]

Updated tests in Litmus:
* https://litmus.mozilla.org/manage_testcases.cgi?testcase_id=42075
* https://litmus.mozilla.org/manage_testcases.cgi?testcase_id=42611