609058 - Timeout waiting for installed extension in testExtensionInstallGetAddons/test1.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[u279076]

MODULE: restartTests/testExtensionInstallGetAddons/test1.js
TEST: testInstallExtension
ERROR: Timeout exceeded for waitForElement Lookup: /id("extensionsManager")/id("addonsMsg")/id("extensionsBox")/[1]/id("extensionsView")/{"addonID":"{8620c15f-30dc-4dba-a131-7c5d20cf4a29}"}
BRANCH: mozilla1.9.1, mozilla1.9.2
PLATFORM: All

Comment 1

[u279076]

Attachment: WIP

I've created a WIP patch and discovered an issue (probably a likely cause for the failure).

In this patch, I've added:
1) a forced click on the Get-Addons pane so we ensure it is loaded.
2) a verification that there is a list of Recommended Add-ons
* both of these ensures the Add-ons Manager is in a searchable state

However, I've discovered that the list does not populate in the Mozmill test.  This is most likely due to the use of preview.amo instead of amo.  I've confirmed this by running through the steps manually.  Can we not just use live AMO in these tests?

Comment 2

[Aaron Train [:aaronmt]]

"Can we not just use live AMO in these tests?" I was told earlier that we don't want to skew download stats

Comment 3

[u279076]

I just switched the test over to using live-AMO and it fixes this test.

So, either we skew their stats or we get them to fix preview-amo.

Comment 4

[u279076]

(In reply to comment #3)
> I just switched the test over to using live-AMO and it fixes this test.
> 
> So, either we skew their stats or we get them to fix preview-amo.

Frankly, I find it hard to believe there isn't someway for them to blacklist the stats coming from Mozmill testing; the same as socorro blacklists stats coming from the CrashMe.xpi extension.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I have seen something similar the last days but it was working some hours later. So not sure if this is just a flaky connection. It shouldn't happen. Can you please talk to fligtar about this issue? preview.amo is the upcoming new version, so if there is a failure, we should make sure it doesn't get published to amo later this week.

Comment 6

[u279076]

Adding Fligtar to this bug so he can give feedback on this issue.

Fligtar, for clarity, we are seeing that the Recommended Add-ons list (nor search) is populated with a list of Extensions when using the Add-on Manager and Preview.AMO.  This affects both Namoroka (Firefox 3.6) and Shiretoko (Firefox 3.5).

The following are the prefs we are using:

extensions.getAddons.browseAddons
https://preview.addons.mozilla.org/%LOCALE%/%APP%

extensions.getAddons.recommended.browseURL
https://preview.addons.mozilla.org/%LOCALE%/%APP%/recommended

extensions.getAddons.recommended.url
https://preview.addons.mozilla.org/%LOCALE%/%APP%/api/%API_VERSION%/list/featured/all/10/%OS%/%VERSION%

extensions.getAddons.search.browseURL
https://preview.addons.mozilla.org/%LOCALE%/%APP%/search?q=%TERMS%

extensions.getAddons.search.url
https://preview.addons.mozilla.org/%LOCALE%/%APP%/api/%API_VERSION%/search/%TERMS%/all/10/%OS%/%VERSION%

Fligtar, is this an AMO problem?

Comment 7

[u279076]

> Fligtar, for clarity, we are seeing that the Recommended Add-ons list (nor
> search) is populated with a list of Extensions when using the Add-on Manager
> and Preview.AMO.

Sorry, that should read:

we are NOT seeing that the Recommended Add-ons list (nor search) is populated with a list of Extensions when using the Add-on Manager and Preview.AMO.

Comment 8

[u279076]

One other thing I'd like to note:

Firefox 4 does not use the two "recommended" prefs (ie. they do not exist on a default profile).

Comment 9

[u279076]

After some discussion with Fligtar on IRC, an API issue was discovered yesterday.  Blocking on bug 609187.

Comment 10

[u279076]

Attachment: Patch v1 (1.9.2)

First attempt at a patch now that the AMO API issue is resolved.

Two things to note:
1. The default timeout of 5s on waitFor is not always long enough for the dynamically generated Add-ons list in the Add-on Manager.  This is especially true for slower connections.  Hence, I've increased it in this test to 10s.

2. Addons installed from preview.amo have a reverse domain (ie. addons.allizom.org instead of addons.mozilla.org).  I've made this correction in the test.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 488357 [details] [diff] [review]
Patch v1 (1.9.2)

>+  // Wait for the Recommended Add-ons to display (default is 5)
>+  // XXX: We can potentially fail here on slow connections 
>+  //      waiting for the add-ons to load
>+  var recommendedAddonsList = addonsManager.getElement({type: "listbox"});
>+  controller.waitFor(function() {
>+    return recommendedAddonsList.getNode().itemCount == 5;
>+  }, "Number of Recommended Add-ons listed should be 5.", 10000);

There is gSearchTimeout defined globally we should use in this case. Also 5 should be replaced by the value set for the max results preference.

Comment 12

[u279076]

> There is gSearchTimeout defined globally we should use in this case. Also 5
> should be replaced by the value set for the max results preference.

I've done some research and talked to Mossop about this.

Firstly, we have the pref extensions.getAddons.maxResults=15 which limits the list to 15.  Second, this list gets limited by a parameter in the AMO URL (currently 10).  Third, this list gets limited more by the number of compatible add-ons and installed add-ons.  Note, there is no pref which explicitly controls "Recommended Add-ons", the same pref is used for recommended and search.

So, this list could be anywhere from 0 to 15 addons.  I'm not sure what the most reliable way to check this is.  Perhaps itemCount > 0?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Move of Mozmill Test related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is this test still valid? The modal dialog rewrite should have mostly bit-rotted this patch.

(In reply to comment #12)
> So, this list could be anywhere from 0 to 15 addons.  I'm not sure what the
> most reliable way to check this is.  Perhaps itemCount > 0?

We could set the max value to 2, and use a search term like "video" or "download" which should always return more than 2 results. Also the number in the "show all results" link below should always be higher than the max value.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Anthony, can you please give an update about the current state? This test is failing now for nearly 2 months. We should get it fixed or disabled as soon as possible.

Comment 16

[u279076]

(In reply to comment #15)
> Anthony, can you please give an update about the current state? This test is
> failing now for nearly 2 months. We should get it fixed or disabled as soon as
> possible.

I've not been able to look at this since mid-November. I'll make it a priority for the test-fails project.  If you think there is value in disabling the test in the meantime, go for it.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #16)
> I've not been able to look at this since mid-November. I'll make it a priority
> for the test-fails project.  If you think there is value in disabling the test
> in the meantime, go for it.

We talked about disabling failed tests, which cannot be fixed in a short manner, during our work week beginning of December. There are also meeting notes.

Anthony, you are assigned to this bug, so please upload a quick patch which will skip this test on all platforms. I'm happy to review and land it if necessary.

Comment 18

[u279076]

> We talked about disabling failed tests, which cannot be fixed in a short
> manner, during our work week beginning of December. There are also meeting
> notes.
> 
We should probably disable all tests which are failing on the branches then.  Most, if not all, existing failures have been occurring for at least a month.

> Anthony, you are assigned to this bug, so please upload a quick patch which
> will skip this test on all platforms. I'm happy to review and land it if
> necessary.
How does one "disable" a test?

Comment 19

[Aaron Train [:aaronmt]]

> > Anthony, you are assigned to this bug, so please upload a quick patch which
> > will skip this test on all platforms. I'm happy to review and land it if
> > necessary.
> How does one "disable" a test?

It's a couple of meta tags that go at the bottom of a test, one for the setupModule and one for the teardownModule:

Example:

setupModule.__force_skip__ = "Bug 609058: Timeout failure in testInstallExtension";
teardownModule.__force_skip__ = true;

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So this test disconnects in the last couple of days because the 'Install' button in the software installation dialog of the ACR extension doesn't get clicked. The dialog stays open. Strange, that should not happen since the modal dialog fixes have been landed.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Patch v1

Ok, the problem on AMO has been fixed and now the extension we are looking for are visible. But now the new issue with the modal dialog has been raised. And that's a simple mistake by calling the constructor of the modal dialog with the controller and not the window.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Landed as:
http://hg.mozilla.org/qa/mozmill-tests/rev/62e5c8885541 (1.9.2)
http://hg.mozilla.org/qa/mozmill-tests/rev/3e1806da0687 (1.9.1)