849780 - Some remote tests are failing due to brokenness of the addons-dev.allizom.org site

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Andreea Matei [:AndreeaMatei]]

Happened today with Nightly, on a Linux x64 machine.

We might need an assert.waitFor() to get the results loaded.

Comment 1

[Daniela Petrovici]

I was able to reproduce this issue locally 3 times out of 200 runs on Beta version with Windows XP while working on bug 732353.

Comment 2

[Andreea Matei [:AndreeaMatei]]

Reproduced on OS X 10.7.5 as well, with Nightly fr locale:
http://mozmill-ci.blargon7.com/#/remote/report/2a6536e9db9f5f44ed48c585101ec8d4

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Might also be a change in Firefox 22. We have to get this investigated and fixed as early as possible. Who can work on it?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This seems to be happening on m-c for every platform now.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Same for 22.0a2:
http://mozmill-ci.blargon7.com/#/remote/report/2a6536e9db9f5f44ed48c58510214e62

It might be that AMO is not working properly and we regressed there?

Comment 6

[Andreea Matei [:AndreeaMatei]]

I believe so, yes, it stopped failing last night, today I see it's passing. From WebQA team I know addons-dev.allizom.org was up and down in the last days.
I'm waiting for today's testruns to be sure it's safe to close.

Comment 7

[Andreea Matei [:AndreeaMatei]]

I feel we should change the requests from addons-dev.allizom.org to addons.mozilla.org since the last one is more stable, in addons.js shared module.
Henrik, what do you think?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andreea Matei [:AndreeaMatei] from comment #7)
> I feel we should change the requests from addons-dev.allizom.org to
> addons.mozilla.org since the last one is more stable, in addons.js shared
> module.

The original intend of of using the dev server was to spot regressions (like installing add-ons) before they are live on production. So I'm not a huge fan of going back to the production server. But I would like to hear Dave's and Stephens opinions.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

> The original intend of of using the dev server was to spot regressions (like installing add-ons) before they are live on production.

That sounds like a specific set of tests to me. We want our Mozmill tests to be reliable, but perhaps the AMO team should be able to run these tests against staging, or we should have another testrun?

Comment 10

[Daniela Petrovici]

Happened again on Windows 7 x86 and x64 and Windows 8 x64 with Nightly FR build:
http://mozmill-ci.blargon7.com/#/remote/report/cb9e33213cf3d2f37d7b0f8dcc1d48d2
http://mozmill-ci.blargon7.com/#/remote/report/cb9e33213cf3d2f37d7b0f8dcc1d3ebc
http://mozmill-ci.blargon7.com/#/remote/report/cb9e33213cf3d2f37d7b0f8dcc1d4597

Comment 11

[Daniela Petrovici]

Started happening on all Windows machines with localized builds and also Ubuntu with Aurora:
http://mozmill-ci.blargon7.com/#/remote/report/cb9e33213cf3d2f37d7b0f8dcc1dd8f5

Comment 12

[Daniela Petrovici]

Attachment: skip patch for Nightly version

Started happening a lot on Nightly version today (about 15 times already) so added skip patch.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume it's the same issue as the last time which was bug 849794. Stephen, are you aware of an outtage of AMO dev today? Did they run an update? If yes we might have to find a better time to push those updates to the server.

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Henrik Skupin (:whimboo) from comment #13)
> I assume it's the same issue as the last time which was bug 849794. Stephen,
> are you aware of an outtage of AMO dev today? Did they run an update? If yes
> we might have to find a better time to push those updates to the server.

Yes, they refreshed/reindexed the DB on dev.

Comment 15

[Hanno Schlichting]

Addon search was unintentionally unavailable for some time on the dev server yesterday. We are going to improve the particular code that caused this, so there won't be any more unavailability for this specific reason.

But addons-dev.allizom.org is still going to be unavailable or broken once in a while. Unfortunately the stage servers at addons.allizom.org aren't really any better, as those aren't used much to at all (and seem to be broken right now).

So you have to make a choice between stability and testing against production and occasional instability and testing against dev. Both have their ups and downs.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Hanno Schlichting [:hannosch] from comment #15)
> Addon search was unintentionally unavailable for some time on the dev server
> yesterday. We are going to improve the particular code that caused this, so
> there won't be any more unavailability for this specific reason.
> 
> But addons-dev.allizom.org is still going to be unavailable or broken once
> in a while. Unfortunately the stage servers at addons.allizom.org aren't
> really any better, as those aren't used much to at all (and seem to be
> broken right now).
> 
> So you have to make a choice between stability and testing against
> production and occasional instability and testing against dev. Both have
> their ups and downs.

Thanks Hanno. So I don't like broken tests on a day when we have a new release candidate of Firefox to test. This will not only cause us trouble but also the QA team given their signoff criteria. I'm really indifferent in what we should do. I can see the benefit testing staging code so that we can catch regressions before they are pushed to production. But due to the different fallout scenarios lately I would better move of to production. Well, unless the timing of updates on staging can be improved and will not conflict with beta/release testing in the future. 

Another option could be that we only make use of the staging server for Aurora and Nightly builds but not for beta and releases. Not sure how easy it would be to implement that, and we also might miss possible regressions.

Krupa, what do you think? Is it valuable to keep testing the staging instance? What can we do to improve the situation? Thanks.

Comment 17

[krupa raj[:krupa]]

I still see a huge value in testing on addons-dev.allizom.org to catch regressions earlier. 

Unless these (once in a while) false alarms are excessively bothersome, I think we should keep the tests running against -dev.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to krupa raj 82[:krupa] from comment #17)
> Unless these (once in a while) false alarms are excessively bothersome, I
> think we should keep the tests running against -dev.

Could we then work on a strategy when the server should not be put down or getting updated? As said, especially on days when release and beta builds are produced this is really annoying.

Comment 19

[Hanno Schlichting]

Some background on what caused it this time:

A developer triggered a maintenance action on the marketplace-dev instance, which should have had no visible impact and do some background processing on marketplace apps (syncing the primary MySQL database and ElasticSearch search index).

The code in question did something else though and dropped the entire ElasticSearch index for both marketplace-dev and addons-dev and rebuild them from scratch. This process took more than an hour and had a very visible impact on search.

I think this sort of mistake is about as likely as a failed push to the stage or production instances. But I'm "new" here and don't know all the back story.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Hanno for letting us know about all the details of this issue.  In that case and the hope it was a one-off we will keep what we currently have and will continue to watch how stable we are over time.

Comment 21

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 735223 [details] [diff] [review]
skip patch for Nightly version

Review of attachment 735223 [details] [diff] [review]:
-----------------------------------------------------------------

Forgot to remove the request, no need for the skip patch at the moment. Thanks Daniela.

Comment 22

[mario garbi]

We have another couple of failures on Linux 12.04 and 12.10 (x86 and x86_64) with Firefox 23.0a1 de and en-US:

http://mozmill-ci.blargon7.com/#/remote/report/ea82256a8ae9808d91b7e8145e7dbd5b
http://mozmill-ci.blargon7.com/#/remote/report/ea82256a8ae9808d91b7e8145e7dd277
http://mozmill-ci.blargon7.com/#/remote/report/ea82256a8ae9808d91b7e8145e7dd53b
http://mozmill-ci.blargon7.com/#/remote/report/ea82256a8ae9808d91b7e8145e7f2d03

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please stop from reopening this bug, at least if it happens only for an hour or so. This is known and might happen all the time when the staging site for AMO gets an update. Only when it happens for tests longer then half a day, reopen the bug. Thanks.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We have again massive failures for this problem again when trying to search for RSS. I really don't want that our own system is getting blown up all the time something happens on the staging server of AMO. So lets do the flip and change the remote server to the public AMO server.

Failures like those kill our own capability to sign off for release testing. So as long as those outages will continue to happen I don't think we should go back to the staging site. But we might want to find out if we can use it for daily builds only.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Wil, can you please give us your feedback on this devastating situation with our tests? What's going on with the staging server of AMO in the last days? Nearly all of our test runs which make use of the add-ons manager search feature reporting failures.

If this does not stop soon, our decision to move to the production server of AMO will manifest. We will not be able to test pre-releases of AMO anymore but we would have to live with that for the time being.

Comment 26

[mario garbi]

Attachment: patch V1

A patch to switch from the preview dev.allizom site to the production AMO site in case we decide to use this approach.

Linux:
http://mozmill-crowd.blargon7.com/#/remote/report/14f8bc4e22e61353662cded4c29f15c9

WinXP:
http://mozmill-crowd.blargon7.com/#/remote/report/14f8bc4e22e61353662cded4c29f4f62

Mac:
http://mozmill-crowd.blargon7.com/#/remote/report/14f8bc4e22e61353662cded4c2a02871

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 751021 [details] [diff] [review]
patch V1

Review of attachment 751021 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/addons.js
@@ +26,5 @@
>  const AMO_PREVIEW_DOMAIN = "addons-dev.allizom.org";
>  const AMO_PREVIEW_SITE = "https://" + AMO_PREVIEW_DOMAIN;
>  
> +// AMO Production site
> +const AMO_PRODUCTION_DOMAIN =  "addons.mozilla.org";

We don't need the preview and production constants. Make it a single one like AMO_DOMAIN. That way we could hot-flip if we want to go back and forth.

::: tests/remote/testAddons/testSearchAddons.js
@@ -17,5 @@
>    am = new addons.AddonsManager(controller);
>    locationBar = new toolbars.locationBar(controller);
>    tabBrowser = new tabs.tabBrowser(controller);
>  
> -  addons.useAmoPreviewUrls();

Keep those in and rename the method to e.g. updateAMOUrls()

Comment 28

[mario garbi]

Attachment: Patch v1.2

Made the requested changes.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A lot of our Mozmill reports are also passing. Can we be sure that this is a failure only on the remote site? Mario, have you checked Firefox manually if it reproduces? If not, please do this ASAP. This should be the most important task for you at the moment. Thanks.

Comment 30

[mario garbi]

Attachment: Screenshot using dev.allizom.org

Attached is the screenshot proof that dev.allizom.org doesn't return search results for "rss" anymore, I am uncertain about why this is happening but the production AMO seems to work ok.

Steps to reproduce manually:
1. Open about:config
2. Change the values for the "extensions.getAddons.*" as seen in addons.js
3. Open Addons Manager
4. Search for "rss"

Expected results:
-A couple of results

Actual results:
-After waiting for a while we get the message "Could not find any matching addons" as seen in the screenshot.

Comment 31

[mario garbi]

I've used Linux 12.04 with Firefox Aurora 23.0a2 en-US (latest version) for the manual testing.

Comment 32

[mario garbi]

We get the same results for Mac 10.6.8 and Win8(x32) with Firefox Nightly and Release. It happens randomly every few searches and also for any search term not just "rss". 
I suspected it to be a connection issue so I've pinged "addons-dev.allizom.org" while manually testing Addons Manager and it seems we have a steady 189ms connection with no Timeouts.

Comment 33

[mario garbi]

Attachment: Screenshot of the addons-dev.allizom.org webpage

We seem to hit this search issue on addons-dev.allizom.org webpage as well not just while using Addons Manager.

Comment 34

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so lets get a bug filed for AMO. Put the used link into the url field for reproduction steps. We should get this fixed quickly.

Comment 35

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A problem in the elastic search cluster was the issue here. It has been brought back and for now it should work. Lets see if the failures have been stopped now.

Comment 36

[mario garbi]

I haven't seen the search issue in about 50 runs, but now we seem to get a few new problems with page loading and category select. I will look into this since it's similar to Bug 809778.

Comment 37

[mario garbi]

It seems that the issues I talked about in comment 36 stopped happening as I can't see those fails anymore.

Comment 38

[mario garbi]

Green reports on all branches, Linux, Win8 and Mac without any patch:
http://mozmill-crowd.blargon7.com/#/remote/reports?branch=All&platform=All&from=2013-05-22&to=2013-05-22

Comment 39

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 751669 [details] [diff] [review]
Patch v1.2

Review of attachment 751669 [details] [diff] [review]:
-----------------------------------------------------------------

We no longer need to change this as it has been fixed.

Comment 40

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Alright. That sounds great! Lets convert this bug into a meta bug then, given that we know that brokenness on the addons-dev.allizom.org site can cause those problems. In that case it was good that we kept testing this site and not the production one.

Whenever something breaks in the future we should file a new bug and add it as dependency to this tracking bug. That way we can keep track of what we have found with our remote tests.