Closed Bug 1058445 Opened 10 years ago Closed 10 years ago

Switch to addons.allizom.org for add-on related remote tests

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

defect

Tracking

(firefox31 wontfix, firefox32 fixed, firefox33 fixed, firefox34 fixed, firefox-esr24 fixed, firefox-esr31 fixed)

RESOLVED FIXED
Tracking Status
firefox31 --- wontfix
firefox32 --- fixed
firefox33 --- fixed
firefox34 --- fixed
firefox-esr24 --- fixed
firefox-esr31 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

(Whiteboard: [mozmill-test-failure])

Attachments

(2 files)

Bug 899923 has shown that an unplanned outage of the addons-dev server can cause lots of test failures for add-on related Mozmill tests. This concerns me a lot especially when it comes to testing Firefox Beta and Release builds. A dev system totally unrelated to the Firefox product itself should absolutely not harm us that hard.

So I propose that we change all the references to the AMO dev server to the production server. This will give way more stability. One thing I could consider is using an AMO staging server if such thing exists. Stephen or Krupa, could you give your feedback on this please?

I know that WebQA wants the Mozmill test to be run against a pre-release of AMO, but e.g. the above implications make our tests unstable. So for testing against the dev server we might want to create a different testrun, which is not run for beta and release.
Flags: needinfo?(stephen.donner)
Flags: needinfo?(krupa.mozbugs)
Whiteboard: [mozmill-test-failure]
My only concern is that this will affect our user analytics data. How often do we run mozmill tests?
Flags: needinfo?(krupa.mozbugs)
(In reply to krupa raj[:krupa] from comment #1)
> My only concern is that this will affect our user analytics data. How often
> do we run mozmill tests?

A gross estimate would be ~2800 testruns per week (850 daily + 2000 for beta / release).
Where each testrun does hit AMO multiple times.

Could we not blacklist the IP range to not be taken into consideration for the analytics data?
Krupa, you haven't replied to my other question yet, if there is a staging server available. Or is amo-dev the staging server?
(In reply to Henrik Skupin (:whimboo) from comment #3)
> Krupa, you haven't replied to my other question yet, if there is a staging
> server available. Or is amo-dev the staging server?

As we discussed before the QA meeting, there is https://addons.allizom.org - Krupa can better-answer whether it's more stable/has the DB content you might need, Henrik.
Flags: needinfo?(stephen.donner)
Just to add... Reason why we had so many failures with add-on related tests against addons-dev might be that this instance gets updates each 15min, which also includes reloads of Apache. So sessions might be lost, or whatever else could happen.

I assume that add-ons.allizom.org is not getting updated at the same time when we have Firefox releases?
Flags: needinfo?(krupa.mozbugs)
https://addons.allizom.org is probably more reliable than dev and i'm fine with using it.
Flags: needinfo?(krupa.mozbugs)
Wonderful. So lets make this change happen!
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Summary: Get rid of using http://addons-dev.allizom.org/ in add-on related tests → Switch to addons.allizom.org for add-on related remote tests
Attached patch amo-staging v1Splinter Review
This patch changes the AMO domain we make use of. Further it removes the AMO site variable given that is nowhere used.
Attachment #8480178 - Flags: review?(andrei.eftimie)
fyi, stage has code updates happen on wed, thur and friday.
Krupa, are there specific times set for those updates? Also how long do those usually take, involving downtime of the service?
Comment on attachment 8480178 [details] [diff] [review]
amo-staging v1

Review of attachment 8480178 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.
All tests are passing.

::: lib/addons.js
@@ +28,5 @@
>  // AMO Preferences
>  const AMO_DISCOVER_URL = 'extensions.webservice.discoverURL';
>  
> +// AMO instance to use
> +const AMO_DOMAIN = "addons.allizom.org";

I do wonder where we used AMO_PREVIEW_SITE. I remember it being necessary at some point, but now opening "addons.allizom.org" directly does get redirected to a workable URL, so we're good.
Attachment #8480178 - Flags: review?(andrei.eftimie) → review+
(In reply to Andrei Eftimie from comment #11)
> I do wonder where we used AMO_PREVIEW_SITE. I remember it being necessary at
> some point, but now opening "addons.allizom.org" directly does get
> redirected to a workable URL, so we're good.

I think we only added that to have the full URL and wouldn't have to add the protocol each time.

Landed on mozilla-central:
https://hg.mozilla.org/qa/mozmill-tests/rev/f96b13c4b55d

Lets observe todays tests and if all is fine lets get it backported asap.
Blocks: 1059730
Andrei tested my patch on beta and release today. All is fine on those branches. I did a quick spot-check for aurora and no failures are visible. So lets get the patch backported to all other branches.

https://hg.mozilla.org/qa/mozmill-tests/rev/ef952fbbdb41 (aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/00cf0f9ce9d1 (beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/1364a442facb (release)

I will have to run some spotchecks for esr branches. So an update will follow soon.

Btw. this will also drastically improve our stability for restart tests, which is a goal this quarter and tracked as bug 1035187.
Applies cleanly on esr31 and all tests pass:
https://hg.mozilla.org/qa/mozmill-tests/rev/97ba5f074ba2 (esr31)

Sadly there are merge conflicts for esr24. I will check shortly.
This should be the last ESR24 build, so lets not push this to esr24.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
No, there is still the chance for a chemspill. And seeing amo-dev down again would be a mess. This bug is important enough to get backported to esr24.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch amo_backport v1Splinter Review
Backport patch for esr24.
Attachment #8480550 - Flags: review?(andrei.eftimie)
Attachment #8480178 - Flags: checkin+
Comment on attachment 8480550 [details] [diff] [review]
amo_backport v1

Review of attachment 8480550 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm
Attachment #8480550 - Flags: review?(andrei.eftimie) → review+
https://hg.mozilla.org/qa/mozmill-tests/rev/488cc02b7a85 (esr24)
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: