Closed Bug 705118 Opened 8 years ago Closed 8 years ago

Mozmill endurance test for open and close the extensions list in the add-ons manager

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect)

x86
Linux
defect
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: vladmaniac, Assigned: vladmaniac)

References

Details

(Whiteboard: [mozmill-endurance][mozmill-aom])

Attachments

(2 files, 2 obsolete files)

Create a mozmill endurance test for open and close the extensions list in the add-ons manager.
Whiteboard: [mozmill-endurance]
A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/21443335
Assignee: nobody → vlad.mozbugs
Status: NEW → ASSIGNED
(In reply to Maniac Vlad Florin (:vladmaniac) from comment #1)
> A Pivotal Tracker story has been created for this Bug:
> https://www.pivotaltracker.com/story/show/21443335

Well I don't know for sure if we let go of Pivotal Tracker - so far I'm going to further create stories in it until something official comes around.
You can retire using Pivotal. I'll be posting something shortly which makes the official announcement.
Whiteboard: [mozmill-endurance] → [mozmill-endurance][mozmill-aom]
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #3)
> You can retire using Pivotal. I'll be posting something shortly which makes
> the official announcement.

Agreed. Thanks for clarifications, Anthony
Blocks: 629065
The following test will use the given approach: 

    Main Test Checkpoints
      1.Addon manager opened
      2.Extensions list loaded
      3.Addon manager closed

    Teardown Module
      1.Force close addon manager
No longer blocks: 629065
Blocks: 629065
Attached patch patch v1.0 all branches (obsolete) — Splinter Review
Initial patch
Attachment #576923 - Flags: review?(anthony.s.hughes)
Attachment #576923 - Flags: feedback?(dave.hunt)
I know it's not common to ask both for feedback and review at the same time but considering that davehunt is the owner of endurance tests and he's in my timestamp, I would want his input on my patches. 

And it makes time economy considering the endurance tests code sprint that i want to do
Attached patch patch v1.1 all branches (obsolete) — Splinter Review
I think this is what you had in mind Dave, is it?
Attachment #576923 - Attachment is obsolete: true
Attachment #576956 - Flags: feedback?(dave.hunt)
Attachment #576923 - Flags: review?(anthony.s.hughes)
Attachment #576923 - Flags: feedback?(dave.hunt)
Comment on attachment 576956 [details] [diff] [review]
patch v1.1 all branches

Looks good. We should probably rename testAddons_OpenAndCloseAddonManager to something lik testAddons_OpenAndCloseGetAddons to make a clear difference between these two tests.
Attachment #576956 - Flags: feedback?(dave.hunt) → feedback+
We are pretty for review
Attachment #576956 - Attachment is obsolete: true
Attachment #576958 - Flags: review?(anthony.s.hughes)
(In reply to Dave Hunt (:davehunt) from comment #10)
> Comment on attachment 576956 [details] [diff] [review] [diff] [details] [review]
> patch v1.1 all branches
> 
> Looks good. We should probably rename testAddons_OpenAndCloseAddonManager to
> something lik testAddons_OpenAndCloseGetAddons to make a clear difference
> between these two tests.

I can't do it here. we apply clean patches for tests only - exception might me slightly API changes with the test. 

It will need another bug for that change, but I'll take care of it and ask you for review. 
Thanks Dave!
Dave, I just noticed that we already have an endurance test extremely similar to this:
http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/endurance/testAddons_OpenAndCloseAddonManager/test1.js

Can you verify this new test is needed?
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #13)
> Dave, I just noticed that we already have an endurance test extremely
> similar to this:
> http://hg.mozilla.org/qa/mozmill-tests/file/default/tests/endurance/
> testAddons_OpenAndCloseAddonManager/test1.js
> 
> Can you verify this new test is needed?

IMO: 

We can test this because testAddons_OpenAndCloseAddonManager/test1.js opens a different type of content (remote content - disc pane), and our current test opens the extension list. 

However, we don't have to test for 'Appearance' or 'Plugins' category since those refer also to a list the same as the 'Extension' category does. 

If we think this way, we can make usage of it. Of its intended to gather information in the same manner as testAddons_OpenAndCloseAddonManager/test1.js then we are definitely duplicating our work
The test was a suggestion for discussion - I assumed it had been discussed and considered appropriate. I agree with the similarity with the existing test, and maybe it's not a valuable enough test on it's own.

Can we arrange a time to discuss the remaining endurance test suggestions? Most of them were thought up in a moment and after further discussion we may find them to be inappropriate, and come up with some other ideas. I have already tried starting this discussion on the mailing list [1].

[1] http://groups.google.com/group/mozilla.dev.automation/browse_thread/thread/f32e902dc35fbfe1#
based on Comment 15 is this a wontfix?
Let me simplify the clarification:

* testAddons_OpenAndCloseAddonsManager tests opening to remote AMO content
* testAddons_OpenAndCloseExtensionsList tests opening to the local list of installed extensions

If this is accurate, this test is unique enough to develop in my opinion.

Dave, I think that all the previously mentioned tests are worth developing at this time. We should definitely have a discussion about where to focus for Q1'2012, but I don't think that discussion negates any current development activity.

Vlad, in my opinion, you can continue developing this test.
Sounds good to me. As previously mentioned we should open another bug to rename the existing test to distinguish it from this one.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #17)
> Let me simplify the clarification:
> 
> * testAddons_OpenAndCloseAddonsManager tests opening to remote AMO content
> * testAddons_OpenAndCloseExtensionsList tests opening to the local list of
> installed extensions
> 
> If this is accurate, this test is unique enough to develop in my opinion.
> 
> Dave, I think that all the previously mentioned tests are worth developing
> at this time. We should definitely have a discussion about where to focus
> for Q1'2012, but I don't think that discussion negates any current
> development activity.
> 
> Vlad, in my opinion, you can continue developing this test.

I do, the patch is waiting for your review Anthony, thanks a lot
Depends on: 706814
Please see bug 637286 why this is an important test, at least with a lot of extensions installed.
Comment on attachment 576958 [details] [diff] [review]
patch v1.2 [landed]

This is fine as a reference test for default add-ons, but where the real value lies (as Henrik has already pointed out) is with multiple add-ons already installed. 

I think this test is fine to check in as is, but we should fork a new test which installs ENTITY number of add-ons before testing.

Also, the dependency bug needs to be resolved before we can check this in.
Attachment #576958 - Flags: review?(anthony.s.hughes) → review+
Dave, do you have thoughts on my proposal in comment 21?
Keywords: checkin-needed
Additional add-ons can be installed for all our testrun scripts using the command line. I think I would prefer to see the endurance tests being run with additional add-ons installed than installing them within the tests themselves.

I'd be keen to hear Henrik's thoughts.
I would still prefer to install random add-ons from remote sites like our ftp server. We could randomly select those. Specifying those on the command line is not that dynamic as you probably want to be, Dave.
Which add-ons are available from our FTP server? I'm not sure about having randomness within our daily endurance testrun as this will cause fluctuation in the metrics gathered, making it difficult to know if a spike in memory usage (for example) is related to a Firefox regression or the add-on that happened to be expected. I see the testruns with add-ons being more suitable for Mozmill Crowd and exploratory test automation.
All add-ons which are hosted on AMO can be downloaded via the FTP server. Here you have the link: http://ftp.mozilla.org/pub/mozilla.org/addons

I agree with your statements re: difficulty in measuring and comparing the memory usage this way. So I'm also open for a number of fixed add-ons to being installed.
I think the discussion so far has been great but is distracting from getting us to check in this test. As stated in comment 21, I think the patch is fine as a baseline, default add-ons test.

Vlad, if no one objects, I will check this test in when I get back from holidays on Dec 27th. In the meantime, we will take the remaining discussion to an email thread and can discuss in the next automation meeting in the new year.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #27)
> I think the discussion so far has been great but is distracting from getting
> us to check in this test. As stated in comment 21, I think the patch is fine
> as a baseline, default add-ons test.
> 
> Vlad, if no one objects, I will check this test in when I get back from
> holidays on Dec 27th. In the meantime, we will take the remaining discussion
> to an email thread and can discuss in the next automation meeting in the new
> year.

Can we still check this in? Or is there any more work to be done ?
Sorry, this fell off my radar. I don't see any objections to this test as a baseline. I'll go ahead and check it in on default to start.
Comment on attachment 576958 [details] [diff] [review]
patch v1.2 [landed]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/02732c693671 (default)
Attachment #576958 - Attachment description: patch v1.2 all branches → patch v1.2 all [checked-in: default]
Vlad, please check the next endurance results to see if this passes. If so, I will land on other branches.

NOTE: With brasstacks outage you may not be able to verify until later in the week.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Vlad, you missed verification of this bug and as a result we missed landing it on other branches. Please verify your patch works for Release so we can land it.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #32)
> Vlad, you missed verification of this bug and as a result we missed landing
> it on other branches. Please verify your patch works for Release so we can
> land it.

Note that you will have to update the patch to use the new MPL2 license block. Also, this does not need to land on aurora or beta since it was uplifted in the last two merges.

Please also do whatever you have to to ensure this does not happen again.
> 
> NOTE: With brasstacks outage you may not be able to verify until later in
> the week.

Yup. This does not happen usually so I found the reason it did. It was in the time when brasstacks were down 

Sorry, I'll update the patch asap
Adding patch for release branch and esr branch 

Test is not failing on default, aurora or beta. 
Branches default, mozilla-aurora, mozilla-beta are in order, so need to update release and esr10 branches
Attachment #611755 - Flags: review?(anthony.s.hughes)
Comment on attachment 611755 [details] [diff] [review]
patch v1.2 (release) [landed]

Patch looks good. Note that I will not be landing this on esr because we don't land new code there.
Attachment #611755 - Flags: review?(anthony.s.hughes) → review+
Comment on attachment 611755 [details] [diff] [review]
patch v1.2 (release) [landed]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/434d0d94d540 (mozilla-release)
Attachment #611755 - Attachment description: patch v1.2 for release branch and esr → patch v1.2 (release) [landed]
Attachment #576958 - Attachment description: patch v1.2 all [checked-in: default] → patch v1.2 [landed]
Status: RESOLVED → VERIFIED
Keywords: checkin-needed
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.