Test Suite for Addons.mozilla.org

RESOLVED FIXED

Status

Testing
WebQA
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: retornam, Assigned: mozwebqa)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 4 obsolete attachments)

(Reporter)

Description

8 years ago
I have attached the new format for test suites for addons.mozilla.org to this bug for review.
(Reporter)

Comment 1

8 years ago
Created attachment 438491 [details]
AMOConfig

this file contains all the configuration options that were in browserConfig
(Reporter)

Comment 2

8 years ago
Created attachment 438492 [details]
AMO_theme

AMO theme testcase in the new format
(Reporter)

Comment 3

8 years ago
Created attachment 438493 [details]
new testsuite

test suite in the new format

Updated

8 years ago
Assignee: nobody → mozwebqa
(In reply to comment #1)
> Created an attachment (id=438491) [details]
> AMOConfig
> 
> this file contains all the configuration options that were in browserConfig

Propose take the renaming another step further and use some variation of testCaseParameters or TCparams.

Suggest the list of browsers comes from gridConnection similar to the other selenium.start parameters.  This allows a gridConnection module on the local client to drive the list rather than editing config/param module.

The config/parameters module has a bunch of variables at the bottom ending with 'index'.  Are there plans for them?
(In reply to comment #2)
> Created an attachment (id=438492) [details]
> AMO_theme
> 
> AMO theme testcase in the new format

To everyone: The sample test case and suite proposes that looping thru the options (ex: sort, application) is handled by the test case instead of the suite.  Awhile ago we decided that simpler tests cases were better and a test case should focus on one thing, such as one sort, one login (or not logged in), one application, etc.  The sample moves away from that, if we feel that is still important.
another comment on the config/parameter module: it has a dictionary for some TC parameters, while the module already serves as a container for almost all TC parameters.  At first glance, the dictionary seems to add another layer without adding any benefits.
(Reporter)

Comment 7

8 years ago
(In reply to comment #5)
> (In reply to comment #2)
> > Created an attachment (id=438492) [details] [details]
> > AMO_theme
> > 
> > AMO theme testcase in the new format
> 
> To everyone: The sample test case and suite proposes that looping thru the
> options (ex: sort, application) is handled by the test case instead of the
> suite.  Awhile ago we decided that simpler tests cases were better and a test
> case should focus on one thing, such as one sort, one login (or not logged in),
> one application, etc.  The sample moves away from that, if we feel that is
> still important.


I think it is best to handle looping through the test case itself rather than in the suite. This method makes  suites simple and remove a lot of programming logic from the test suite itself. I left the indexes untouched because I have not rewritten all the test cases using the new format. I prefer we leave them as they are until we are sure the new format satisfies every old test case we have
(Reporter)

Comment 8

8 years ago
(In reply to comment #6)
> another comment on the config/parameter module: it has a dictionary for some TC
> parameters, while the module already serves as a container for almost all TC
> parameters.  At first glance, the dictionary seems to add another layer without
> adding any benefits.
 


Which module are u talking about here. I feel i'd understand your comments well if you describe specific files / test cases
(In reply to comment #8)
> (In reply to comment #6)
> > another comment on the config/parameter module: it has a dictionary for some TC
> > parameters, while the module already serves as a container for almost all TC
> > parameters.  At first glance, the dictionary seems to add another layer without
> > adding any benefits.
> 
> 
> 
> Which module are u talking about here. I feel i'd understand your comments well
> if you describe specific files / test cases

talking about the browserConfig/AMOconfig/TCparams module
(Reporter)

Comment 10

8 years ago
We dont have any module / file called TCparams please either explain to me personally  as I still do not understand the point you are making. Thanks
(In reply to comment #10)
> We dont have any module / file called TCparams please either explain to me
> personally  as I still do not understand the point you are making. Thanks

browserConfig/AMOconfig/TCparams is referring the old browserConfig module which will be renamed in the near future.  sorry for the confusion.
Attachment #438491 - Attachment mime type: text/x-python-script → text/plain
Attachment #438492 - Attachment mime type: text/x-python-script → text/plain
Attachment #438493 - Attachment mime type: text/x-python-script → text/plain
Created attachment 438636 [details]
throwing it down

Here's a base class and test loader that will automatically run tests against all the browsers you want.  Example usage and docs are inline.

It's not ready to go right now, but it should be easy to fix BrowserTestCase to do what you guys want.

Also, please print out a copy of http://www.python.org/dev/peps/pep-0008/ and put it under your pillow after you read it every night.
Attachment #438493 - Attachment is obsolete: true
Attachment #438636 - Flags: feedback?(vish.mozilla)
Attachment #438636 - Flags: feedback?(tsmith)
Attachment #438636 - Flags: feedback?(stephen.donner)
Attachment #438636 - Flags: feedback?(mozbugs.retornam)
Attachment #438636 - Flags: feedback?(krupa.raj82)

Updated

8 years ago
Attachment #438636 - Attachment mime type: text/x-python-script → text/plain
vish_moz: Jeff's example is completely diff from the two previous versions and a lot better too
[12:00pm] vish_moz: he's using a Base class(Parent class) which defines all browsers
[12:01pm] vish_moz: and then if we just want to run our test in a particular browser if not all then we will override the browsers[] array in our child class (test case)
[12:01pm] vish_moz: brilliant idea!!
(Reporter)

Comment 14

8 years ago
Created attachment 438850 [details]
Testcases with various options

I  made edits based on Jeff's recommendations  and I'm submitting them  for Jeff to review.  Thanks for your help Jeff
(Reporter)

Updated

8 years ago
Attachment #438850 - Attachment mime type: text/x-python-script → text/plain
(Reporter)

Updated

8 years ago
Attachment #438636 - Flags: feedback?(mozbugs.retornam) → feedback+
(Reporter)

Updated

8 years ago
Attachment #438850 - Flags: review-
Attachment #438850 - Flags: feedback?(jbalogh)
(Reporter)

Updated

8 years ago
Attachment #438850 - Flags: review- → review?
Created attachment 438925 [details]
With config selection from the command line
Attachment #438636 - Attachment is obsolete: true
Attachment #438850 - Attachment is obsolete: true
Attachment #438925 - Flags: feedback?(vish.mozilla)
Attachment #438925 - Flags: feedback?(tsmith)
Attachment #438925 - Flags: feedback?(stephen.donner)
Attachment #438925 - Flags: feedback?(mozbugs.retornam)
Attachment #438925 - Flags: feedback?(krupa.raj82)
Attachment #438636 - Flags: feedback?(vish.mozilla)
Attachment #438636 - Flags: feedback?(tsmith)
Attachment #438636 - Flags: feedback?(stephen.donner)
Attachment #438636 - Flags: feedback?(krupa.raj82)
Attachment #438850 - Flags: review?
Attachment #438850 - Flags: feedback?(jbalogh)
Created attachment 445972 [details]
limitless variety

Here's a test loader that creates test case variations on all the lists in TestCase.vary.
Attachment #438925 - Attachment is obsolete: true
Attachment #438925 - Flags: feedback?(vish.mozilla)
Attachment #438925 - Flags: feedback?(tsmith)
Attachment #438925 - Flags: feedback?(stephen.donner)
Attachment #438925 - Flags: feedback?(mozbugs.retornam)
Attachment #438925 - Flags: feedback?(krupa.raj82)
Raymond/Truman: is there anything left to do, here?
One of the attachments contains interesting code/function ideas from Jeff at the time we were discussing the assertion-centric design.  It is no longer an active discussion as far as I know.
Closing, then; thanks!
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.