580807 - Multiprocessing / Category-Focused Test Runner for AMO and SUMO

Status: RESOLVED INVALID

(Testing Graveyard :: WebQA, defect)

Description

[Tanay Gavankar [:tgavankar]]

Attachment: AMO driver

I have attached the source codes for the new test runners/drivers for both AMO and SUMO. 

They slightly differ because AMO has TCparams and SUMO does not, etc. However, the driver itself should not have to be modified at all - all the config options are in tests.py (one in each repository, so one for AMO and one for SUMO). As always, you can modify TCparams in tests.py for AMO, and the browser lists/etc are still in GridConnection.

SUMO has the browsers listed in vars.py. 


The drivers have been written in such a way that there should be no changes required in the test cases themselves. 

One thing to note that applies to SUMO tests is that the setUp() methods of every test case are overridden by the driver. All the tests case at the moment have the same (or very similar) setUp() methods, so modeling future test cases after them will ensure that they function as desired.


I have tried to comment the drivers to the best of my ability; hopefully they are not too complicated to follow. 


Features:
* Multiprocessing/multithreading
- This driver uses multithreading/multiprocessing to shorten test run times. It runs all the different environments for a test type in parallel, so instead of the entire suite taking the same amount of time as running the suite in each browser individually summed, it will take only as long as the slowest environment. 

* Keyword/category tagging
- Tests can be tagged keywords in addition to placing them in smoketest/bft/fft. There is no limit to the number of keywords a test can have, nor the total number of different keywords. When running the suite, you simply pass whatever (and how many ever) keywords you want as command line arguments and the suite will automatically run all the tests tagged with any of those keywords. The current plan is to set up multiple hudson builds with the combinations / different key words so that the user does not have to go in and modify the hudson job to get a specific category build. Of course, the option to run a custom mix of keywords will still be available by modifying the hudson job configuration. 

* Command line test level declaration
- Rather than having separate suite files for smoketests, bfts, and ffts, the user adds the tests to the appropriate granularity in the tests.py file, and then passes either "smoketests", "bft", or "fft" somewhere in the command line parameters. This will cause the tests to run the browser environments identified with the test type, and if no other keywords are present, will run all the tests listed in that test type. If other test keywords are listed along with a test type, The tests that are tagged with the keywords will be run at the granularity of the desired test type. If multiple test types are passed as arguments, only the first one will be used. If no test type is given, the driver will default to FFT options.


Example:
In Hudson, giving the following shell command will run all tests tagged with "search" OR "layout" in the environments defined for a BFT and the options of a BFT as well (in regards to TCparams):

python amo_suite.py bft search layout

Note that the keywords and test type can be in any order, so "python amo_suite.py search bft layout" and "python amo_suite.py layout search bft" would have the same results.

Comment 1

[Tanay Gavankar [:tgavankar]]

Attachment: AMO tests.py

Note that the tagging in both the tests.py files have yet to be done.

Comment 2

[Tanay Gavankar [:tgavankar]]

Attachment: SUMO driver

Comment 3

[Tanay Gavankar [:tgavankar]]

Attachment: SUMO tests.py

Comment 4

[Vishal Kamdar [:vish_moz]]

Suggestions:

 1. default should be smoketests
 2. put stderr on top instead of stdout

for rest of the 80% of the code which I don't understand you will have to explain line by line. lol.

Comment 5

[Truman Smith [:truman]]

AMO_tests.py contains the line:  BFT.extend(smoketests)
Does that mean that the smoke tests suites is contained in BFT?  It may be a like a subset in a vague sense, but in practice smoke tests, BFT, and FFT are independent suites.  Any test cases that they share are run with different parameters.

Comment 6

[Truman Smith [:truman]]

A formatting thing - multiple tcparams might be easier to read if they are vertical, especially if there are several.  For example:

"tcparams":{"maxPagesPerInnerOption":1, 
            "maxItemsPerPage":1}})

Comment 7

[Truman Smith [:truman]]

Recently we started experimenting with separate Hudson jobs for amo.next and amo.preview.  If the idea sticks it's not obvious how the driver could support that.

Comment 8

[Tanay Gavankar [:tgavankar]]

The extend only applies to the names of the test - the parameters that are applied are pulled from whether you run a smoketest, BFT, or FFT. This way, the tests that are run during smoketests will also be run in BFT/FFT, except with different (higher) parameters, as well as more browsers, so it will be testing a different set of functionality. 

As for TCparams, that's what would happen, formatting wise - there just weren't any TCs in the original suites (I copied from there) that had multiple TCparam modifications.

The AMO version of the driver was done before we had decided to run on both preview and next. It wouldn't be difficult to add support for an additional command line param (like if you pass "nextURL" or something as an argument) it would replace TCparams.website with the appropriate website. Same thing with if we want support for prod. We could also do something like instead of hardcoding the URL, set it so that if you do -Uhttp://www.next.addons.mozilla.org (or whatever URL) it'll replace website with that url. Definitely many options that are relatively straightforward to implement - we can discuss it at the automation meeting.

Comment 9

[Tanay Gavankar [:tgavankar]]

Implemented your suggestions, Vishal. 

Also added the option for -U which lets us specify a custom url for the entire suite (used for preview vs next for AMO). Add the command line -Uhttp://next.addons.mozilla.org anywhere, order doesn't matter - note that there is NOT a space between -U and the url itself.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Closing this old bug out.