668063 - Allow testruns to specify a list of tags

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Automation, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

For endurance tests we are going to need to filter results run from a specific machine so as to limit the variables when running tests with multiple Firefox builds. I suggest adding a --tags command line option to specify one of more tags. We should then be able to use these to filter results in the dashboard.

Comment 1

[u415141]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/15491557

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Allow testruns to specify a list of tags. v1.0

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 589890 [details] [diff] [review]
Allow testruns to specify a list of tags. v1.0

Looks good. One thing which we better should do is to not use a comma separated list for the command line. Add-ons are getting specified with separate options so we should probably do the same here for tags. Otherwise we have an inconsistency. Sorry that I missed that yesterday in the call.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Allow testruns to specify a list of tags. v1.1

Changed to match behavior of addons. Now each time the --tag command line argument is used, the tag is appended to a list, which is attached to the report.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 590682 [details] [diff] [review]
Allow testruns to specify a list of tags. v1.1

>+                      ("--tag",): dict(dest="tags",
>+                                        action="append",
>+                                        default=[ ],

This has to default to None. Same as for addons in the mozrunner.__init.py__ file.

>+        report['tags'] = self.options.tags

We should only add the property if tags have been specified.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Allow testruns to specify a list of tags. v1.2

As discussed on IRC changing default to None is not needed. Added if statement to only add tags section to report if there's at least one tag.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 590691 [details] [diff] [review]
Allow testruns to specify a list of tags. v1.2

>+                      ("--tag",): dict(dest="tags",
>+                                        action="append",
>+                                        default=[ ],
>+                                        metavar="TAG",
>+                                        help="Tag to apply to the report")

Can you please fix the indentation before the check-in?

Otherwise looks good. r=me.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/2b525372b8ca

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So this is fixed now. Lets close it out. Thanks Dave!