1115699 - talos compare.py should have an abbreviated mode that only outputs improvements/regressions

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently talos: compare.py outputs data for all tests specified (defaults to all).

It would be great if we had a --verbose flag that was required for all this data and when it wasn't specified that it would only output improvements or regressions.

Comment 1

[Vaibhav (:vaibhav1994)]

Attachment: verbose.patch

The output by using this patch:

1) Verbose flag not used:

vaibhav@vaibhav:~/talos/talos$ python compare.py --rev 8991815dd086 --branch Firefox --masterbranch Firefox --platform Linux64
   test			min.	->	max.	rev.
Linux64:

:) media_tests       	    4.0	->	    5.2	    4.0

2) Verbose flag used

vaibhav@vaibhav:~/talos/talos$ python compare.py --rev 8991815dd086 --branch Firefox --masterbranch Firefox --platform Linux64 --verbose
   test			min.	->	max.	rev.
Linux64:

   tresize           	   17.1	->	   22.1	   17.5
   kraken            	 1431.3	->	 1466.5	 1443.3
   v8_7              	17808.5	->	18817.5	18565.7
   dromaeo_css       	 4906.5	->	 5424.2	 5151.1
   dromaeo_dom       	  847.0	->	  945.8	  884.0
   a11yr             	  309.0	->	  334.0	  313.0
   ts_paint          	  865.8	->	  919.9	  895.5
   tpaint            	  201.2	->	  226.0	  210.4
   tsvgr_opacity     	  121.0	->	  286.0	  198.5
   tp5o              	  279.9	->	  299.7	  287.6
   tart              	    5.0	->	    5.1	    5.1
   tcanvasmark       	 5437.5	->	 6898.0	 6605.0
   tsvgx             	  574.4	->	  598.8	  594.6
   tscrollx          	    8.5	->	    9.1	    9.0
   sessionrestore    	 1630.2	->	 1688.4	 1688.4
   sessionrestore_no_auto_restore	  641.6	->	  682.7	  669.4
   glterrain         	   21.5	->	   24.1	   23.4
   cart              	   35.1	->	   38.2	   36.5
   tp5o_scroll       	    8.9	->	    9.3	    9.1
:) media_tests       	    4.0	->	    5.2	    4.0

Joel, please review this. Thanks!

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8541769 [details] [diff] [review]
verbose.patch

Review of attachment 8541769 [details] [diff] [review]:
-----------------------------------------------------------------

overall the code is great, just one small thing to adjust.  Marking as r+ as this is more of a nice to have cleanup.

::: talos/compare.py
@@ +362,5 @@
>                          string = "\t%s" % url
> +                    # Output data for all tests in case of --verbose.
> +                    # If not --verbose, then output in case of improvements or regression.
> +                    if verbose or (not verbose and (':(' in string or ':)' in string)):
> +                        output.append(string)

how about just check for status != '', then the if statement would look a lot cleaner like this:
if verbose and status != '':
    output.append(string)

Comment 3

[Vaibhav (:vaibhav1994)]

Attachment: verbose.patch

Fixed up to make the code more readable. We will require an 'or' keyword instead of 'and' since we want to print out all tests in case of verbose. Joel, thanks for the review!

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/build/talos/rev/e64e2fa17681

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks, this is great!