994860 - update compare.py to use current platforms/tests

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Our platforms and tests switch over time, but not all tools that reference them change.

to get started with talos, here are some first steps:
https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code

here is a primer on compare.py:
http://elvis314.wordpress.com/2012/07/13/mozilla-a-team-how-to-compare-talos-numbers-from-try-server-to-trunk-13/

I recommend running this tool first and seeing what is missing.

Some things I know need changing:
* remove OSX 10.7
* remove tspaint_places_med and tspaint_places_max
* fix tscrollx

Comment 1

[Rafael Vieira]

Hi Joel. I'm interested to be assigned to!

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Rafael, thanks for your interest!  You are now assigned to this bug.  Please don't feel shy to ask in here about any questions or in IRC.  Looking forward to your patch!

Comment 3

[Rafael Vieira]

Hi, Joel. I've started working here and now I built the Talos locally and obtain a result of the test.

" [{"test_machine": {"platform": "x86", "osversion": "Ubuntu 12.10", "os": "linux", "name": "qm-pxp01"}, "talos_aux": {"shutdown": [463, 422, 539, 726, 831, 18, 124, 204, 304, 512, 621, 714, 879, 987, 184, 271, 364, 498, 670, 773]}, "testrun": {"date": 1399669772, "suite": "ts", "options": {"responsiveness": false, "tpmozafterpaint": true, "tpchrome": true, "tppagecycles": 1, "tpcycles": 10, "tprender": false, "shutdown": true, "extensions": [{"name": "[e-mail]"}], "rss": false}}, "results": {"ts": [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]}, "test_build": {"version": "32.0a1", "revision": "cf89b5d018f8", "id": "20140509025042", "branch": "", "name": "Firefox"}}] "

I've changed the compare.py code, deleting OSX 10.7 and tspaint_palces_med/max. But I wish understand a bit more about the test. Now How could I analyse the current plataform/tests to make a full update and see the changes necessaries?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

if you run compare.py:
python compare.py --rev 7586c29906b5 --branch Firefox

A lot of the tests are defined here (along with extras):
https://github.com/jmaher/alert_manager/blob/master/parse_news.py#L66

Here is where we define running the tests:
http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json

That should give you a good idea.  It sounds like you are well on your way to your patch!

Comment 5

[Rafael Vieira]

I've made that changes:
* remove OSX 10.7
* remove tspaint_places_med and tspaint_places_max
* fix tscrollx

and I'm receveing this result for Tscroll:
   test			min.	->	max.	rev.	Datazilla
Linux:
   tscrollx          	    9.5	->	    9.8	    9.6	[N/A] [PGO: N/A]

This value of "tscrollx" correspond to test "tscroll-ASAP" that I changed from "tscroll-ASAP MozAfterPaint", because I noticed that the second don't have values associated to him. All plataforms are now showing the value os "tscrollx".

I noticed that my result was captured all from "Firefox-Non-PGO" (id = 94). Have I forgot Something? Is It necessary I'm running on PGO mode? Is tp5n active? I just see tp5o test.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks for the progress on this!

tp5n is only active on windows 7 and we know it as xperf, there is a --xperf mode which takes care of that.  I would remove the tp5n reference also.

Recently we added:
sessionrestore (id 313)
sessionrestore_no_auto_restore (id 315)

pgo vs non pgo is confusing, we could probably do a better job of ensuring we have proper results for everything, but we could tackle that in a separate bug.

Comment 7

[Rafael Vieira]

I've made this update putting sessionrestore and sessionrestore_no_auto_restore and the values are being showed. I repeated the names of tests on tbplname field. How could I send the code to be checked?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

awesome!  Here is a link to working with mercurial queues: https://developer.mozilla.org/en-US/docs/Mercurial_Queues.  

I would try to work through that, it will give a consistent diff (look at the changes required to .hgrc) and can keep the author information.

When you get done with the queues (of if you can't get it working), do a 'hg diff > 994860.patch' and upload that as an attachment to this bug!

Comment 9

[Rafael Vieira]

Attachment: Bug 994860 - fixing tscrollx, creating sessionrestore and sessionrestore_no_auto_restore, removing OSX10.7 and tspaint_places_med/max

I guess that I made correctly

Comment 10

[Rafael Vieira]

Attachment: Bug 994860 - fixing tscrollx, creating sessionrestore and sessionrestore_no_auto_restore, removing OSX10.7 and tspaint_places_med/max (Correct)

Sorry by the last. I forgot something important

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8422176 [details] [diff] [review]
Bug 994860 - fixing tscrollx, creating sessionrestore and sessionrestore_no_auto_restore, removing OSX10.7 and tspaint_places_med/max (Correct)

Review of attachment 8422176 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

landed:
https://hg.mozilla.org/build/talos/rev/53c8b1ad4828

Rafael, Congrats on finishing your bug! We have more issues in Talos as well as other fun projects to work on as well.  Please ask around if you have trouble finding stuff.  We are in IRC in the #ateam channel.

Thanks again for your contribution!