699512 - allow for adjusting the tpcycles via the commandline

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

example commandline:
  url :  '-tp page_load_test/tp5/tp5.manifest -tpchrome -tpnoisy -tpformat tinderbox -tpcycles 10'

possible new commandline:
  url :  '-tp page_load_test/tp5/tp5.manifest -tpchrome -tpnoisy -tpformat tinderbox -tpcycles 25'


this should be something like --test-cycles 25 on the perfconfigurator.py commandline.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

ok, this isn't a dup of that bug, but talking with :wlach, we think that just adding a custom .config file will be all we need.  The patch to change this will be very hacky without refactoring a lot of code.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

so thinking about this a bit more, we are looking to test various durations of the test to prove out how many runs we need for stability.  This is easily done with a custom .config file, but it will require us to checkin and redeploy talos each time.  

I think if when we do this on a custom branch if we can point talos to a different instance instead of the default talos.zip which everything uses we can be successful.

Comment 4

[cmtalbert]

Guys, the intern that is on the hook to measure this goes back to school in January.  When do we intend to see something land here?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add a config file with tpcycles 50 for tp5

not sure if 50 is the number of cycles, and this is only for tp5.

Comment 6

[Stephen Lewchuk]

Comment on attachment 575081 [details] [diff] [review]
add a config file with tpcycles 50 for tp5

Review of attachment 575081 [details] [diff] [review]:
-----------------------------------------------------------------

The tp test changed is actually tp3 not tp5.  Would it be possible to set all, tp5, tdhtml, tsvg, a11y and tsspider to the higher cycle count?

Unless there is a reason I would feel better with 100 iterations but 50 should also be sufficient.

Finally, this looks like a windows only config: 'browser_path: firefox/firefox.exe', will this modified config be run across multiple OSs?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add a config file with tpcycles 100 for most talos tests (2.0)

ok, updated with the requests.  Firefox.exe is hardcoded to the config file but is overwritten with commandline arguments from buildbot when running on all different operating systems.

Comment 8

[Stephen Lewchuk]

Comment on attachment 575247 [details] [diff] [review]
add a config file with tpcycles 100 for most talos tests (2.0)

Review of attachment 575247 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

landed the talos bits as a cycles.config file:
http://hg.mozilla.org/build/talos/rev/36c0f5cb8b35

next steps are to get a branch and we can work on the buildbot configs to use this and configure the tests properly.