1192965 - ./mach talos-test should allow changing the number of tests run

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

When running ./mach talos-test chromez-e10s for example, the resize test runs multiple times. While this is fine as a default, sometimes I just want to run a single iteration of the test and there doesn't appear to be a way to do that. A flag for talos-test or env variable to control this behaviour would be nice.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

there are a few use cases:
* you want what is by default
* you want a few data points
* you want a single data point

I think we should figure out a way to add a flag to mach to describe scenarios, and if we pass in something like 'chromez-e10s', it would use production since that is a suite, but if you pass in 'tresize' e10s, it would assume local development and run a single data point.  I am not sure how mach handles commands.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

(In reply to Joel Maher (:jmaher) from comment #1)
> I think we should figure out a way to add a flag to mach to describe
> scenarios, and if we pass in something like 'chromez-e10s', it would use
> production since that is a suite, but if you pass in 'tresize' e10s, it
> would assume local development and run a single data point.

This would be nice for sure. Just having some way to control it at all is a "good enough" start for me though :)

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Attachment: WIP

Comment 4

[Julien Pagès (:parkouss)]

Kartikaya, maybe you could pass the talos options ('--cycles', '--tpcycles' and '--tppagecycles') using 'talos_extra_options' from the make_config function in mach_commands.py. I suspect this way you won't need to make any changes into talos.py.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Thanks for the tip! I had tried doing that earlier but I was putting talos_extra_options in make_args and that didn't work. Putting it in make_config works though.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Attachment: WIP v2

Comment 7

[Julien Pagès (:parkouss)]

Thanks Kartikaya for reporting this and the work here! This is a great (almost required!) feature for running Talos using mach.

Hmm, not sure why but I can't apply the patch on m-c because of conflicts. Could you look into that please ?

hg patch https://bug1192965.bmoattachments.org/attachment.cgi?id=8646300
applying https://bug1192965.bmoattachments.org/attachment.cgi?id=8646300
patching file testing/talos/mach_commands.py
Hunk #1 FAILED at 18
Hunk #2 FAILED at 76
Hunk #3 FAILED at 116
3 out of 3 hunks FAILED -- saving rejects to file testing/talos/mach_commands.py.rej
abort: patch failed to apply


Otherwise this looks good to me and we should land that I think - assuming it works (don't need to push try, this is not used by harness, just a local test).

:jmaher, thoughts ?

Comment 8

[Kartikaya Gupta (email:[e-mail])]

The patch applies on top of the patch in bug 1192964, but unfortunately that one has try failures. I can rebase it so this one applies first but ideally I'd like to land them both anyway.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this is sort of an ugly hack as we are setting cycles, tpcycles, tppagecycles all the same.  If we have iterations >1, it will be confusing.

one thought is we could fix tcanvasmark to either use tppagecycles or cycles and then we would have everything using tpcycles=1, this could eliminate that variable.

How to differentiate between cycles/tppagecycles?  We could split it on tests which are pageloader vs startup.  I think for now ignoring tpcycles would be ok, and we can figure out how to simplify our usage internally while making cycles more clear inside of talos.

Comment 10

[Julien Pagès (:parkouss)]

ok, now we can just use the standard talos flags (from bug 1206415) so '--cycles', '--tpcycles' and '--tppagecycles' are available.

Closing the bug, please reopen it if that does not work or if I'm missing something.