Closed Bug 1192965 Opened 5 years ago Closed 5 years ago

./mach talos-test should allow changing the number of tests run

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: kats, Unassigned)

References

Details

Attachments

(1 file, 1 obsolete file)

When running ./mach talos-test chromez-e10s for example, the resize test runs multiple times. While this is fine as a default, sometimes I just want to run a single iteration of the test and there doesn't appear to be a way to do that. A flag for talos-test or env variable to control this behaviour would be nice.
there are a few use cases:
* you want what is by default
* you want a few data points
* you want a single data point

I think we should figure out a way to add a flag to mach to describe scenarios, and if we pass in something like 'chromez-e10s', it would use production since that is a suite, but if you pass in 'tresize' e10s, it would assume local development and run a single data point.  I am not sure how mach handles commands.
(In reply to Joel Maher (:jmaher) from comment #1)
> I think we should figure out a way to add a flag to mach to describe
> scenarios, and if we pass in something like 'chromez-e10s', it would use
> production since that is a suite, but if you pass in 'tresize' e10s, it
> would assume local development and run a single data point.

This would be nice for sure. Just having some way to control it at all is a "good enough" start for me though :)
Attached patch WIP (obsolete) — Splinter Review
Kartikaya, maybe you could pass the talos options ('--cycles', '--tpcycles' and '--tppagecycles') using 'talos_extra_options' from the make_config function in mach_commands.py. I suspect this way you won't need to make any changes into talos.py.
Thanks for the tip! I had tried doing that earlier but I was putting talos_extra_options in make_args and that didn't work. Putting it in make_config works though.
Attached patch WIP v2Splinter Review
Attachment #8645929 - Attachment is obsolete: true
Thanks Kartikaya for reporting this and the work here! This is a great (almost required!) feature for running Talos using mach.

Hmm, not sure why but I can't apply the patch on m-c because of conflicts. Could you look into that please ?

hg patch https://bug1192965.bmoattachments.org/attachment.cgi?id=8646300
applying https://bug1192965.bmoattachments.org/attachment.cgi?id=8646300
patching file testing/talos/mach_commands.py
Hunk #1 FAILED at 18
Hunk #2 FAILED at 76
Hunk #3 FAILED at 116
3 out of 3 hunks FAILED -- saving rejects to file testing/talos/mach_commands.py.rej
abort: patch failed to apply


Otherwise this looks good to me and we should land that I think - assuming it works (don't need to push try, this is not used by harness, just a local test).

:jmaher, thoughts ?
The patch applies on top of the patch in bug 1192964, but unfortunately that one has try failures. I can rebase it so this one applies first but ideally I'd like to land them both anyway.
this is sort of an ugly hack as we are setting cycles, tpcycles, tppagecycles all the same.  If we have iterations >1, it will be confusing.

one thought is we could fix tcanvasmark to either use tppagecycles or cycles and then we would have everything using tpcycles=1, this could eliminate that variable.

How to differentiate between cycles/tppagecycles?  We could split it on tests which are pageloader vs startup.  I think for now ignoring tpcycles would be ok, and we can figure out how to simplify our usage internally while making cycles more clear inside of talos.
See Also: → 1206415
ok, now we can just use the standard talos flags (from bug 1206415) so '--cycles', '--tpcycles' and '--tppagecycles' are available.

Closing the bug, please reopen it if that does not work or if I'm missing something.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.