1280414 - Unclear what the different cycling options do

Status: NEW

(Testing :: Talos, enhancement, P3)

Description

[William Lachance (:wlach)]

We have way too many options for specifying the # of cycles in talos tests: 

--cycles "number of browser cycles to run"
--tpcycles "number of pageloader cycles to run"
--tppagecycles "number of pageloader cycles to run for each page in"

My understanding is that we only need two types of cycling: "browser cycles" (number of times to restart browser and re-run the tests?) and "page cycles" (number of times to reload page within a browser cycle)

Proposal:

* Eliminate tppagecycles altogether
* Rename "browser cycles" to "test runs" (maybe specify with --runs)
* Ideally come up with a better way of describing -tpcycles, though I'm not sure what that would be

jmaher: does this make sense to you? if so, maybe we could find someone to do this as part of the quarter of contribution

Comment 1

[Avi Halachmi (:avih)]

I _think_ the difference between tpcycles and tppagecycles is row-major vs column major, i.e. tppagecycles reloads each page few times in a row, then continues to the next page, while tpcycles cycles the whole pageloader run few times, i.e. does page 1, page 2, ... page N, then tpcycle again to page 1, page 2 etc. (I'm guessing taking tppagecycles into account).

I don't recall which tests use which and whether or not it's still relevant, so maybe the first step should be to list which tests use/still-require which cycling mechanism.

I think there's still value in all three cycling mechanisms, though I absolutely agree it can be confusing to casual users where you use 1 for one of them and still get few from another cycling param.

I guess that if we really wanted to, and if it's actually meaningful, we could come up with a config at test.py and then another param which controls all three cycling systems together, such that, for instance, if test.py configs 4 (browser) cycles, 3 tpcycles and 2 tppagecycles, and the user sets "overallcycles=1", then it will run 1 cycle, 1 tpcycle and 1 tppagecycle. And if the user chooses overallcycles=2, then it will use 2 for one of those and leave the other intact, according to some internal priority.

Or, just leave the options as is and communicate their semantics better? I don't think it would be hard to explain/and or grasp what each of those cycles are.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to remove tpcycles and just call it cycles, they are effectively the same.  In fact we use cycles for ts type tests (i.e. no pageloader), and there is tpcycles=1 for pageloader tests (except for tcanvasmark which is tpcycles=5).

In fact, I think tcanvasmark could be run with tppagecycles and we could just simplify that way.  Lastly if we assume tppagecycles (row major) is used in all tests, then we could remove that and just call it cycles.

that is a strict method of thinking about it, but we could do: cycles, pagecycles and call it good.

Comment 3

[Avi Halachmi (:avih)]

I agree we could let it come down to "browser cycles" and "component cycles".

So browser cycles sounds quite straight forward to me, but components cycles might have slightly different semantics depending on the kind of test it is, and also not all tests would necessarily support it.

In general, component cycles could mean "repeat each component of the test N times before continuing to the next test component", like tppagecycles with pageloader.