1495091 - documented syntax for running multiple tests is incorrect

Status: RESOLVED WORKSFORME

(Testing :: Talos, enhancement)

Description

[Dana Keeler (she/her) [:keeler]]

The documentation for talos says that one can run two single tests by separating their names with a colon, ala:

   ./mach talos-test --activeTests tart:damp

When I try this, I get:

13:45:16    FATAL - Test name is missing or invalid
13:45:16    FATAL - Running post_fatal callback...
13:45:16    FATAL - Exiting -1

(along with a page of info debug output)

(The documentation in this case is both https://wiki.mozilla.org/Performance_sheriffing/Talos/Running and `./mach talos-test --help`)

P.S. What is the actual syntax?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

odd, the code splits tests by ':'
https://searchfox.org/mozilla-central/source/testing/talos/talos/config.py#184

maybe this is an issue in ./mach

Comment 2

[Florin Strugariu [:Bebe]]

I investigated this and it look that it would require multiple changes in the way we run talos from ./mach

While initiating the talos tests ./mach sets up the dependencies for the suite name determined from the test name.
As we want to run multiple tests, possibly from different suites we would need to setup the env for each test/suite

See [1] to understand how we determine the suite name.
[1] https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py#283

One option would be to run each test separate so we can do the setup/tear-down for each test.

Please advise on how we should proceed with this bug.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I agree that we need to setup/teardown for each test, this ensures we are using the same process as automation and not writing code that is at risk of being out of date, etc.

So going forward, I would say we need to:
1) parse the list of tests to be returned in an array
2) when finding the suite and doing the setup, keep that singular, but loop through the array for each test

I would be curious to know if we need to support multiple tests from the ./mach command line- :keeler, since you seem to be the first person to use this in a while, is there a preferred use case or things we should ensure are done to make this easy to run and interpret the results?

Comment 4

[Dana Keeler (she/her) [:keeler]]

I was trying to reproduce a failure locally, and I think I was under the assumption that an earlier test was interfering somehow, so that's why I was trying to run more than one at a time. If the setup/teardown means that that's not possible, then this probably doesn't even need to be a feature if no one is using it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

lets close this for now; :keeler, if you are eager to do more with this, we are happy to work on it.

Comment 6

[Dana Keeler (she/her) [:keeler]]

Works for me, but I think it would be a good idea to remove the misleading documentation.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

:keeler, you are awesome- thanks for making sure we resolved this fully.  I dropped the ball on this and didn't update the docs, they are now updated:
https://wiki.mozilla.org/Performance_sheriffing/Talos/Running#Locally