1459251 - Running |mach raptor-test| with no test specified results in a traceback

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Description

[Andrew Halberstadt [:ahal]]

This should probably have a better error message. Semi-relatedly, how can one fine the list of valid test names that can be passed in? It might be good to make the 'test' argument a choices action so that they can be discovered via |mach talos-test --help|.

Comment 1

[Andrew Halberstadt [:ahal]]

Making it a 'choices' action would also make sure we fail fast if an invalid test were passed in (currently it does ~20 seconds worth of mozharness setup before failing).

Comment 2

[Robert Wood [:rwood]]

Thanks yes I have this fixed already in the patch I'm working on for processing output. I'll move it to a separate patch here tho thanks.

Comment 3

[Robert Wood [:rwood]]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> Making it a 'choices' action would also make sure we fail fast if an invalid
> test were passed in (currently it does ~20 seconds worth of mozharness setup
> before failing).

Yeah so I use manifest parser, that builds a list of available tests first. If a test argument is passed in then I check the list of tests manifest parser created. If it's in that list then it's valid, otherwise errors out invalid test. If no test is specified then all available tests will run.

Comment 4

[Robert Wood [:rwood]]

Is there a way to do what you want and still use test manifests? Maybe make the test argument a choices argument but get the choices from manifest parser...

Comment 5

[Robert Wood [:rwood]]

Attachment: Bug 1459251 - Running |mach raptor-test| with no test specified results in a traceback;

Review commit: https://reviewboard.mozilla.org/r/241754/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/241754/

Comment 6

[Robert Wood [:rwood]]

Comment on attachment 8973289 [details]
Bug 1459251 - Running |mach raptor-test| with no test specified results in a traceback;

This patch fixes the case of running:

./mach raptor-test

Now it will find and run all available tests. Note, for me on OSX, if you provide an invalid test name i.e:

./mach raptor-test invalid-test-name

It only takes 2 seconds before it errors out with:

09:58:12     INFO -  manifest abort: specified test doesn't exist!
09:58:12     INFO -  raptor-main abort: no tests found
09:58:12    ERROR - Return code: 1

Which is fine IMO.

Comment 7

[Robert Wood [:rwood]]

Comment on attachment 8973289 [details]
Bug 1459251 - Running |mach raptor-test| with no test specified results in a traceback;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/241754/diff/1-2/

Comment 8

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #6)
 
> ./mach raptor-test invalid-test-name
> 
> It only takes 2 seconds before it errors out with:
> 
> 09:58:12     INFO -  manifest abort: specified test doesn't exist!
> 09:58:12     INFO -  raptor-main abort: no tests found
> 09:58:12    ERROR - Return code: 1
> 
> Which is fine IMO.

Ah, because I already ran Raptor previously and the raptor-venv already existed, otherwise yeah I see about 20 seconds before it errors out. I think it's fine as that's only the first time if the raptor-venv doesn't exist yet.

Comment 9

[Robert Wood [:rwood]]

Comment on attachment 8973289 [details]
Bug 1459251 - Running |mach raptor-test| with no test specified results in a traceback;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/241754/diff/2-3/

Comment 10

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b3d9ebc33860bbb984cfcc6bd65baf2ddc2f3c7

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 8973289 [details]
Bug 1459251 - Running |mach raptor-test| with no test specified results in a traceback;

https://reviewboard.mozilla.org/r/241754/#review248934

Thanks!

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fe2a0b79e7af
Running |mach raptor-test| with no test specified results in a traceback; r=ahal

Comment 13

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/fe2a0b79e7af