1532316 - Improve Raptor test manifest error output

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P3)

Description

[Robert Wood [:rwood]]

Currently if you try to run ./mach raptor-test --test <test-name> with a test name that is invalid or is disabled etc. Raptor ultimately error's out with:

15:23:45 INFO - raptor-manifest abort: specified test name doesn't exist
15:23:45 INFO - raptor-main abort: no tests found

The problem is that the last line (no tests found) is the line that is picked up by the sheriffs when filing intermittent bugs. Also if someone is running Raptor and sees that single line it isn't very helpful. The final error output should be more detailed i.e. the name of the test attempted to run, etc, or whatever other error happens in manifest.py.

The first error is generated at [0]; the final 'no tests found' error is at [1]. The error message from manifest.py should be passed back to the main raptor.py in order to make it more detailed instead of just 'no tests found'.

[0] https://searchfox.org/mozilla-central/rev/92d11a33250a8e368c8ca3e962e15ca67117f765/testing/raptor/raptor/manifest.py#239

[1] https://searchfox.org/mozilla-central/rev/92d11a33250a8e368c8ca3e962e15ca67117f765/testing/raptor/raptor/raptor.py#657

Comment 1

[Kyra]

Hello, I would like to take this bug. To clarify, is updating the "no tests found" to an error message that says "no tests found for <test-name>" here enough?

Comment 2

[Robert Wood [:rwood]]

(In reply to Kyra from comment #1)

Hello, I would like to take this bug. To clarify, is updating the "no tests found" to an error message that says "no tests found for <test-name>" here enough?

Hi Kyra, that's great, thanks. There may be some other manifest errors that should be carried forward to the final output line too, for example when a certain type of test requires a specific test INI option but it is missing from the test INI, i.e. [0].

[0] https://searchfox.org/mozilla-central/rev/92d11a33250a8e368c8ca3e962e15ca67117f765/testing/raptor/raptor/manifest.py#51

Comment 3

[Kyra]

(In reply to Robert Wood [:rwood] from comment #2)

Hello Robert, from observing the code the function "validate_test_ini" will not be run if the test name is invalid or is disabled.. this is because of the if statement on line 230. Thus the "no tests found" message would never need to include anything from this function as the "tests_to_run" variable array will be empty. Please let me know if this is wrong.

Comment 4

[Robert Wood [:rwood]]

(In reply to Kyra from comment #3)

(In reply to Robert Wood [:rwood] from comment #2)

Hello Robert, from observing the code the function "validate_test_ini" will not be run if the test name is invalid or is disabled.. this is because of the if statement on line 230. Thus the "no tests found" message would never need to include anything from this function as the "tests_to_run" variable array will be empty. Please let me know if this is wrong.

Hi Kyra,

Apologies for not seeing your question - in the future when you have a question please select the 'Request information from' check box below, and that will set the 'Need Info' flag to draw my attention, thanks. It's been quite sometime - is this bug something that you're still interested/available to work on?

To answer your question yes that is correct. There are a couple of conditions that can cause errors here - a) the specified test name provided on the command line doesn't actually exist; b) the test exists but it is missing some required setting in the test definition / INI file. Either way it would be great to capture the errors. Suggestion - maybe have get_raptor_test_list return an additional value like 'test_errors', then if the tests_to_run list is empty fail out and report the error(s) returned.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

No response in 4 months. Clearing needinfo.

Comment 6

[Myeongjun Go]

I'll try to work on it.

Comment 7

[Myeongjun Go]

Attachment: Bug 1532316 - Improve Raptor test manifest error output r=sparky

Comment 8

[Myeongjun Go]

I read the guide carefully.
I didn't feel the need to make a new variable and return it in get_raptor_test_list. [0]
Can you tell me if there's anything more to check about the error output?
I'll revise it and submit patch again.

[0]
https://dxr.mozilla.org/mozilla-central/rev/31360ced8ff85ed36e79fc1350b9d1db79457526/testing/raptor/raptor/manifest.py#287

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/28e9b3d352db
Improve Raptor test manifest error output r=sparky,perftest-reviewers

Comment 10

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/28e9b3d352db