Closed Bug 1429388 Opened 2 years ago Closed 2 years ago

wdspec outputs test percentage progress after pytest upgrade

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox59 fixed)

RESOLVED FIXED
mozilla59
Tracking Status
firefox59 --- fixed

People

(Reporter: ato, Assigned: ato)

Details

Attachments

(1 file)

After the recent upgrade of pytest upstream that yesterday landed
in m-c, the WPT wdspec tests outputs a percentage progress number
of how far along it is running a test.

This is misleading because we invoke pytest on a per-file level which
means the percentage is only accurate on for one test file at a time,
but more importantly it causes a lot more output which is distracting:

>  0:12.08 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_option 
>  0:12.09 pid:27064 1515590214026	mozrunner::runner	INFO	Running command: "/home/ato/src/gecko/obj-x86_64-pc-linux-gnu/dist/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.yIWe3T4R3eWL"
>  0:12.12 pid:27064 *** You are running in headless mode.
>  0:12.46 pid:27064 1515590214394	Marionette	INFO	Enabled via --marionette
>  0:13.67 pid:27064 1515590215609	Marionette	INFO	Listening on port 2828
>  0:14.21 INFO STDOUT: PASSED
>  0:14.21 INFO STDOUT:  [  9%]
>  0:14.29 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_multiple_option 
>  0:14.62 INFO STDOUT: PASSED
>  0:14.62 INFO STDOUT:  [ 18%]
>  0:14.67 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_preselected_option 
>  0:15.23 INFO STDOUT: PASSED
>  0:15.23 INFO STDOUT:  [ 27%]
>  0:15.27 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_preselected_multiple_option 
>  0:15.85 INFO STDOUT: PASSED
>  0:15.85 INFO STDOUT:  [ 36%]
>  0:15.89 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_deselects_others 
>  0:16.86 INFO STDOUT: PASSED
>  0:16.86 INFO STDOUT:  [ 45%]
>  0:16.93 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_multiple_does_not_deselect_others 
>  0:17.68 INFO STDOUT: PASSED
>  0:17.68 INFO STDOUT:  [ 54%]
>  0:17.72 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_selected_option 
>  0:18.25 INFO STDOUT: PASSED
>  0:18.25 INFO STDOUT:  [ 63%]
>  0:18.29 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_click_selected_multiple_option 
>  0:18.81 INFO STDOUT: PASSED
>  0:18.81 INFO STDOUT:  [ 72%]
>  0:18.84 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_out_of_view_dropdown 
>  0:19.14 INFO STDOUT: PASSED
>  0:19.14 INFO STDOUT:  [ 81%]
>  0:19.17 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_out_of_view_multiple 
>  0:19.46 INFO STDOUT: PASSED
>  0:19.46 INFO STDOUT:  [ 90%]
>  0:19.50 INFO STDOUT: testing/web-platform/tests/webdriver/tests/element_click/select.py::test_option_disabled 
>  0:19.81 INFO STDOUT: PASSED
>  0:19.81 INFO STDOUT:  [100%]
>  0:19.85 INFO STDOUT: ========================== 11 passed in 7.78 seconds ===========================
>  0:19.88 TEST_END: Test OK. Subtests passed 11/11. Unexpected 0
Do we know what the pytest option is to disable this new “feature”?
Flags: needinfo?(dburns)
You can set console_output_style = classic in the config file, or use -o console_output_style=classic on the command line.
(In reply to Dave Hunt (:davehunt) from comment #2)

> You can set console_output_style = classic in the config file, or
> use -o console_output_style=classic on the command line.

Unfortunately I cannot get this working with the way we invoke
pytest.main():

> pytest.main(["-o", "console_output_style=classic"])

Even with variations of that I still keep getting this error:

> 0:03.55 INFO STDERR: ERROR: -o/--override-ini expects option=value style.
I've raised https://github.com/pytest-dev/pytest/issues/3103 to address this in pytest, however I believe passing as follows should work for us in the meantime:

> pytest.main(["-o=console_output_style=classic"])
(In reply to Dave Hunt (:davehunt) from comment #4)
> […] however I believe passing as follows should work for us in the
> meantime:
> 
> > pytest.main(["-o=console_output_style=classic"])

I can confirm that this works.
Flags: needinfo?(dburns)
Assignee: nobody → ato
Comment on attachment 8941463 [details]
Bug 1429388 - Use classic pytest output style for WPT.

https://reviewboard.mozilla.org/r/211746/#review217534
Attachment #8941463 - Flags: review?(dave.hunt) → review+
Comment on attachment 8941463 [details]
Bug 1429388 - Use classic pytest output style for WPT.

https://reviewboard.mozilla.org/r/211746/#review217576
Attachment #8941463 - Flags: review?(james) → review+
Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4f065bc82a46
Use classic pytest output style for WPT. r=davehunt,jgraham
https://hg.mozilla.org/mozilla-central/rev/4f065bc82a46
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
You need to log in before you can comment on or make changes to this bug.