1201963 - The test suite should print something useful when you try to run something that is not a test

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[(no longer active)]

Currently this is what gets printed which is not quite helpful.

$ ./mach web-platform-tests testing/web-platform/mozilla/tests/service-workers/service-worker/resources/fetch-canvas-tainting-iframe.html
 0:00.48 LOG: MainThread INFO Using 1 client processes
 0:00.52 LOG: MainThread wptserve INFO Starting http server on 127.0.0.1:8001
 0:00.52 LOG: MainThread wptserve INFO Starting http server on 127.0.0.1:8000
 0:00.52 LOG: MainThread wptserve INFO Starting http server on 127.0.0.1:8443
 0:02.51 SUITE_START: MainThread 0
 0:02.51 LOG: MainThread INFO Running testharness tests
 0:02.52 LOG: MainThread INFO No testharness tests to run
 0:02.52 LOG: MainThread INFO Running reftest tests
 0:02.52 LOG: MainThread INFO No reftest tests to run
 0:02.52 LOG: MainThread INFO Got 0 unexpected results
 0:02.52 SUITE_END: MainThread
Summary
=======

Ran 0 tests
Expected results: 0
Unexpected results: 0

OK
 0:02.53 LOG: MainThread INFO Closing logging queue
 0:02.53 LOG: MainThread INFO queue closed

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Three years later, this is what prints when you run something that's not a test:


$ ./mach wpt testing/web-platform/tests/not-a-test
 0:00.00 INFO Skipping manifest download because existing file is recent
 0:06.20 INFO Using 1 client processes
 0:09.45 INFO STDERR: WARNING:root:Unable to install font: Ahem.ttf
 0:09.50 INFO Starting http server on 127.0.0.1:8000
 0:09.50 INFO Starting http server on 127.0.0.1:8001
 0:09.53 INFO Starting https server on 127.0.0.1:8443
 0:10.11 SUITE_START: web-platform-test - running 0 tests
 0:10.11 INFO Running reftest tests
 0:10.11 INFO No reftest tests to run
 0:10.11 INFO Running wdspec tests
 0:10.11 INFO No wdspec tests to run
 0:10.11 INFO Running testharness tests
 0:10.11 INFO No testharness tests to run
 0:10.11 INFO Got 0 unexpected results
 0:10.11 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 0 checks ()
Expected results: 0
OK
 0:10.14 INFO Closing logging queue
 0:10.14 INFO queue closed
 0:10.14 ERROR No tests ran


So, it's better. Enough to call this fixed?

Comment 2

[(no longer active)]

Well, compare with:

$ ./mach mochitest am/i/seeing/whats/not/there
The mochitest command could not find any mochitests under the following
test path(s):

am/i/seeing/whats/not/there

Please check spelling and make sure there are mochitests living there.

I guess we can close the bug because of statute of limitations, but I wouldn't really call this output three years after reporting this a lot more informative in telling the developer, hey my friend, you just made a typo.  :-)

If you think this bug will continue sitting here without receiving any attention, then fine let's close it I guess.

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1201963 - Give a better warning when no tests can be found in the specified path r?jgraham

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

With this, the output is this:

wkocher-vffvh8:mozilla-central wkocher$ ./mach wpt testing/not/a/test some/other/path
 0:00.00 INFO Skipping manifest download because existing file is recent
 0:05.49 ERROR Unable to find any tests at the path(s):
 0:05.49 ERROR   testing/not/a/test
 0:05.49 ERROR   some/other/path
 0:05.49 ERROR Please check spelling and make sure there are tests in the specified path(s).
 0:05.49 INFO Closing logging queue
 0:05.49 INFO queue closed
 0:05.50 ERROR No tests ran




Word-smithing welcome :)

Comment 5

[(no longer active)]

Beautiful, thank you!

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/13d9e2a48dbb
Give a better warning when no tests can be found in the specified path r=jgraham

Comment 7

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14700 for changes under testing/web-platform/tests

Comment 8

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/14700
* continuous-integration/travis-ci/pr (https://travis-ci.com/web-platform-tests/wpt/builds/96167396?utm_source=github_status&utm_medium=notification)

Comment 9

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 10

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/13d9e2a48dbb

Comment 11

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged