1515245 - ./mach test can't run web-platform-tests on Windows

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[Brian Birtles (:birtles)]

On a fresh checkout of mozilla-central on Windows 10 I get the following:

  /c/moz/src1 (debug-opt)
  $ ./mach test testing/web-platform/tests/web-animations/
  cat: backend.TestManifestBackend.in: No such file or directory
  Build configuration changed. Regenerating backend.
  No handlers could be found for logger "mozbuild.frontend.reader"
  UNKNOWN TEST: testing/web-platform/tests/web-animations/
  I was unable to find tests from the given argument(s).

  You should specify a test directory, filename, test suite name, or
  abbreviation. If no arguments are given, there must be local file
  changes and corresponding IMPACTED_TESTS annotations in moz.build
  files relevant to those files.

  It's possible my little brain doesn't know about the type of test you are
  trying to execute. If you suspect this, please request support by filing
  a bug at
  https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=General.

However if I use `./mach wpt` instead of `./mach test` it works. On Linux both work.

Comment 1

[Brian Birtles (:birtles)]

This is probably a regression since I seem to recall this used to work on Windows.

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

For this test, I used `./mach test testing/web-platform/tests/web-animations/timing-model/animation-effects` on both OSX and Windows10.

So in this check[1], `path` is "testing/web-platform/tests/web-animations/timing-model/animation-effects" on both OSX and Windows, `path in self._test_dirs` is False on both, and `any(p.startswith(path) for p in self._tests_by_path)` is True on OSX but False on Windows. 

The keys in self._tests_by_path on OSX look like 'testing/web-platform/tests/foo/bar/baz.html'. On Windows, those same keys look like 'testing\\web-platform\\tests\\foo\\bar\\baz.html'.

What would be the preferred way of fixing this? Converting `path` to the Windows-escaped paths that are in _tests_by_path?



1. https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#427-428

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

FWIW, `./mach test testing/web-platform/tests/web-animations` works if I add `path = path.replace("/", "\\")`before line 427, but obviously that would break non-windows.

Comment 4

[James Graham [:jgraham]]

Yeah, but path.replace("\", os.path.sep) should work. It might be more efficient to do something like

if os.path.sep != "/":
    path = path.replace("/", os.path.sep)

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1515245 - Make sure windows paths are supported for wpt via generic 'mach test' r?jgraham

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 9033679 [details]
Bug 1515245 - Make sure windows paths are supported for wpt via generic 'mach test' r?jgraham

This works for me on mac and windows.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1923d02bb66d
Make sure windows paths are supported for wpt via generic 'mach test' r=jgraham

Comment 8

[Narcis Beleuzu [:NarcisB]]

Backed out for build bustages.

Backout link: https://hg.mozilla.org/integration/autoland/rev/251b7f4419f1289c12301ef1e7d1eb8cb7d01caf
Push link: https://hg.mozilla.org/integration/autoland/rev/1923d02bb66df7e1aeeebc78682816cf921c4a00
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219324545&repo=autoland&lineNumber=45759

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8040640ee1cd
Make sure windows paths are supported for wpt via generic 'mach test' r=jgraham

Comment 10

[Narcis Beleuzu [:NarcisB]]

Backed out for mozbase failures 

Backout link: https://hg.mozilla.org/integration/autoland/rev/f87eda1691a15119f93561c974fe7115702cb346
Push link: https://hg.mozilla.org/integration/autoland/rev/8040640ee1cd603043d7dfafe70365fd187a5307
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219336338&repo=autoland&lineNumber=1420

Comment 11

[James Graham [:jgraham]]

Hmm, so maybe the right fix here is to ensure that all the wpt paths are added with unix-style path seperators irrespective of the actual platform.

Comment 12

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

KWierso and jgraham:

This is really annoying bug to me. What's the status of this bug?

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I don't see myself getting back to this anytime soon. I think a workaround would be to use ./mach wpt path/to/tests rather than ./mach test path/to/tests

Comment 14

[James Graham [:jgraham]]

I don't have Windows installed at the moment to investigate this, but I'm about 90% sure that the problem will be in [1] where we add the list of wpt tests to the manifest used by mach test. My prediction is that something in the output of that loop has different path separator characters to the entries from other testsuites, or maybe just src_path has the wron path seperators. I think if you compare the items we add there to other items in the _tests_by_path dict then it should be obvious where the bug is.

[1] https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#482

Comment 15

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

(In reply to Wes Kocher (:KWierso) from comment #13)

I don't see myself getting back to this anytime soon. I think a workaround would be to use ./mach wpt path/to/tests rather than ./mach test path/to/tests

Ah, I might have met different bug... Even if I useed ./mach wpt ... on Windows, I couldn't run the test actually. However, today, I succeeded to run a WPT on my machine. So, the bug which I met has already been fixed by somebody.

(And ./mach test <a path to WPT> still doesn't work exactly same as this bug.)

Comment 16

[Brian Birtles (:birtles)]

This is still a minor annoyance when developing on Windows.