1763552 - ./mach wpt fails with that git returned non-zero exit status 128 on Windows

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Description

•

3 years ago

run ./mach wpt on Win11.

I got failure with the following error:

subprocess.CalledProcessError: Command '['git', 'diff-index', '--relative', '--no-renames', '--name-only', '-z', 'HEAD']' returned non-zero exit status 128.

  File "M:\src\testing\web-platform\mach_commands.py", line 531, in run_wpt
    return run_web_platform_tests(command_context, **params)
  File "M:\src\testing\web-platform\mach_commands.py", line 520, in run_web_platform_tests
    return wpt_runner.run(logger, **params)
  File "m:\src\testing\web-platform\mach_commands_base.py", line 42, in run
    self.update_manifest(logger)
  File "m:\src\testing\web-platform\mach_commands_base.py", line 65, in update_manifest
    return manifestupdate.run(
  File "m:\src\testing\web-platform\manifestupdate.py", line 146, in run
    manifests = load_and_update(
  File "m:\src\testing\web-platform\manifestupdate.py", line 244, in load_and_update
    m = manifest.manifest.load_and_update(
  File "M:\src\testing\web-platform\tests\tools\manifest\manifest.py", line 423, in load_and_update
    tree = vcs.get_tree(tests_root, manifest, manifest_path, cache_root,
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 43, in get_tree
    tree = FileSystem(tests_root,
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 105, in __init__
    self.hash_cache = git.hash_cache()
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 79, in hash_cache
    local_changes = self._local_changes()
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 63, in _local_changes
    data = self.git(*cmd)
  File "M:\src\testing\web-platform\tests\tools\manifest\utils.py", line 60, in gitfunc
    return subprocess.check_output(full_cmd, cwd=path, stderr=subprocess.STDOUT).decode('utf8')
  File "C:\mozilla-build\python3\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\mozilla-build\python3\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,

Sentry event ID: d21bae9d448945b896835a9f832506b2

which git returns /c/Program Files/Git/cmd/git and the git version is 2.35.1.windows.2, I use MozillaBuild 4.0 on Win11, I run start-shell.bat in the terminal in vscode, which ssh returns /c/Windows/System32/OpenSSH/ssh and ssh-agent runs as a service. When I upgrade MozillaBuild to 4.0, I cleaned up my environment so that I could break something in my environment...

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 1

•

3 years ago

FYI: I can run wpt with --no-manifest-update, but it seems that it cannot run new tests.

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 2

•

3 years ago

My git configurations are:

gitconfig in the installed directory:

[diff "astextplain"]
	textconv = astextplain
[filter "lfs"]
	clean = git-lfs clean -- %f
	smudge = git-lfs smudge -- %f
	process = git-lfs filter-process
	required = true
[http]
	sslBackend = openssl
	sslCAInfo = C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
[core]
	autocrlf = false
	fscache = true
	symlinks = false
[pull]
	rebase = false
[credential]
	helper = manager-core
[credential "https://dev.azure.com"]
	useHttpPath = true
[init]
	defaultBranch = master

.gitconfig in the home directory:

[winUpdater]
	recentlySeenVersion = 2.25.0.windows.1
[core]
	editor = \"C:\\Users\\toybox\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\code\" --wait

Mitchell Hentges [:mhentges] 🦀

Comment 3

•

3 years ago

Interesting, I think the next step is to see how that git command is failing.
Can you add a print(path) to M:\src\testing\web-platform\tests\tools\manifest\utils.py, then cd to that path in your terminal and run git diff-index --relative --no-renames --name-only -z HEAD?

Flags: needinfo?(masayuki)

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 4

•

3 years ago

Thanks. It's called with M:\src\testing\web-platform\tests, the I got this:

$ git diff-index --relative --no-renames --name-only -z HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

And I got this...

$ git log
fatal: your current branch 'master' does not have any commits yet

Flags: needinfo?(masayuki) → needinfo?(mhentges)

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 5

•

3 years ago

Both ./mach wpt --manifest-update and ./mach wpt --manifest-download fail so that I have no idea how to initialize my WPT environment with ./mach.

Mitchell Hentges [:mhentges] 🦀

Comment 6

•

3 years ago

Sending to the WPT crew, as they'll have a better idea what's happening here.

Component: General → web-platform-tests

Flags: needinfo?(mhentges)

Product: Firefox Build System → Testing

James Graham [:jgraham]

Comment 7

•

3 years ago

So the short term workaround here is to use mach wpt-manifest-update --rebuild, which won't try to update an existing manifest. The tradeoff is it's slow.

In terms of the actual problem, the issue seems to be that we end up running the code at https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/manifest/utils.py#54-74 to check if we're in a git repo. If we are we assume that we can use git to give us the hashes of locally changed files. But we don't actually depend on that; there's another path for when we're not in a git checkout.

So: I think what's broken in this case is that git rev-parse --show-toplevel is returning a success exit status, but we're not actually in a git repo. It would be interesting to know what that command is returning.

In terms of better workarounds, we could put the git command at https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/manifest/vcs.py#80 inside a try/except and just return an empty cache if it fails.

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 8

•

3 years ago

(In reply to James Graham [:jgraham] from comment #7)

So: I think what's broken in this case is that git rev-parse --show-toplevel is returning a success exit status, but we're not actually in a git repo. It would be interesting to know what that command is returning.

M:/src is returned (run in /m/src/testing/web-platform/tests).

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 9

•

3 years ago

(In reply to James Graham [:jgraham] from comment #7)

So the short term workaround here is to use mach wpt-manifest-update --rebuild, which won't try to update an existing manifest. The tradeoff is it's slow.

No, it does not work:

$ ./mach wpt-manifest-update --rebuild
created virtual environment CPython3.9.10.final.0-64 in 92ms
  creator CPython3Windows(dest=M:\fx64-dbg\_virtualenvs\wpt, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting tox==3.12.1
  Using cached tox-3.12.1-py2.py3-none-any.whl (79 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.6.0-py3-none-any.whl (10.0 kB)
Requirement already satisfied: setuptools>=30.0.0 in m:\src\third_party\python\setuptools (from tox==3.12.1) (51.2.0)
Collecting virtualenv>=14.0.0
  Downloading virtualenv-20.14.1-py2.py3-none-any.whl (8.8 MB)
     |████████████████████████████████| 8.8 MB 6.8 MB/s
Collecting toml>=0.9.4
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pluggy<1,>=0.3.0
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: six<2,>=1.0.0 in m:\src\third_party\python\six (from tox==3.12.1) (1.13.0)
Collecting py<2,>=1.4.17
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB)
Collecting platformdirs<3,>=2
  Using cached platformdirs-2.5.1-py3-none-any.whl (14 kB)
Installing collected packages: platformdirs, filelock, distlib, virtualenv, toml, py, pluggy, tox
Successfully installed distlib-0.3.4 filelock-3.6.0 platformdirs-2.5.1 pluggy-0.13.1 py-1.11.0 toml-0.10.2 tox-3.12.1 virtualenv-20.14.1
Collecting ujson==4.0.2
  Using cached ujson-4.0.2-cp39-cp39-win_amd64.whl (44 kB)
Installing collected packages: ujson
Successfully installed ujson-4.0.2
 0:00.01 WARNING The wpt manifest is now automatically updated, so running this command is usually unnecessary
 0:00.01 INFO Creating config file M:/fx64-dbg\_tests\web-platform\wptrunner.local.ini
 0:00.01 INFO Creating directory M:\fx64-dbg\_tests\web-platform\meta
 0:00.01 INFO Creating directory M:\fx64-dbg\_tests\web-platform\mozilla\meta
Error running mach:

    ['wpt-manifest-update', '--rebuild']
<snip>
subprocess.CalledProcessError: Command '['git', 'diff-index', '--relative', '--no-renames', '--name-only', '-z', 'HEAD']' returned non-zero exit status 128.

  File "M:\src\testing\web-platform\mach_commands.py", line 572, in wpt_manifest_update
    return 0 if wpt_runner.update_manifest(logger, **params) else 1
  File "m:\src\testing\web-platform\mach_commands_base.py", line 65, in update_manifest
    return manifestupdate.run(
  File "m:\src\testing\web-platform\manifestupdate.py", line 146, in run
    manifests = load_and_update(
  File "m:\src\testing\web-platform\manifestupdate.py", line 244, in load_and_update
    m = manifest.manifest.load_and_update(
  File "M:\src\testing\web-platform\tests\tools\manifest\manifest.py", line 423, in load_and_update
    tree = vcs.get_tree(tests_root, manifest, manifest_path, cache_root,
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 43, in get_tree
    tree = FileSystem(tests_root,
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 105, in __init__
    self.hash_cache = git.hash_cache()
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 79, in hash_cache
    local_changes = self._local_changes()
  File "M:\src\testing\web-platform\tests\tools\manifest\vcs.py", line 63, in _local_changes
    data = self.git(*cmd)
  File "M:\src\testing\web-platform\tests\tools\manifest\utils.py", line 60, in gitfunc
    return subprocess.check_output(full_cmd, cwd=path, stderr=subprocess.STDOUT).decode('utf8')
  File "C:\mozilla-build\python3\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\mozilla-build\python3\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,

Sentry event ID: fa23fe25d6094f668d965c9bc78df610

Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sick)

Reporter

Comment 10

•

3 years ago

Oh, after I rename .git in m:/src, it works. I manage my source tree with Mercurial, so I have no idea who created the .git directory...

BugBot [:suhaib / :marco/ :calixte]

Comment 11

•

3 years ago

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

James Graham [:jgraham]

Comment 12

•

3 years ago

We could handle this situation better, but it's an unusual setup to have the hg repo inside a git repo.

Severity: -- → S3

Flags: needinfo?(james)

Priority: -- → P3

Bugzilla

./mach wpt fails with that git returned non-zero exit status 128 on Windows

Categories

(Testing :: web-platform-tests, defect, P3)

Tracking

(Not tracked)

People

(Reporter: masayuki, Unassigned)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12