1739613 - configwizard fails on Windows if a line in .hgrc has both a quoted argument and other arguments

Status: REOPENED

(Developer Services :: Mercurial: configwizard, defect, P3)

Description

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

subprocess.CalledProcessError: Command '['c:/Users/toybox/.mozbuild\_virtualenvs\mach\Scripts\python.exe', 'm:\src\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']' returned non-zero exit status 1.

File "m:\src\python/mozboot/mozboot/mach_commands.py", line 42, in bootstrap
bootstrapper.bootstrap(command_context.settings)
File "m:\src\python\mozboot\mozboot\bootstrap.py", line 377, in bootstrap
state_dir, checkout_root, application
File "m:\src\python\mozboot\mozboot\bootstrap.py", line 224, in maybe_install_private_packages_or_exit
self.instance.ensure_node_packages(state_dir, checkout_root)
File "m:\src\python\mozboot\mozboot\mozillabuild.py", line 245, in ensure_node_packages
self.install_toolchain_artifact(state_dir, checkout_root, node_artifact)
File "m:\src\python\mozboot\mozboot\base.py", line 428, in install_toolchain_artifact
subprocess.check_call(cmd, cwd=state_dir)
File "c:\mozilla-build\python3\lib\subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)

Sentry event ID: 9712d8cff2214e5ea90185419224ce64
Sentry is attempting to send 0 pending error messages

before this error, I got:

** unknown exception encountered, please report by visiting
** https://mercurial-scm.org/wiki/BugTracker
** Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)]
** Mercurial Distributed SCM (version 5.3+3-847e582f3cc9)
** Extensions loaded: strip, mq, firefoxtree, push-to-try, fsmonitor, blackbox, absorb, histedit, rebase, amend, evolve, clang-format, js-format, show, configwizard
Traceback (most recent call last):
File "c:\mozilla-build\python\Scripts\hg", line 36, in <module>
dispatch.run()
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 110, in run
status = dispatch(req)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 253, in dispatch
ret = _runcatch(req) or 0
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 427, in _runcatch
return _callcatch(ui, _runcatchfunc)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 436, in _callcatch
return scmutil.callcatch(ui, func)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\scmutil.py", line 152, in callcatch
return func()
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 417, in _runcatchfunc
return _dispatch(req)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 1178, in _dispatch
lui, repo, cmd, fullargs, ui, options, d, cmdpats, cmdoptions
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 865, in runcommand
ret = runcommand(ui, options, cmd, d)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 1189, in runcommand
return cmdfunc()
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\dispatch.py", line 1175, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\util.py", line 1854, in check
return func(*args, **kwargs)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\util.py", line 1854, in check
return func(*args, **kwargs)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\hgext\mq.py", line 4218, in mqcommand
return orig(ui, repo, *args, **kwargs)
File "C:\Users\toybox\AppData\Roaming\Python\Python27\site-packages\mercurial\util.py", line 1854, in check
return func(*args, **kwargs)
File "c:/Users/toybox/.mozbuild/version-control-tools/hgext\configwizard_init.py", line 525, in configwizard
cw = configobjwrapper(path)
File "c:/Users/toybox/.mozbuild/version-control-tools/hgext\configwizard_init.py", line 1347, in init
write_empty_values=True, list_values=False)
File "c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1207, in init
self._load(infile, configspec)
File "c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1304, in _load
raise error
configobj.ParseError: Parse error in value at line 3.
Error running mach:

['artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']

I did ./mach bootstrap (choosing "2" (Firefox for Desktop) and choose 'Y' for 'Would you like to run a configuration wizard to ensure Mercurial is optimally configured?', but I got this error.

I'm not sure what I should provide my environment's information...

Additionally, I cannot build Firefox with the following error:

0:06.64 checking for nasm... not found
0:06.64 ERROR: Nasm is required to build with AV1, FFVPX, JPEG and VPX, but it was not found.
Error running mach:

['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['c:\Users\toybox\.mozbuild\_virtualenvs\mach\Scripts\python.exe', 'm:/src\configure.py']

File "m:\src\python/mozbuild/mozbuild/build_commands.py", line 229, in build
append_env=append_env,
File "m:\src\python\mozbuild\mozbuild\controller\building.py", line 1149, in build
append_env=append_env,
File "m:\src\python\mozbuild\mozbuild\controller\building.py", line 1529, in configure
append_env=append_env,
File "m:\src\python\mozbuild\mozbuild\base.py", line 853, in _run_command_in_objdir
return self.run_process(cwd=self.topobjdir, **args)
File "m:\src\python\mach\mach\mixin\process.py", line 175, in run_process
"Process executed with non-0 exit code %d: %s" % (status, args)

Sentry event ID: d1d631d84f9e4d59ab78be3be83e2787
Sentry is attempting to send 0 pending error messages

Comment 1

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Even if I choose n for Would you like to run a configuration wizard to ensure Mercurial is optimally configured? (Yn), I got same error.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

File "c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1207, in init
self._load(infile, configspec)
File "c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1304, in _load
raise error
configobj.ParseError: Parse error in value at line 3.

Looks like an issue parsing a config file in "ini" format. Probably your ~/.hgrc or .hg/hgrc.
Here's the steps I'd recommend:

1. Try moving/renaming your ~/.hgrc, then see if it behaves properly. If it does, then your ~/.hgrc has invalid contents.
2. If the issue isn't in the ~/.hgrc, then add a print(path) to c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1304, in _load_, so you can find out which file has invalid contents.

Let me know how that works :)

Comment 3

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Indeed. This line was error:

editor = "C:\Users\toybox\AppData\Local\Programs\Microsoft VS Code\Code.exe" -w

but rewriting it to:

editor = "C:\Users\toybox\AppData\Local\Programs\Microsoft VS Code\Code.exe -w"

And then, I hit bug 1739520, but after pulling the latest m-c, I got succeeded! Thank you!

Comment 4

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Hmm, but rewriting it as the new one causes Mercurial fails to launch VSCode. So I think the former line should be handled without error.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

I think the next steps to diagnose this are to find out which two HG commands are in conflict here: it sounds like one of them expects the -w outside the quotes, and the other one needs -w to be part of the full editor string.

I'm guessing that the one that should cause the launch of VSCode is hg commit.
So, I'd like to know which hg command is failing as part of ./mach artifact toolchain --bootstrap --from-build win64-node. Can you share that command's stack trace?

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

NI-ing Connor here, as perhaps we're parsing the ~/.hgrc in a different way than HG does internally:

File "c:/Users/toybox/.mozbuild/version-control-tools/hgext\configwizard_init.py", line 1347, in init
write_empty_values=True, list_values=False)
File "c:\Users\toybox.mozbuild\version-control-tools\pylib\configobj\configobj.py", line 1207, in init
self._load(infile, configspec)

(The failing load is happening in version-control-tools)

Connor, do you have any ideas/guesses here?

Comment 7

[Connor Sheehan [:sheehan]]

Yes, configwizard is parsing the hgrc in a different way than hg does internally. hg uses it's own config file parser which is much simpler and doesn't support writing the in-memory config back to a file. Since configwizard reads the hgrc files from disk and makes changes to it (ie enabling extensions) we need to be able to write back to disk, so we have a copy of configobj vendored into version-control-tools.

:masayuki, could you try the following:

editor = "'C:\Users\toybox\AppData\Local\Programs\Microsoft VS Code\Code.exe' -w"

or if that doesn't work,

editor = '"C:\Users\toybox\AppData\Local\Programs\Microsoft VS Code\Code.exe" -w'

Perhaps having the full value in quotes, but then the path with spaces quoted within the value will make both parsers happy.

Comment 8

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Unfortunately, both do not work with hg amend -e. The error message claims that it's not valid path. And also only "" is allowed to wrap the path. Using '' causes same error.

Comment 9

[Connor Sheehan [:sheehan]]

I made an attempt at re-writing the configwizard parser to use a dict instead of a list of lines as it's infile passed to ConfigObj, but sadly this doesn't preserve comments, %include's, and doesn't preserve the order of the file in general, which makes it unusable for us.

I also tried escaping all the quotes and \'s but couldn't get my test reproduction to pass, so I don't think that's going to work.

Unfortunately I don't think I'll have any more time to throw at this in the near future.

:masayuki, could you try creating a link/Windows shortcut (I think that's the right terminology?) to VS Code in a non-spaced path and then see if passing the value as editor = "C:\Users\toybox\Desktop\vscode.lnk -w" works as expected?

Comment 10

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Thanks. Creating a shotcut works for me, but the step is:

1. Create a shortcut for editor
2. Open property of the shortcut and add command line arguments after the path (outside "")
3. Make editor in .hgrc has the path to the shortcut

I think that leaving open this bug may help other new developers who hit this bug.