1511594 - hg format-source fails with "mach exited with status 1"

Status: VERIFIED FIXED

(Developer Infrastructure :: Lint and Formatting, defect)

Description

[Masatoshi Kimura [:emk]]

When I tried to rebase with the following command:

  hg rebase -k -b . -d 138f65958545 --debug

I got the follwing error:

 dom/ipc/TabParent.cpp: versions differ -> m (premerge)
Files to be: dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp
running e:\m\mozilla-unified\mach clang-format --assume-filename $HG_FILENAME -p c:\users\kimu\appdata\local\temp\tmpet53ir.cpp
abort: clang-format: mach exited with status 1

Do I have to manually resolve the conflict? It is really painful.

Comment 1

[Andi [:andi]]

Thanks for running this id debug mode!
Please copy 'dom/ipc/TabParent.cpp' from the base revision to the destination revision, overwrite it and run:

>>mach clang-format --assume-filename dom/ipc/TabParent.cpp -p dom/ipc/TabParent.cpp

Please copy and paste here the output.

Also do did you run 'mach bootstrap', do you have in ~/.mozbuild/clang-tools/ ?

Comment 2

[Masatoshi Kimura [:emk]]

Attachment: output from `mach clang-format --assume-filename dom/ipc/TabParent.cpp -p dom/ipc/TabParent.cpp`

(In reply to Andi-Bogdan Postelnicu [:andi] from comment #1)
> Please copy 'dom/ipc/TabParent.cpp' from the base revision to the
> destination revision, overwrite it and run:
> 
> >>mach clang-format --assume-filename dom/ipc/TabParent.cpp -p dom/ipc/TabParent.cpp
> 
> Please copy and paste here the output.

I attached the output because it was large.

> Also do did you run 'mach bootstrap', do you have in
> ~/.mozbuild/clang-tools/ ?

Yes, I re-ran `./mach bootstrap` just in case. I also confirmed that I have `~/.mozbuild/clang-tools/` folder. By the way, my `.mozbuild` is an NTFS junction to another drive because my system drive is a small capacity SSD. Is it relevant?

Comment 3

[Andi [:andi]]

Can you upload a remote bookmark with the changeset that needs to be rebased?

Comment 4

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/try/rev/2d4d022a4a7251bf6598f23ba2f0619676d0f656

If this is not what you want, please teach me a command.

Comment 5

[Andi [:andi]]

Are you sure that’s the revision since I don’t see the problematic cpp inside it?

Comment 6

[Masatoshi Kimura [:emk]]

I modified format-source to see error messages from mach:
--- a/hgext/format-source/__init__.py
+++ b/hgext/format-source/__init__.py
@@ -235,17 +235,21 @@ def run_tools(ui, root, tool, cmd, filep
     ui.debug('running %s\n' % format_cmd)
     ui.pushbuffer(subproc=True)
     try:
         ui.system(format_cmd,
                   environ=env,
                   cwd=root,
                   onerr=error.Abort,
                   errprefix=tool)
-    finally:
+    except:
+        newcontent = ui.popbuffer()
+        ui.debug('result %s\n' % newcontent)
+        raise
+    else:
         newcontent = ui.popbuffer()
     return newcontent

 def touched(repo, old_ctx, new_ctx, paths):
     matcher = rootedmatch(repo, new_ctx, paths)
     if any(path in new_ctx for path in paths):
         status = old_ctx.status(other=new_ctx, match=matcher)
         return bool(status.modified or status.added)

Result:
 dom/ipc/TabParent.cpp: versions differ -> m (premerge)
Files to be: dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp
running e:\m\mozilla-unified\mach clang-format --assume-filename $HG_FILENAME -p c:\users\kimu\appdata\local\temp\tmpyuyb9r.cpp
result 'e:\m\mozilla-unified\mach' は、内部コマンドまたは外部コマンド、
操作可能なプログラムまたはバッチ ファイルとして認識されていません。
(English: 'e:\m\mozilla-unified\mach' is not recognized as an internal or external command,
operable program or batch file.)

I added following settings to ~/.hgrc:
[format-source]
clang-format = /e/m/mozilla-unified/mach clang-format --assume-filename $HG_FILENAME -p
clang-format:configpaths = .clang-format, .clang-format-ignore
clang-format:fileext = .cpp, .c, .h

Then I got the following error:
 dom/ipc/TabParent.cpp: versions differ -> m (premerge)
Files to be: dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp dom/ipc/TabParent.cpp
running /e/m/mozilla-unified/mach clang-format --assume-filename $HG_FILENAME -p c:\users\kimu\appdata\local\temp\tmpeaopjl.cpp
result 指定されたパスが見つかりません。
(English: The system cannot find the file specified.)

Maybe this bug is Windows-specific and Windows-to-UNIX (mingw) path conversions are required?

Comment 7

[Masatoshi Kimura [:emk]]

On Windows, the command processor (cmd.exe) has no way to parse and execute shbang, so it could not run mach directly.
This setting worked for me:
[format-source]
clang-format = sh -c mach clang-format --assume-filename $HG_FILENAME -p
clang-format:configpaths = .clang-format, .clang-format-ignore
clang-format:fileext = .cpp, .c, .h

But I found another (probably plaform-independent) problem. I'll file another bug.

Comment 8

[Masatoshi Kimura [:emk]]

(In reply to Masatoshi Kimura [:emk] from comment #7)
> But I found another (probably plaform-independent) problem. I'll file
> another bug.

Filed bug 1511668.

Comment 9

[Andi [:andi]]

(In reply to undefined from comment #undefined)
> 
Don’t you run from the build environment? Are you running from cmd directly?

Comment 10

[Masatoshi Kimura [:emk]]

(In reply to Andi-Bogdan Postelnicu [:andi] from comment #9)
> (In reply to undefined from comment #undefined)
> > 
> Don’t you run from the build environment? Are you running from cmd directly?

I run `hg rebase` from the msys shell. But apparently ui.system() invokes the command via cmd on Windows. I have no control over that.

Comment 11

[Andi [:andi]]

This is very strange, is your setup custom? I'm asking this because other engineers have used this on windows without any issues.

Comment 12

[Masatoshi Kimura [:emk]]

As I already said, I installed files to D: as much as possible.
My objdir is on an exFAT drive.
I did not enalbe fsmonitor because it only times out on my environment.

Comment 13

[Jan Henning [:JanH]]

I have the same problem and I'm using the normal MozillaBuild 3.2 build shell (start-shell.bat).

Comment 14

[Andi [:andi]]

Did the fix from :emk work?

Comment 15

[Jan Henning [:JanH]]

I no longer get the clang-format error, but instead some of my changesets were dropped (which I guess could be bug 1511668).

Comment 16

[Jan Henning [:JanH]]

Actually even worse - in my case it dropped *all* changes to .cpp/.h files.

Comment 17

[Andi [:andi]]

Can you poște onto the new bug a log? Does it have the same behavior on other platforms?

Comment 18

[Masatoshi Kimura [:emk]]

(In reply to Andi-Bogdan Postelnicu [:andi] from comment #11)
> I'm asking this because other engineers have used this on windows without any issues.

Are they using native win32 mercurial? I heard Windows developers had to use WSL to deal with Phacricator problems.

Comment 19

[Andi [:andi]]

You might have a point with the WSL, thanks for mentioning this.
I will come up to you with a resolution, will try to reproduce this in a win vm.

Comment 20

[Jan Henning [:JanH]]

(In reply to Andi-Bogdan Postelnicu [:andi] from comment #17)
> Can you poște onto the new bug a log? Does it have the same behavior on
> other platforms?

Posted my patch set to bug 1511668 as well.

(In reply to Masatoshi Kimura [:emk] from comment #18)
> Are they using native win32 mercurial? I heard Windows developers had to use
> WSL to deal with Phacricator problems.

Good point, I'm not using WSL, either (and couldn't even if I wanted to, because my desktop is still on Windows 7).

Comment 22

[Andi [:andi]]

Attachment: extensions: Improve hg-format source with default parameters on windows (Bug 1511594) r?sheehan

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/42179015d473
extensions: Improve hg-format source with default parameters on windows r=sheehan

Comment 24

[Masatoshi Kimura [:emk]]

I got `Command Not Found: machclang-format` error with this patch. Proposed fix:
--- a/hgext/format-source/__init__.py
+++ b/hgext/format-source/__init__.py
@@ -90,17 +90,17 @@ if util.safehasattr(registrar, 'configit
 file_storage_path = '.hg-format-source'


 def return_default_clang_format(repo):
     arguments = ['clang-format', '--assume-filename', '$HG_FILENAME', '-p']

     # On windows we need this to call the command in a shell, see Bug 1511594
     if os.name == 'nt':
-        clang_format_cmd = 'sh mach' + ' '.join(arguments)
+        clang_format_cmd = 'sh mach ' + ' '.join(arguments)
     else:
         clang_format_cmd = os.path.join(repo.root, "mach") + ' '.join(arguments)

     clang_format_cfgpaths = ['.clang-format', '.clang-format-ignore']
     clang_fortmat_fileext = ('.cpp', '.c', '.cc', '.h')
     return clang_format_cmd, clang_format_cfgpaths, clang_fortmat_fileext

Comment 25

[Andi [:andi]]

Right good catch, otherwise did it work with your fix?

Comment 26

[Masatoshi Kimura [:emk]]

Yes, rebase succeeded (including all .cpp files) with my fix. Thanks for the great work!

Comment 27

[Andi [:andi]]

Attachment: extensions: hg-format add missing space between command and arguments (Bug 1511594) r?sheehan

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/efdd5165c82e
extensions: hg-format add missing space between command and arguments r=sheehan

Comment 29

[Jan Henning [:JanH]]

(In reply to Masatoshi Kimura [:emk] from comment #26)
> Yes, rebase succeeded (including all .cpp files) with my fix. Thanks for the
> great work!

Seconded, thanks as well.