1598702 - `hg rebase` takes enormous amount of time (hours) for large patches modifying also .js files, likely because of formatting

Status: RESOLVED INVALID

(Developer Infrastructure :: Lint and Formatting, defect, P3)

Description

[Honza Bambas (:mayhemer)]

Windows mingw32, current m-c, all up to date.

Filing in this component because I suspect the formatting hooks cause this. If you pull https://phabricator.services.mozilla.com/D43623 and do hg rebase of that patch on a newer changeset, it takes a very long time, like if each file being rebased started a processing of the whole tree or something.

This started a long ago after we added the formatting hooks.

I can see c:\mozilla\mozilla-build\python\python.exe c:/Mozilla/mozilla-build/python/Scripts/hg rebase -b backtrack -d 1e1617c67238dfb685fb3d07bf1793232c4469fa running all the time with 100% CPU use.

my .hgrc:

[extensions]
hgext.graphlog = 
rebase = 
progress = 
histedit = 
mq = 
push-to-try = c:\Users\user\.mozbuild\version-control-tools\hgext\push-to-try
absorb = 
evolve = c:\Users\user\.mozbuild/evolve/hgext3rd/evolve
clang-format = c:\Users\user\.mozbuild\version-control-tools\hgext\clang-format
shelve = 
js-format = c:\Users\user\.mozbuild\version-control-tools\hgext\js-format
format-source = c:\Users\user\.mozbuild\version-control-tools\hgext\format-source
purge =

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Honza Bambas (:mayhemer)]

Ahal, this has a serious impact on my work efficiency. It takes long minutes (on a fast machine) to finish e.g. hg next --evolve.

What info would help you to figure this out?

Comment 3

[Sylvestre Ledru [:Sylvestre]]

For clang-format hooks, we don't reformat in case of rebase:
https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgext/clang-format/__init__.py#l68

Comment 4

[Honza Bambas (:mayhemer)]

(In reply to Sylvestre Ledru [:Sylvestre] from comment #3)

For clang-format hooks, we don't reformat in case of rebase:
https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgext/clang-format/__init__.py#l68

The slowdown happens during merge. I can see the python process to eat (single) CPU at 100%. It's hard to find out what is happening exactly. Can you propose some way to narrow this down?

Comment 5

[Andrew Halberstadt [:ahal]]

I'm not very involved with the formatting side of things, Sylvestre (or someone on his team) or possibly :sheehan would be best positioned to help you out.

That said, maybe a log with hg --debug <command> would help.

Comment 6

[Honza Bambas (:mayhemer)]

format-source is definitely involved and likely the cause. I took few samples during the long-hanging phase using py-spy and they all looked like this:

__call__ (mercurial\match.py:307)
    matches (mercurial\manifest.py:531)
    diff (mercurial\manifest.py:550)
    _buildstatus (mercurial\context.py:121)
    _buildstatus (mercurial\context.py:1631)
    status (mercurial\context.py:350)
    touched (format-source\__init__.py:311)
    formatted (format-source\__init__.py:343)
    allformatted (format-source\__init__.py:357)
    _update_filemerge_content (format-source\__init__.py:462)
    wrap_filemerge44 (format-source\__init__.py:445)
    premerge (mercurial\filemerge.py:1028)
    _resolve (mercurial\merge.py:596)
    preresolve (mercurial\merge.py:643)
    applyupdates (mercurial\merge.py:1760)
    update (mercurial\merge.py:2186)
    wrap_update (format-source\__init__.py:479)
    wrapupdate (hgext\fsmonitor\__init__.py:748)
    closure (mercurial\extensions.py:439)
    graft (mercurial\merge.py:2234)
    _evolvemerge (hgext3rd\evolve\evolvecmd.py:1055)
    relocate (hgext3rd\evolve\evolvecmd.py:972)
    _solveunstable (hgext3rd\evolve\evolvecmd.py:183)
    _solveone (hgext3rd\evolve\evolvecmd.py:80)
    _nextevolve (hgext3rd\evolve\__init__.py:1172)
    cmdnext (hgext3rd\evolve\__init__.py:1154)
    check (mercurial\util.py:1680)
    mqcommand (hgext\mq.py:3631)
    check (mercurial\util.py:1680)
    check (mercurial\util.py:1680)
    <lambda> (mercurial\dispatch.py:1018)
    _runcommand (mercurial\dispatch.py:1030)
    runcommand (mercurial\dispatch.py:756)
    _dispatch (mercurial\dispatch.py:1021)
    _runcatchfunc (mercurial\dispatch.py:367)
    callcatch (mercurial\scmutil.py:167)
    _callcatch (mercurial\dispatch.py:384)
    _runcatch (mercurial\dispatch.py:376)
    dispatch (mercurial\dispatch.py:225)
    run (mercurial\dispatch.py:99)
    <module> (hg:43)

should I disable the format-source extension?

Comment 7

[Connor Sheehan [:sheehan]]

Yes, format-source should be disabled. Running ./mach vcs-setup should disable this for you. If you've been running mach vcs-setup and it hasn't been disabling the extension, there could be a bug in the disabling code.

Comment 8

[Honza Bambas (:mayhemer)]

(In reply to Connor Sheehan [:sheehan] from comment #7)

Yes, format-source should be disabled. Running ./mach vcs-setup should disable this for you. If you've been running mach vcs-setup and it hasn't been disabling the extension, there could be a bug in the disabling code.

Thanks! I'm only letting vcs-update do update of the binaries and extensions, but I never let it touch my hgrc.
This is then INVALID.