1677186 - Broken pipe when pushing to try via ./mach try or ./mach try chooser on Mac

Status: RESOLVED WORKSFORME

(Firefox Build System :: Try, defect)

Description

[Emma Malysz]

I've recently been unable to push to try. I've tried running ./mach bootstrap and ./mach vcs-setup, but that doesn't seem to fix the issue.

Here's my full output

searching for changes
remote: packet_write_wait: Connection to 63.245.208.203 port 22: Broken pipe
temporary commit removed, repository restored
abort: unexpected response: empty string
Error running mach:

    ['try']

The details of the failure are as follows:

subprocess.CalledProcessError: Command '('/usr/local/bin/hg', 'push-to-try', '-m', 'Tasks automatically selected.\n\nPushed via `mach try auto`')' returned non-zero exit status 255.

  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/mach_commands.py", line 227, in try_default
    return self._mach_context.commands.dispatch(
  File "/Users/emmamalysz/Desktop/firefox-source/python/mach/mach/registrar.py", line 161, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/python/mach/mach/registrar.py", line 107, in _run_command_handler
    result = fn(**kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/mach_commands.py", line 365, in try_auto
    return self.run(**kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/mach_commands.py", line 197, in run
    return mod.run(**kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/selectors/auto.py", line 107, in run
    return push_to_try(
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/push.py", line 213, in push_to_try
    vcs.push_to_try(commit_message)
  File "/Users/emmamalysz/Desktop/firefox-source/python/mozversioncontrol/mozversioncontrol/__init__.py", line 482, in push_to_try
    subprocess.check_call(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

I tried disabling clang-format and js-format, but that didn't seem to fix the issue.

Any help would be much appreciated!

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

hg push-try-try is printing out the following:

remote: packet_write_wait: Connection to 63.245.208.203 port 22: Broken pipe
temporary commit removed, repository restored
** Unknown exception encountered with possibly-broken third-party extension clang-format
** which supports versions 5.3 of Mercurial.
** Please disable clang-format and try your action again.
** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System&component=Lint%20and%20Formatting
** Python 3.8.5 (default, Jul 21 2020, 10:42:08) [Clang 11.0.0 (clang-1100.0.33.17)]
** Mercurial Distributed SCM (version 5.5.2)
** Extensions loaded: absorb, histedit, purge, rebase, evolve, fsmonitor, blackbox, firefoxtree, push-to-try, clang-format, js-format

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Rebooting to Mac, let me see if I can reproduce this.
If you go to central with a clean checkout and you ./mach try ..., can you still reproduce the problem?

Comment 3

[Emma Malysz]

Attachment: Trace

```

Comment 4

[Emma Malysz]

Attachment: Traceback

```

Comment 5

[Emma Malysz]

Thanks for the help :mhentges!

At least for now, the problem seems to have gone away. Mitchell suggested updating mercurial to version 5.6 and pulling a fresh copy of central, and I cannot currently reproduce this issue. I also removed all of my untracked files

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

I'm not able to reproduce this, but I'm glad that it's gone away for some reason or another :)
One theory is that this is could've been an intermittent timeout issue. If so, if this happens again, adding the following to ~/.ssh/config may be a helpful workaround:

Host hg.mozilla.org
  ServerAliveInterval 30
  ServerAliveCountMax 5

Also, if this issue is encountered in the future by some internet denizen, and the above SSH config chunk fixes the issue, please mention so in this bug :)

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Note: we did some more investigation and tried with the following options:

Host hg.mozilla.org
  ServerAliveInterval 120
  ServerAliveCountMax 5
  IPQoS=throughput
  TCPKeepAlive no

but, this still didn't work.

Also worth noting:

• This could be reproduced with a raw hg push ssh://hg.mozilla.org/try -r <try revision>.
• This could be reproduced by SSH-ing into any server and pushing data quickly (e.g.: cat /dev/urandom, SFTP-ing a large file).
• If Emma used cell data (tethering) instead of wifi, the issue went away.
• In the last week, an ISP technician came by to update the local router. This could've changed a network config, thereby causing this failure.

The solution in this case is that the ISP will be notified that "when SSHing from the local network, pushing data quickly (e.g.: sending files) causes a 'broken pipe' error".