Closed Bug 1677186 Opened 2 years ago Closed 2 years ago

Broken pipe when pushing to try via ./mach try or ./mach try chooser on Mac


(Firefox Build System :: Try, defect)



(Not tracked)



(Reporter: emmamalysz, Assigned: mhentges)


(Blocks 1 open bug)



(2 files)

I've recently been unable to push to try. I've tried running ./mach bootstrap and ./mach vcs-setup, but that doesn't seem to fix the issue.

Here's my full output

searching for changes
remote: packet_write_wait: Connection to port 22: Broken pipe
temporary commit removed, repository restored
abort: unexpected response: empty string
Error running mach:


The details of the failure are as follows:

subprocess.CalledProcessError: Command '('/usr/local/bin/hg', 'push-to-try', '-m', 'Tasks automatically selected.\n\nPushed via `mach try auto`')' returned non-zero exit status 255.

  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/", line 227, in try_default
    return self._mach_context.commands.dispatch(
  File "/Users/emmamalysz/Desktop/firefox-source/python/mach/mach/", line 161, in dispatch
    return self._run_command_handler(handler, context, **kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/python/mach/mach/", line 107, in _run_command_handler
    result = fn(**kwargs)
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/", line 365, in try_auto
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/", line 197, in run
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/selectors/", line 107, in run
    return push_to_try(
  File "/Users/emmamalysz/Desktop/firefox-source/tools/tryselect/", line 213, in push_to_try
  File "/Users/emmamalysz/Desktop/firefox-source/python/mozversioncontrol/mozversioncontrol/", line 482, in push_to_try
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

I tried disabling clang-format and js-format, but that didn't seem to fix the issue.

Any help would be much appreciated!

hg push-try-try is printing out the following:

remote: packet_write_wait: Connection to port 22: Broken pipe
temporary commit removed, repository restored
** Unknown exception encountered with possibly-broken third-party extension clang-format
** which supports versions 5.3 of Mercurial.
** Please disable clang-format and try your action again.
** If that fixes the bug please report it to
** Python 3.8.5 (default, Jul 21 2020, 10:42:08) [Clang 11.0.0 (clang-1100.0.33.17)]
** Mercurial Distributed SCM (version 5.5.2)
** Extensions loaded: absorb, histedit, purge, rebase, evolve, fsmonitor, blackbox, firefoxtree, push-to-try, clang-format, js-format
Assignee: nobody → mhentges

Rebooting to Mac, let me see if I can reproduce this.
If you go to central with a clean checkout and you ./mach try ..., can you still reproduce the problem?

Flags: needinfo?(emalysz)
Attached file Trace
Attached file Traceback

Thanks for the help :mhentges!

At least for now, the problem seems to have gone away. Mitchell suggested updating mercurial to version 5.6 and pulling a fresh copy of central, and I cannot currently reproduce this issue. I also removed all of my untracked files

Flags: needinfo?(emalysz)

I'm not able to reproduce this, but I'm glad that it's gone away for some reason or another :)
One theory is that this is could've been an intermittent timeout issue. If so, if this happens again, adding the following to ~/.ssh/config may be a helpful workaround:

  ServerAliveInterval 30
  ServerAliveCountMax 5

Also, if this issue is encountered in the future by some internet denizen, and the above SSH config chunk fixes the issue, please mention so in this bug :)

Closed: 2 years ago
Resolution: --- → WORKSFORME

Note: we did some more investigation and tried with the following options:

  ServerAliveInterval 120
  ServerAliveCountMax 5
  TCPKeepAlive no

but, this still didn't work.

Also worth noting:

  • This could be reproduced with a raw hg push ssh:// -r <try revision>.
  • This could be reproduced by SSH-ing into any server and pushing data quickly (e.g.: cat /dev/urandom, SFTP-ing a large file).
  • If Emma used cell data (tethering) instead of wifi, the issue went away.
  • In the last week, an ISP technician came by to update the local router. This could've changed a network config, thereby causing this failure.

The solution in this case is that the ISP will be notified that "when SSHing from the local network, pushing data quickly (e.g.: sending files) causes a 'broken pipe' error".

You need to log in before you can comment on or make changes to this bug.