1354356 - All the SeaMonkey builders (for all platforms) are busted at timing out with the checkout step

Status: RESOLVED WONTFIX

(SeaMonkey :: Release Engineering, defect)

Description

[Edmund Wong (:ewong)]

Every single tree, every single platform and builders are busted with
the following timeout:

python client.py checkout --skip-comm --mozilla-repo=https://hg.mozilla.org/releases/mozilla-release
 in dir /builds/slave/c-rel-lnx/build (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'client.py', 'checkout', '--skip-comm', '--mozilla-repo=https://hg.mozilla.org/releases/mozilla-release']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/seabld
  HOSTNAME=sea-hp-linux64-6.community.scl3.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=seabld
  MAIL=/var/spool/mail/seabld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/seabld/bin
  PWD=/builds/slave/c-rel-lnx/build
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=seabld
  _=/tools/buildbot/bin/python
 using PTY: False

command timed out: 3600 seconds without output running ['python', 'client.py', 'checkout', '--skip-comm', '--mozilla-repo=https://hg.mozilla.org/releases/mozilla-release'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3600.003580


When I log on to a random builder and do the same command in the same
build directory,  I get the clone.

So something is broken in our infra.

Comment 1

[Gregory Szorc [:gps]]

Since it is taking >1 hr to clone, you are likely running Mercurial 3.5 or older. Upgrading to a modern Mercurial client will fix the problem. And this should have been done ~1 year ago because 3.7.3 contains security fixes.

https://hg.mozilla.org/mozilla-central/rev/4bfa4d528d3b contains tooltool artifacts for .rpm and .deb packages, which you should be able to download and install. If you need a package for a specific distro, let me know and I can package one.

Comment 2

[Edmund Wong (:ewong)]

(In reply to Gregory Szorc [:gps] from comment #1)
> Since it is taking >1 hr to clone, you are likely running Mercurial 3.5 or
> older. Upgrading to a modern Mercurial client will fix the problem. And this
> should have been done ~1 year ago because 3.7.3 contains security fixes.

Yeah,  we're using 3.2.1 and never upgraded due to a plethora of backlogged
problems with our infra.

> 
> https://hg.mozilla.org/mozilla-central/rev/4bfa4d528d3b contains tooltool
> artifacts for .rpm and .deb packages, which you should be able to download
> and install. If you need a package for a specific distro, let me know and I
> can package one.

Would the CentOS 6.4(or maybe 5) use those tooltool packages?

If it's ok, could you create a macosx64 dmg file (OSX 10.7).

Thanks for the help there!

Comment 3

[Edmund Wong (:ewong)]

Callek gave me the 3.9.1 mercurial rpms and I had them installed on the linux
slaves.  That unhorked them.

Next up: OSX64.

Comment 4

[Gregory Szorc [:gps]]

I cannot easily produce a Mercurial DMG, especially one for 10.7.

I recommend installing the latest version of pip and wheel on OS X 10.7, obtaining the Mercurial source code, and running `pip wheel .` to produce a Python wheel which is effectively a binary distribution of Mercurial.

Of course, you'll need a modern version of pip to install that wheel. But this is likely easier than producing a DMG.

FWIW, Mercurial does offer a .pkg at https://www.mercurial-scm.org/downloads. I just don't know if it works with 10.7 or how easy/hard one would be to build for 10.7.

Comment 5

[Edmund Wong (:ewong)]

OSX64 slaves have been upgraded to 3.9.1 but we need to 
upgrade the current python version to whatever is > 2.7.1

Comment 6

[Gregory Szorc [:gps]]

If you upgrade Python, please make it 2.7.9+, ideally 2.7.13, which is the latest available. There are tons of bug fixes. But the real reason to get 2.7.9+ is its ssl module isn't horrible and it allows you to establish reasonable connection security from Python code. Before 2.7.9, there's a real risk that TLS connections won't actually be secure.

Comment 7

[Edmund Wong (:ewong)]

Thanks :gps! 

So Callek, what's the best/right way of installing 2.7.13 on all our builders?  

Same way as installing mercurial? (I suppose install the package in /tools/python27_13 and
point all future python paths to this?)

Of course, we'll need to build such a package for all platforms.

Any advice?

Comment 8

[Edmund Wong (:ewong)]

Made irrelevant with scl3 being shut down.