Closed Bug 499097 Opened 15 years ago Closed 15 years ago

Mac try server fails to get the source from mozilla-1.9.1

Categories

(Mozilla Messaging Graveyard :: Release Engineering, defect)

All
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Usul, Unassigned)

References

()

Details

I've been trying to build debug for mac on the last three days all my try failed: 

tinderbox: tree: ThunderbirdTry
tinderbox: builddate: 1245317486
tinderbox: status: busted
tinderbox: build: Try server comm-central mac hg builder
tinderbox: errorparser: unix
tinderbox: binaryurl: 
tinderbox: logcompression: bzip2
tinderbox: logencoding: base64
tinderbox: END
======== BuildStep started ========
identify build
=== Output ===
TinderboxPrint: ludovic@mozillamessaging.com
TinderboxPrint: 1245317090
Comments: Debug build

=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
'rm -rf ...'
=== Output ===
rm -rf mozilla/
 in dir /Volumes/Build/sendchange-mac-hg/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['rm', '-rf', 'mozilla/']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
program finished with exit code 0
elapsedTime=28.438874
=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
'env | ...'
=== Output ===
/bin/sh -c env | sort
 in dir /Volumes/Build/sendchange-mac-hg/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['/bin/sh', '-c', 'env | sort']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
CCACHE_DIR=/Volumes/Build/ccache
COMMAND_MODE=unix2003
CVS_RSH=ssh
DISPLAY=/tmp/launch-3zp8YK/:0
HOME=/Users/cltbld
LANG=en_US.UTF-8
LOGNAME=cltbld
MAIL=/var/mail/cltbld
MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
PWD=/Volumes/Build/sendchange-mac-hg
PYTHONHOME=/tools/python
PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
SHELL=/bin/bash
SHLVL=3
SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
TERM=xterm-color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=240
TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
USER=cltbld
_=/usr/bin/env
__CF_USER_TEXT_ENCODING=0x1F5:0:0
program finished with exit code 0
elapsedTime=7.825637
=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
update
=== Output ===
starting mercurial operation
rm -rf /Volumes/Build/sendchange-mac-hg/mozilla/
 in dir /Volumes/Build/sendchange-mac-hg (timeout 3600 secs)
 watching logfiles {}
 argv: ['rm', '-rf', '/Volumes/Build/sendchange-mac-hg/mozilla/']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
elapsedTime=0.128422
/tools/python/bin/hg clone http://hg.mozilla.org/comm-central /Volumes/Build/sendchange-mac-hg/mozilla/
 in dir /Volumes/Build/sendchange-mac-hg (timeout 3600 secs)
 watching logfiles {}
 argv: ['/tools/python/bin/hg', 'clone', 'http://hg.mozilla.org/comm-central', '/Volumes/Build/sendchange-mac-hg/mozilla/']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
requesting all changes
adding changesets
adding manifests
adding file changes
added 2867 changesets with 16410 changes to 7083 files (+3 heads)
updating working directory
5632 files updated, 0 files merged, 0 files removed, 0 files unresolved
elapsedTime=124.368793
/tools/python/bin/hg identify
 in dir /Volumes/Build/sendchange-mac-hg/mozilla/
 watching logfiles {}
 argv: ['/tools/python/bin/hg', 'identify']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LC_ALL=C
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/:/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
elapsedTime=2.643391
program finished with exit code 0
=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
python client.py failed
=== Output ===
python client.py --skip-venkman --skip-chatzilla --hg-options=--verbose --time --traceback checkout
 in dir /Volumes/Build/sendchange-mac-hg/mozilla (timeout 14400 secs)
 watching logfiles {}
 argv: ['python', 'client.py', '--skip-venkman', '--skip-chatzilla', '--hg-options=--verbose --time --traceback', 'checkout']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-RnhTnm/Render
  CCACHE_DIR=/Volumes/Build/ccache
  COMMAND_MODE=unix2003
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-3zp8YK/:0
  HOME=/Users/cltbld
  LANG=en_US.UTF-8
  LOGNAME=cltbld
  MAIL=/var/mail/cltbld
  MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man
  PATH=/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/opt/local/libexec/ccache:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/Users/cltbld
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=2
  SSH_AUTH_SOCK=/tmp/launch-YJsM0Z/Listeners
  TERM=xterm-color
  TERM_PROGRAM=Apple_Terminal
  TERM_PROGRAM_VERSION=240
  TMPDIR=/var/folders/PH/PHjGUMPrGcyiLXjk3eHtVU+++TI/-Tmp-/
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: True
Executing command: ['hg', 'pull', '-R', './.', '-r', 'tip', '--verbose', '--time', '--traceback']
pulling from http://hg.mozilla.org/comm-central
searching for changes
no changes found
Time: real 0.217 secs (user 0.083+0.000 sys 0.067+0.000)
Executing command: ['hg', 'update', '-r', 'default', '-R', './.', '--verbose', '--time', '--traceback']
resolving manifests
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Time: real 1.083 secs (user 0.517+0.000 sys 0.400+0.000)
Updated to revision fb756d0170ea118e2936fc8942ab90cc49067a75.
Executing command: ['hg', 'clone', '--verbose', '--time', '--traceback', 'http://hg.mozilla.org/releases/mozilla-1.9.1/', './mozilla']
requesting all changes
adding changesets
adding manifests
transaction abort!
rollback completed
Traceback (most recent call last):
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 41, in _runcatch
    return _dispatch(ui, args)
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 372, in _dispatch
    return runcommand(lui, repo, cmd, fullargs, ui, options, d)
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 247, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 417, in _runcommand
    return checkargs()
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 377, in checkargs
    return cmdfunc()
  File "/tools/python/lib/python2.5/site-packages/mercurial/dispatch.py", line 371, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/tools/python/lib/python2.5/site-packages/mercurial/util.py", line 718, in check
    return func(*args, **kwargs)
  File "/tools/python/lib/python2.5/site-packages/mercurial/commands.py", line 603, in clone
    update=not opts.get('noupdate'))
  File "/tools/python/lib/python2.5/site-packages/mercurial/hg.py", line 215, in clone
    dest_repo.clone(src_repo, heads=revs, stream=stream)
  File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 2152, in clone
    return self.pull(remote, heads)
  File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 1495, in pull
    return self.addchangegroup(cg, 'pull', remote.url())
  File "/tools/python/lib/python2.5/site-packages/mercurial/localrepo.py", line 2031, in addchangegroup
    self.manifest.addgroup(chunkiter, revmap, trp)
  File "/tools/python/lib/python2.5/site-packages/mercurial/revlog.py", line 1192, in addgroup
    for chunk in revs:
  File "/tools/python/lib/python2.5/site-packages/mercurial/changegroup.py", line 31, in chunkiter
    c = getchunk(source)
  File "/tools/python/lib/python2.5/site-packages/mercurial/changegroup.py", line 25, in getchunk
    % (len(d), l - 4))
Abort: premature EOF reading chunk (got 20675 bytes, expected 52566)
abort: premature EOF reading chunk (got 20675 bytes, expected 52566)
Time: real 95.233 secs (user 41.450+0.000 sys 44.600+0.000)
Traceback (most recent call last):
  File "client.py", line 339, in <module>
    do_hg_pull('mozilla', options.mozilla_repo, options.hg, options.mozilla_rev)
  File "client.py", line 177, in do_hg_pull
    check_call_noisy([hg, 'clone'] + hgopts + [repository, fulldir])
  File "client.py", line 54, in check_call_noisy
    check_call(cmd, *args, **kwargs)
  File "/tools/python/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['hg', 'clone', '--verbose', '--time', '--traceback', 'http://hg.mozilla.org/releases/mozilla-1.9.1/', './mozilla']' returned non-zero exit status 255
program finished with exit code 1
elapsedTime=60.893359
=== Output ended ===
======== BuildStep ended ========
Summary: Mac try server fails to get the source from mozilla-central → Mac try server fails to get the source from mozilla-1.9.1
For some reason, the Mac OS X builders have been hit hardest with the 'hg clone' problems, which explains the problems you've seen.

I've just recently switched over the try builders to using our own hg mirror (for testing) and I've re-queued a bunch of failed builds. Over the last 24 hours, all hg clone operations succeded, so its looking like it's working around the problem we are having.

As a bonus, if you watch the waterfall, you'll see your failed builds have been rebuild and can be downloaded.
Since bug 493152 is implemented, try builders have been talking to our hg mirror.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.