834761 - ASan Nightly builds failing "sendchange"

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Ed Morley [:emorley]]

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19127855&tree=Firefox

{
========= Started cleanup old symbols failed (results: 2, elapsed: 0 secs) (at 2013-01-25 03:15:48.909111) =========
find 20* -maxdepth 2 -mtime +7 -exec rm -rf {} \;
 in dir /builds/slave/m-cen-lnx64-dbg-asan-ntly/. (timeout 1200 secs)
 watching logfiles {}
 argv: find 20* -maxdepth 2 -mtime +7 -exec rm -rf {} \;
 environment:
  CCACHE_BASEDIR=/builds/slave/m-cen-lnx64-dbg-asan-ntly
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_HASHDIR=
  CCACHE_UMASK=002
  DISPLAY=:2
  G_BROKEN_FILENAMES=1
  HG_SHARE_BASE_DIR=/builds/hg-shared
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-ec2-389.build.aws-us-east-1.mozilla.com
  IS_NIGHTLY=yes
  LANG=en_US.UTF-8
  LC_ALL=C
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=obj-firefox
  MOZ_SYMBOLS_EXTRA_BUILDID=linux64-dbg-asan
  MOZ_UPDATE_CHANNEL=nightly
  PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin
  POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
  PWD=/builds/slave/m-cen-lnx64-dbg-asan-ntly
  SHELL=/bin/bash
  SHLVL=1
  SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com
  SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
  SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa
  SYMBOL_SERVER_USER=ffxbld
  TERM=linux
  TINDERBOX_OUTPUT=1
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
find: `20*': No such file or directory
program finished with exit code 1
elapsedTime=0.011775
========= Finished cleanup old symbols failed (results: 2, elapsed: 0 secs) (at 2013-01-25 03:15:49.112693) =========
}

and...

{
========= Started sendchange (results: 2, elapsed: 0 secs) (at 2013-01-25 04:10:12.910240) =========
[Failure instance: Traceback: <type 'exceptions.KeyError'>: 'testsUrl'
/builds/buildbot/build1/lib/python2.6/site-packages/twisted/internet/defer.py:260:addCallback
/builds/buildbot/build1/lib/python2.6/site-packages/twisted/internet/defer.py:249:addCallbacks
/builds/buildbot/build1/lib/python2.6/site-packages/twisted/internet/defer.py:441:_runCallbacks
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/buildstep.py:769:_startStep_2
--- <exception caught here> ---
/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/steps/misc.py:109:start
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/properties.py:108:render
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/properties.py:106:render
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/properties.py:194:render
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/properties.py:169:__getitem__
/builds/buildbot/build1/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/process/properties.py:50:__getitem__
]
========= Finished sendchange (results: 2, elapsed: 0 secs) (at 2013-01-25 04:10:12.914640) =========
}

Comment 1

[[e-mail] (:bhearsum)]

symbol cleanup is non-fatal, looks like sendchange is the only thing causing burning

Comment 2

[Christian Holler (:decoder)]

Is there anything we can do to help push this forward? It seems to me that this is the only non-test failure that turns builds orange, and there are also no perma-orange tests left in the build process.

Comment 3

[Chris AtLee [:catlee]]

Attachment: enable packageTests for asan builds

Comment 4

[Rail Aliiev [:rail]]

in production

Comment 5

[Christian Holler (:decoder)]

First green ASan build, thanks guys!