811195 - Intermittent red on OS X64 trunk build "compare current leak logs failed"

Status: RESOLVED DUPLICATE of bug 774844

(SeaMonkey :: Release Engineering, defect)

Description

[Edmund Wong (:ewong)]

We're seeing an intermittent red during the compare leakstats stage.
Currently happened on sea-mini-osx64-1 and -2.

On sea-mini-osx64-1:

======== BuildStep started ========
compare current leak logs failed
=== Output ===
objdir/mozilla/dist/bin/leakstats ../malloc.log
 in dir /builds/slave/comm-cen-trunk-osx64-dbg/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['objdir/mozilla/dist/bin/leakstats', '../malloc.log']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-afMbHR/Render
  CCACHE_BASEDIR=/builds/slave/comm-cen-trunk-osx64-dbg
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-4zQabB/org.x:0
  HOME=/Users/seabld
  LC_ALL=C
  LOGNAME=seabld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=objdir
  PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/comm-cen-trunk-osx64-dbg/build
  PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-JEv852/Listeners
  TMPDIR=/var/folders/bm/kc_3cyjj2v15yn2741pj8s6r0000gn/T/
  USER=seabld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  XPCOM_DEBUG_BREAK=stack-and-abort
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 using PTY: False
objdir/mozilla/dist/bin/leakstats starting at Mon Nov 12 19:07:40 2012
objdir/mozilla/dist/bin/leakstats: no callsite for 'F' (70)!
Unknown event type 0xffffff90
objdir/mozilla/dist/bin/leakstats: log file incomplete
program finished with exit code 1
elapsedTime=0.290109
Unable to parse leakstats output
=== Output ended ===

on sea-mini-osx64-2: (the first appearance)
objdir/mozilla/dist/bin/leakstats ../malloc.log
 in dir /builds/slave/comm-cen-trunk-osx64-dbg/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['objdir/mozilla/dist/bin/leakstats', '../malloc.log']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-ru5A1S/Render
  CCACHE_BASEDIR=/builds/slave/comm-cen-trunk-osx64-dbg
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-8zEbVw/org.x:0
  HOME=/Users/seabld
  LC_ALL=C
  LOGNAME=seabld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=objdir
  PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/comm-cen-trunk-osx64-dbg/build
  PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-0ki0sr/Listeners
  TMPDIR=/var/folders/bm/kc_3cyjj2v15yn2741pj8s6r0000gn/T/
  USER=seabld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  XPCOM_DEBUG_BREAK=stack-and-abort
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 using PTY: False
objdir/mozilla/dist/bin/leakstats starting at Sat Nov 10 16:08:58 2012
Unknown event type 0x0
objdir/mozilla/dist/bin/leakstats: log file incomplete
program finished with exit code 1
elapsedTime=0.389661

2nd Appearance:

objdir/mozilla/dist/bin/leakstats ../malloc.log
 in dir /builds/slave/comm-cen-trunk-osx64-dbg/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['objdir/mozilla/dist/bin/leakstats', '../malloc.log']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-UACXxG/Render
  CCACHE_BASEDIR=/builds/slave/comm-cen-trunk-osx64-dbg
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-CjWiXZ/org.x:0
  HOME=/Users/seabld
  LC_ALL=C
  LOGNAME=seabld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=objdir
  PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/comm-cen-trunk-osx64-dbg/build
  PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-l7mAAz/Listeners
  TMPDIR=/var/folders/bm/kc_3cyjj2v15yn2741pj8s6r0000gn/T/
  USER=seabld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  XPCOM_DEBUG_BREAK=stack-and-abort
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 using PTY: False
objdir/mozilla/dist/bin/leakstats starting at Sun Nov 11 20:07:44 2012
Unknown event type 0x0
objdir/mozilla/dist/bin/leakstats: log file incomplete
program finished with exit code 1
elapsedTime=0.331275

So there is either one weird issue that's doing 'random' stuff, or two issues on different machines or some combination of the two.

Comment 1

[Justin Wood (:Callek)]

To describe here, why I think 811199 might fix this, so that we are not confused in the future...

http://hg.mozilla.org/mozilla-central/diff/52b59d7cff7c/tools/trace-malloc/leakstats.c touched leakstats.c changing a struct.

Part 1 of that bug's patch added the DEFINE we are talking about in 811199

We certainly use NSPR in c-c code.

My theory is that a leak in c-c code that gets changed by that define, could be causing confusion with leakstats.c's code, and thus cause an actual problem. This is only a theory, and the patch there can only help even if it doesn't fix this.