590224 - OS X64 debug intermittent failure "alive test failed" on Try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Shawn Wilsher :sdwilsh]

This might be try only, but I've noticed this on 3/3 pushes of my SQLite upgrade.  I started doing some digging, and I've noticed that I'm not the only one seeing it.  We only seem to be seeing it on the try server too (cc'ing releng).

Logs look like this:
======== BuildStep started ========
alive test failed
=== Output ===
python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log
 in dir /builds/slave/tryserver-macosx64-debug/build/obj-firefox/_leaktest (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'leaktest.py', '--', '--trace-malloc', 'malloc.log', '--shutdown-leaks=sdleak.log']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-3wQKYa/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-AKcsxk/:0
  HOME=/Users/cltbld
  LC_ALL=C
  LOGNAME=cltbld
  MINIDUMP_STACKWALK=/builds/slave/tryserver-macosx64-debug/tools/breakpad/osx64/minidump_stackwalk
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=obj-firefox
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/tryserver-macosx64-debug/build/obj-firefox/_leaktest
  PYTHONPATH=/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-Bo9ae0/Listeners
  TMPDIR=/var/folders/7I/7I253dv+HLesSBUBGCX08E+++TM/-Tmp-/
  USER=cltbld
  XPCOM_DEBUG_BREAK=stack-and-abort
 closing stdin
 using PTY: True
INFO | automation.py | Application pid: 95046
WARNING: NS_ENSURE_TRUE(zipFile) failed: file /builds/slave/tryserver-macosx64-debug/build/modules/libjar/nsJAR.cpp, line 165
WARNING: Ignoring duplicate observer.: file /builds/slave/tryserver-macosx64-debug/build/modules/libpref/src/nsPrefBranch.cpp, line 724
WARNING: couldn't get schema version!: file /builds/slave/tryserver-macosx64-debug/build/extensions/cookie/nsPermissionManager.cpp, line 270
WARNING: 1 sort operation has occurred for the SQL statement '0x10b578838'.  See https://developer.mozilla.org/En/Storage/Warnings details.: file /builds/slave/tryserver-macosx64-debug/build/storage/src/mozStoragePrivateHelpers.cpp, line 138
++DOCSHELL 0x10b656dd0 == 1
++DOMWINDOW == 1 (0x10b7baf98) [serial = 1] [outer = 0x0]
++DOCSHELL 0x11137c9d0 == 2
++DOMWINDOW == 2 (0x111380c78) [serial = 2] [outer = 0x0]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 3 (0x11240d208) [serial = 3] [outer = 0x111380c10]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 4 (0x10d5f43a8) [serial = 4] [outer = 0x10b7baf30]
WARNING: Unable to retrieve pref: plugins.unloadASAP: file /builds/slave/tryserver-macosx64-debug/build/modules/plugin/base/src/nsPluginHost.cpp, line 316
************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "'[JavaScript Error: "enum.getNext().gBrowser is undefined" {file: "file:///builds/slave/tryserver-macosx64-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/components/nsBrowserGlue.js" line: 100}]' when calling method: [nsIObserver::observe]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: resource://services-sync/ext/Observers.js :: anonymous :: line 122"  data: yes]
************************************************************
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
++DOCSHELL 0x10d230360 == 3
++DOMWINDOW == 5 (0x10d232b98) [serial = 5] [outer = 0x0]
++DOCSHELL 0x10d24ade0 == 4
++DOMWINDOW == 6 (0x10d24b648) [serial = 6] [outer = 0x0]
Chrome file doesn't exist: /builds/slave/tryserver-macosx64-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/chrome/toolkit/content/global/autocomplete.css
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file /builds/slave/tryserver-macosx64-debug/build/layout/style/Loader.cpp, line 2092
++DOCSHELL 0x110cdbfc0 == 5
WARNING: NS_ENSURE_TRUE(browserChrome) failed: file /builds/slave/tryserver-macosx64-debug/build/docshell/base/nsDocShell.cpp, line 10597
WARNING: Something wrong when creating the docshell for a frameloader!: file /builds/slave/tryserver-macosx64-debug/build/content/base/src/nsFrameLoader.cpp, line 1440
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/slave/tryserver-macosx64-debug/build/content/base/src/nsFrameLoader.cpp, line 275
++DOCSHELL 0x110d52830 == 6
++DOMWINDOW == 7 (0x110d55498) [serial = 7] [outer = 0x0]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 8 (0x110fe2608) [serial = 8] [outer = 0x110d55430]

command timed out: 3600 seconds without output, killing pid 95045
process killed by signal 9
program finished with exit code -1
elapsedTime=4406.782607
=== Output ended ===

Here are four logs from today that see this.  Note that none of them are on the same slave.
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1282657796.1282668277.12648.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1282658575.1282669765.19112.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1282630615.1282641114.4813.gz&fulltext=1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1282658995.1282670958.24875.gz&fulltext=1

Comment 1

[Shawn Wilsher :sdwilsh]

The other fun thing to note here is that tests seem to usually pass on these (in that, it doesn't hang starting up mochitest).

Comment 2

[Aki Sasaki (not active)]

Afaict we turned this off on m-c but it's on on Try.
On my list to turn it off for Try.

Comment 3

[Aki Sasaki (not active)]

Attachment: include macosx64-debug

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 468746 [details] [diff] [review]
include macosx64-debug

I believe self.platform is 'macosx64' even for debug builds.

I think you need to update TryBuildFactory.addLeakTestSteps for this.

Comment 5

[Aki Sasaki (not active)]

Attachment: TryBuildFactory patch

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 469211 [details] [diff] [review]
TryBuildFactory patch

http://hg.mozilla.org/build/buildbotcustom/rev/460f7a6b097a