Closed Bug 774844 Opened 12 years ago Closed 11 years ago

Intermittent OS X 10.7 leakstats | log file incomplete (after "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90")

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: emorley, Unassigned)

References

(
URL
)

Details

(Keywords: intermittent-failure, Whiteboard: [red])

Not sure if this should be in Core::* or release engineering. OS X 10.7 64-bit mozilla-inbound leak test build on 2012-07-17 08:28:49 PDT for push 0888a88ab4c7 slave: bld-lion-r5-080 https://tbpl.mozilla.org/php/getParsedLog.php?id=13605090&tree=Mozilla-Inbound { ========= Started compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-17 09:02:59.237050) ========= obj-firefox/dist/bin/leakstats ../malloc.log in dir /builds/slave/m-in-osx64-dbg/build (timeout 1200 secs) watching logfiles {} argv: ['obj-firefox/dist/bin/leakstats', '../malloc.log'] environment: Apple_PubSub_Socket_Render=/tmp/launch-6NvL74/Render CCACHE_BASEDIR=/builds/slave/m-in-osx64-dbg CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 CVS_RSH=ssh DISPLAY=/tmp/launch-DIb19R/org.x:0 HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld LC_ALL=C LOGNAME=cltbld MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SIGN_CMD=python /builds/slave/m-in-osx64-dbg/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-osx64-dbg/signing_cache -t /builds/slave/m-in-osx64-dbg/token -n /builds/slave/m-in-osx64-dbg/nonce -c /builds/slave/m-in-osx64-dbg/tools/release/signing/host.cert -H mac-signing3.build.scl1.mozilla.com:9100 -H mac-signing4.build.scl1.mozilla.com:9100 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-in-osx64-dbg/build PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-ckJ4IH/Listeners TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/ USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1F5:0:0 using PTY: False obj-firefox/dist/bin/leakstats starting at Tue Jul 17 09:02:59 2012 obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! Unknown event type 0xffffff90 obj-firefox/dist/bin/leakstats: log file incomplete program finished with exit code 1 elapsedTime=0.245636 Unable to parse leakstats output ========= Finished compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-17 09:02:59.506804) ========= }
Summary: OS X 10.7 build failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1" → OS X 10.7 compare current leak logs failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1"
https://tbpl.mozilla.org/php/getParsedLog.php?id=13761534&tree=Mozilla-Inbound { ========= Started compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-22 19:33:16.858487) ========= obj-firefox/dist/bin/leakstats ../malloc.log in dir /builds/slave/m-in-osx64-dbg/build (timeout 1200 secs) watching logfiles {} argv: ['obj-firefox/dist/bin/leakstats', '../malloc.log'] environment: Apple_PubSub_Socket_Render=/tmp/launch-9hZd6s/Render CCACHE_BASEDIR=/builds/slave/m-in-osx64-dbg CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 CVS_RSH=ssh DISPLAY=/tmp/launch-ohFZM5/org.x:0 HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld LC_ALL=C LOGNAME=cltbld MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SIGN_CMD=python /builds/slave/m-in-osx64-dbg/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-osx64-dbg/signing_cache -t /builds/slave/m-in-osx64-dbg/token -n /builds/slave/m-in-osx64-dbg/nonce -c /builds/slave/m-in-osx64-dbg/tools/release/signing/host.cert -H mac-signing3.build.scl1.mozilla.com:9100 -H mac-signing4.build.scl1.mozilla.com:9100 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-in-osx64-dbg/build PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-IMAG00/Listeners TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/ USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1F5:0:0 using PTY: False obj-firefox/dist/bin/leakstats starting at Sun Jul 22 19:33:16 2012 Unknown event type 0xffffffe8 obj-firefox/dist/bin/leakstats: log file incomplete program finished with exit code 1 elapsedTime=0.174396 Unable to parse leakstats output ========= Finished compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-22 19:33:17.055132) ========= }
Depends on: 539334
Whiteboard: [orange][red] → [red]
Depends on: 828946
Summary: OS X 10.7 compare current leak logs failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1" → Intermittent OS X 10.7 leakstats | log file incomplete (after "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90")
Note this doesn't appear on OrangeFactor due to bug 694170.
Please may you find someone to own this (is a top orange at the moment)? :-)
Flags: needinfo?(catlee)
dbaron, any ideas about this?
Flags: needinfo?(dbaron)
Flags: needinfo?(catlee)
Is there any chance these machines are running out of disk space while running the test? Do we know how large the files typically are? Given that the diffbloatdump output isn't functioning correctly, I suspect the answer might be that we should just stop running the compare-logs part of this, at least. (Though maybe it's working on other platforms.) Though the part that's failing isn't actually that part. (It used to show run-to-run differences of allocation stacks, so if a lot more memory were allocated in a particular stack than in the previous run, we'd show the stack for that allocation, in tree form. In other words, the point of the log comparison was that for leak regressions in this test, the log would actually often have enough data in the log to debug the problem without doing anything else.) Maybe chat with the memshrink folks, though; they might know if anybody is still using this. (Given that we've mostly stopped caring about not cleaning stuff up at shutdown, this has become a less useful test. It also had UI that worked better with tinderbox and doesn't work well with TBPL, so people stopped paying attention to it post tbpl-switch.)
Flags: needinfo?(dbaron) → needinfo?(khuey)
Except s/previous run/some other run/, since it stopped being a compare to anything other than "some random other run" the minute we got our second buildslave running on a platform, or at least within a few hours, the first time that one push did a clobber build while a later push did a dep build and finished before the previous push's build. I think last time we wanted to kill it, Standard8 and one other person, I forget who, spoke up on its behalf.
One of the costs of this failure which doesn't appear in the bug is the way it is dealt with on Try - nearly everyone retriggers the build, and thus also triggers twice as many tests (on all three OS X versions). An amusing, and very simple, way of dealing with it would be to just turn it into not-an-error, by taking out the TEST-UNEXPECTED-FAIL and returning 0. If we then completely break it without realizing it because it's silent, surely the people who rely on it working will notice, won't they?
Depends on: 887234