Closed Bug 774844 Opened 14 years ago Closed 13 years ago

Intermittent OS X 10.7 leakstats | log file incomplete (after "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90")

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: emorley, Unassigned)

References

(
URL
)

Details

(Keywords: intermittent-failure, Whiteboard: [red])

Not sure if this should be in Core::* or release engineering. OS X 10.7 64-bit mozilla-inbound leak test build on 2012-07-17 08:28:49 PDT for push 0888a88ab4c7 slave: bld-lion-r5-080 https://tbpl.mozilla.org/php/getParsedLog.php?id=13605090&tree=Mozilla-Inbound { ========= Started compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-17 09:02:59.237050) ========= obj-firefox/dist/bin/leakstats ../malloc.log in dir /builds/slave/m-in-osx64-dbg/build (timeout 1200 secs) watching logfiles {} argv: ['obj-firefox/dist/bin/leakstats', '../malloc.log'] environment: Apple_PubSub_Socket_Render=/tmp/launch-6NvL74/Render CCACHE_BASEDIR=/builds/slave/m-in-osx64-dbg CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 CVS_RSH=ssh DISPLAY=/tmp/launch-DIb19R/org.x:0 HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld LC_ALL=C LOGNAME=cltbld MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SIGN_CMD=python /builds/slave/m-in-osx64-dbg/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-osx64-dbg/signing_cache -t /builds/slave/m-in-osx64-dbg/token -n /builds/slave/m-in-osx64-dbg/nonce -c /builds/slave/m-in-osx64-dbg/tools/release/signing/host.cert -H mac-signing3.build.scl1.mozilla.com:9100 -H mac-signing4.build.scl1.mozilla.com:9100 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-in-osx64-dbg/build PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-ckJ4IH/Listeners TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/ USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1F5:0:0 using PTY: False obj-firefox/dist/bin/leakstats starting at Tue Jul 17 09:02:59 2012 obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! Unknown event type 0xffffff90 obj-firefox/dist/bin/leakstats: log file incomplete program finished with exit code 1 elapsedTime=0.245636 Unable to parse leakstats output ========= Finished compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-17 09:02:59.506804) ========= }
Summary: OS X 10.7 build failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1" → OS X 10.7 compare current leak logs failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1"
https://tbpl.mozilla.org/php/getParsedLog.php?id=13761534&tree=Mozilla-Inbound { ========= Started compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-22 19:33:16.858487) ========= obj-firefox/dist/bin/leakstats ../malloc.log in dir /builds/slave/m-in-osx64-dbg/build (timeout 1200 secs) watching logfiles {} argv: ['obj-firefox/dist/bin/leakstats', '../malloc.log'] environment: Apple_PubSub_Socket_Render=/tmp/launch-9hZd6s/Render CCACHE_BASEDIR=/builds/slave/m-in-osx64-dbg CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 CVS_RSH=ssh DISPLAY=/tmp/launch-ohFZM5/org.x:0 HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld LC_ALL=C LOGNAME=cltbld MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox MOZ_SIGN_CMD=python /builds/slave/m-in-osx64-dbg/tools/release/signing/signtool.py --cachedir /builds/slave/m-in-osx64-dbg/signing_cache -t /builds/slave/m-in-osx64-dbg/token -n /builds/slave/m-in-osx64-dbg/nonce -c /builds/slave/m-in-osx64-dbg/tools/release/signing/host.cert -H mac-signing3.build.scl1.mozilla.com:9100 -H mac-signing4.build.scl1.mozilla.com:9100 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-in-osx64-dbg/build PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-IMAG00/Listeners TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/ USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1F5:0:0 using PTY: False obj-firefox/dist/bin/leakstats starting at Sun Jul 22 19:33:16 2012 Unknown event type 0xffffffe8 obj-firefox/dist/bin/leakstats: log file incomplete program finished with exit code 1 elapsedTime=0.174396 Unable to parse leakstats output ========= Finished compare current leak logs failed (results: 2, elapsed: 0 secs) (at 2012-07-22 19:33:17.055132) ========= }
Depends on: 539334
Whiteboard: [orange][red] → [red]
Depends on: 828946
Summary: OS X 10.7 compare current leak logs failure with: "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90 | obj-firefox/dist/bin/leakstats: log file incomplete | program finished with exit code 1" → Intermittent OS X 10.7 leakstats | log file incomplete (after "obj-firefox/dist/bin/leakstats: no callsite for 'F' (70)! | Unknown event type 0xffffff90")
Note this doesn't appear on OrangeFactor due to bug 694170.
Please may you find someone to own this (is a top orange at the moment)? :-)
Flags: needinfo?(catlee)
dbaron, any ideas about this?
Flags: needinfo?(dbaron)
Flags: needinfo?(catlee)
Is there any chance these machines are running out of disk space while running the test? Do we know how large the files typically are? Given that the diffbloatdump output isn't functioning correctly, I suspect the answer might be that we should just stop running the compare-logs part of this, at least. (Though maybe it's working on other platforms.) Though the part that's failing isn't actually that part. (It used to show run-to-run differences of allocation stacks, so if a lot more memory were allocated in a particular stack than in the previous run, we'd show the stack for that allocation, in tree form. In other words, the point of the log comparison was that for leak regressions in this test, the log would actually often have enough data in the log to debug the problem without doing anything else.) Maybe chat with the memshrink folks, though; they might know if anybody is still using this. (Given that we've mostly stopped caring about not cleaning stuff up at shutdown, this has become a less useful test. It also had UI that worked better with tinderbox and doesn't work well with TBPL, so people stopped paying attention to it post tbpl-switch.)
Flags: needinfo?(dbaron) → needinfo?(khuey)
Except s/previous run/some other run/, since it stopped being a compare to anything other than "some random other run" the minute we got our second buildslave running on a platform, or at least within a few hours, the first time that one push did a clobber build while a later push did a dep build and finished before the previous push's build. I think last time we wanted to kill it, Standard8 and one other person, I forget who, spoke up on its behalf.
One of the costs of this failure which doesn't appear in the bug is the way it is dealt with on Try - nearly everyone retriggers the build, and thus also triggers twice as many tests (on all three OS X versions). An amusing, and very simple, way of dealing with it would be to just turn it into not-an-error, by taking out the TEST-UNEXPECTED-FAIL and returning 0. If we then completely break it without realizing it because it's silent, surely the people who rely on it working will notice, won't they?
Depends on: 887234
(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #328) > Is there any chance these machines are running out of disk space while > running the test? Do we know how large the files typically are? I _could_ add checks for disk space before and after the step runs. (In reply to Phil Ringnalda (:philor) from comment #331) > One of the costs of this failure which doesn't appear in the bug is the way > it is dealt with on Try - nearly everyone retriggers the build, and thus > also triggers twice as many tests (on all three OS X versions). > > An amusing, and very simple, way of dealing with it would be to just turn it > into not-an-error, by taking out the TEST-UNEXPECTED-FAIL and returning 0. > If we then completely break it without realizing it because it's silent, > surely the people who rely on it working will notice, won't they? If no one is looking the results, and failures just get blindly retriggered, we should turn it off. I'll roll a patch for that unless someone convinces me otherwise.
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: -- → P2
That patch already exists, it's in bug 887234 waiting for review.
Assignee: coop → nobody
Status: ASSIGNED → NEW
Priority: P2 → --
(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #328) > Is there any chance these machines are running out of disk space while > running the test? Do we know how large the files typically are? malloc.log is about 14MB, while the mac compile machines typically have 100s of GB free. More likely that something is going wrong writing the log.
(In reply to Nick Thomas [:nthomas] from comment #402) > (In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from > comment #328) > > Is there any chance these machines are running out of disk space while > > running the test? Do we know how large the files typically are? iirc, I think at one time we were hitting this with the Thunderbird boxes (can't remember which infra though). One thought I did have was that we've only got one place on ftp that these logs write to. I don't know enough about ftp, but could there be some sort of clash when uploading files?
(In reply to Phil Ringnalda (:philor) from comment #329) > Except s/previous run/some other run/, since it stopped being a compare to > anything other than "some random other run" the minute we got our second > buildslave running on a platform, or at least within a few hours, the first > time that one push did a clobber build while a later push did a dep build > and finished before the previous push's build. > > I think last time we wanted to kill it, Standard8 and one other person, I > forget who, spoke up on its behalf. At the time, I'd certainly been using trace-malloc to locate leaks in Thunderbird that I couldn't have located otherwise, and I think I caught a couple of regressions that way. I haven't done enough in that area recently to know what the state of all our tools are to know if it is worth keeping or not.
Yeah this test is a lot less important these days. I wouldn't be sad if we turned it off, though we should probably ask dev-platform before we go ahead and nuke it.
Flags: needinfo?(khuey)
Depends on: 617441
Found in triage. Note: catlee asked in dev.platform (https://groups.google.com/forum/#!topic/mozilla.dev.platform/o1JWvVnXBOU) if these should be disabled, as they have been failing intermittently for ~1 year now with no-one looking at them. If you care about these tests continuing to be run, consuming CPU cycles, please respond on that thread.
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
We turned off leak tests instead.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.