Closed Bug 793016 Opened 12 years ago Closed 12 years ago

Very frequent hangs during leak tests on new linux build system

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(firefox17 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox17 --- fixed

People

(Reporter: philor, Assigned: jhopkins)

References

Details

(Keywords: intermittent-failure, Whiteboard: [qa-])

Attachments

(1 file)

proposed mozilla-central workaround suggested by :ted
836 bytes, patch
ted
: review+
catlee
: review+
jhopkins
: checked-in+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=15386072&tree=Mozilla-Inbound Linux x86-64 mozilla-inbound leak test build on 2012-09-20 10:50:41 PDT for push 5543e5e5bcd3 slave: bld-linux64-ec2-041 localhost.localdomain - - [20/Sep/2012 14:46:12] "GET /grid.html HTTP/1.1" 200 - ++DOMWINDOW == 14 (0x3b9ff90) [serial = 19] [outer = 0x46a88d0] command timed out: 3600 seconds without output, attempting to kill program finished with exit code 247 https://tbpl.mozilla.org/php/getParsedLog.php?id=15387455&tree=Mozilla-Inbound bld-linux64-ec2-010 https://tbpl.mozilla.org/php/getParsedLog.php?id=15388348&tree=Mozilla-Inbound bld-linux64-ec2-033 https://tbpl.mozilla.org/php/getParsedLog.php?id=15387848&tree=Mozilla-Inbound bld-linux64-ec2-019 https://tbpl.mozilla.org/php/getParsedLog.php?id=15389762&tree=Mozilla-Inbound bld-linux64-ec2-021 https://tbpl.mozilla.org/php/getParsedLog.php?id=15389924&tree=Mozilla-Inbound bld-linux64-ec2-045 https://tbpl.mozilla.org/php/getParsedLog.php?id=15388873&tree=Firefox bld-linux64-ec2-056
Depends on: 793079
cltbld 1221 1208 0 16:35 ? 00:00:00 Xvfb :2 -screen 0 1280x1024x24 cltbld 1700 1 0 16:36 ? 00:00:08 /tools/buildbot-0.8.4-pre-moz2/bin/python2.7 /tools/buildbot/bin/twistd --no_save --logfile /builds/slave/twistd.log --python /builds/slave/buildbot.tac cltbld 7402 1 0 20:36 ? 00:00:00 /usr/libexec/gam_server cltbld 7531 7505 0 20:37 ? 00:00:00 python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log cltbld 7535 7531 0 20:37 ? 00:00:06 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log cltbld 7536 7531 0 20:37 ? 00:00:00 /usr/bin/perl /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/fix-linux-stack.pl cltbld 7606 7535 0 20:37 ? 00:00:00 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp
No stack trace given in the log... PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Crash dump filename: /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/minidumps/010ac078-817a-80ff-51dee7bf-6e49c685.dmp No symbols path given, can't process dump.
Summary: Very frequent hangs during leak tests on AWS → Very frequent hangs during leak tests on new linux build system
This is also happening on the HP machines.
Severity: normal → major
Assignee: nobody → jhopkins
(In reply to Chris AtLee [:catlee] from comment #9) > minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp The dump is not very useful without the symbol files for the corresponding build.
possibly related? -------------------------------------------------------------------------------- Exception in /builds/buildbot/try1/master/twistd.log: 2012-09-22 16:45:28-0700 [-] Unhandled Error Traceback (most recent call last): File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks self._runCallbacks() File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback self._startRunCallbacks(result) File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1072, in <lambda> d.addCallback(lambda res: self.createSummary(cmd.logs['stdio'])) File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 274, in createSummary if m: File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 75, in formatBytes bytes=float(str(bytes) + '.0') exceptions.ValueError: invalid literal for float(): __codesizeDiff.0
Blocks: 793675
Depends on: 793782
Depends on: 794174
Attachment #664586 - Flags: review?(ted.mielczarek)
Attachment #664586 - Flags: review?(catlee)
Attachment #664586 - Flags: review?(catlee) → review+
Attachment #664586 - Flags: review?(ted.mielczarek) → review+
Comment on attachment 664586 [details] [diff] [review] proposed mozilla-central workaround suggested by :ted Landed in http://hg.mozilla.org/mozilla-central/rev/aa3266ba9cab
Attachment #664586 - Flags: checked-in+
Looks like this is fixed now!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Depends on: 813039
Whiteboard: [orange][qa-] → [qa-]
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: