Last Comment Bug 793016 - Very frequent hangs during leak tests on new linux build system
: Very frequent hangs during leak tests on new linux build system
Status: RESOLVED FIXED
[qa-]
: intermittent-failure
Product: Release Engineering
Classification: Other
Component: General Automation (show other bugs)
: other
: All Linux
: -- major (vote)
: ---
Assigned To: John Hopkins (:jhopkins)
: Chris AtLee [:catlee]
Mentors:
Depends on: 794174 793079 793782 813039
Blocks: 438871 772446 793675
  Show dependency treegraph
 
Reported: 2012-09-20 15:28 PDT by Phil Ringnalda (:philor)
Modified: 2013-08-12 21:54 PDT (History)
6 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
fixed


Attachments
proposed mozilla-central workaround suggested by :ted (836 bytes, patch)
2012-09-25 11:19 PDT, John Hopkins (:jhopkins)
ted: review+
catlee: review+
jhopkins: checked‑in+
Details | Diff | Review

Description Phil Ringnalda (:philor) 2012-09-20 15:28:47 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15386072&tree=Mozilla-Inbound
Linux x86-64 mozilla-inbound leak test build on 2012-09-20 10:50:41 PDT for push 5543e5e5bcd3
slave: bld-linux64-ec2-041

localhost.localdomain - - [20/Sep/2012 14:46:12] "GET /grid.html HTTP/1.1" 200 -
++DOMWINDOW == 14 (0x3b9ff90) [serial = 19] [outer = 0x46a88d0]

command timed out: 3600 seconds without output, attempting to kill
program finished with exit code 247

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387455&tree=Mozilla-Inbound
bld-linux64-ec2-010

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388348&tree=Mozilla-Inbound
bld-linux64-ec2-033

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387848&tree=Mozilla-Inbound
bld-linux64-ec2-019

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389762&tree=Mozilla-Inbound
bld-linux64-ec2-021

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389924&tree=Mozilla-Inbound
bld-linux64-ec2-045

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388873&tree=Firefox
bld-linux64-ec2-056
Comment 1 Phil Ringnalda (:philor) 2012-09-20 16:10:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15391409&tree=Firefox
bld-linux64-ec2-035
Comment 2 Ryan VanderMeulen [:RyanVM] 2012-09-20 17:20:59 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15394022&tree=Firefox
s: bld-linux64-ec2-035
Comment 3 Ryan VanderMeulen [:RyanVM] 2012-09-20 17:21:37 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15394069&tree=Firefox
s: bld-linux64-ec2-026
Comment 4 Phil Ringnalda (:philor) 2012-09-20 18:12:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395290&tree=Fx-Team
bld-linux64-ec2-058
Comment 6 Phil Ringnalda (:philor) 2012-09-20 18:29:01 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395678&tree=Mozilla-Inbound
bld-linux64-ec2-032
Comment 7 Phil Ringnalda (:philor) 2012-09-20 20:55:04 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15398365&tree=Cedar
bld-linux64-ec2-045
Comment 8 Phil Ringnalda (:philor) 2012-09-20 21:27:57 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15390311&tree=Cedar
bld-centos6-hp-017
Comment 9 Chris AtLee [:catlee] 2012-09-20 21:34:39 PDT
cltbld    1221  1208  0 16:35 ?        00:00:00 Xvfb :2 -screen 0 1280x1024x24
cltbld    1700     1  0 16:36 ?        00:00:08 /tools/buildbot-0.8.4-pre-moz2/bin/python2.7 /tools/buildbot/bin/twistd --no_save --logfile /builds/slave/twistd.log --python /builds/slave/buildbot.tac
cltbld    7402     1  0 20:36 ?        00:00:00 /usr/libexec/gam_server
cltbld    7531  7505  0 20:37 ?        00:00:00 python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7535  7531  0 20:37 ?        00:00:06 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7536  7531  0 20:37 ?        00:00:00 /usr/bin/perl /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/fix-linux-stack.pl
cltbld    7606  7535  0 20:37 ?        00:00:00 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log

minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp
Comment 10 Chris AtLee [:catlee] 2012-09-20 21:35:05 PDT
No stack trace given in the log...

PROCESS-CRASH | Main app process exited normally | application crashed (minidump found)
Crash dump filename: /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/minidumps/010ac078-817a-80ff-51dee7bf-6e49c685.dmp
No symbols path given, can't process dump.
Comment 11 Phil Ringnalda (:philor) 2012-09-20 22:58:26 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15400937&tree=Mozilla-Inbound
bld-linux64-ec2-009
Comment 13 Chris AtLee [:catlee] 2012-09-21 05:32:59 PDT
This is also happening on the HP machines.
Comment 14 Phil Ringnalda (:philor) 2012-09-21 10:08:41 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15412541&tree=Firefox
bld-linux64-ec2-058
Comment 16 Phil Ringnalda (:philor) 2012-09-21 10:25:28 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15414893&tree=Mozilla-Inbound
bld-linux64-ec2-024
Comment 20 Phil Ringnalda (:philor) 2012-09-21 23:01:21 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15433875&tree=Mozilla-Inbound
bld-linux64-ec2-007
Comment 21 Mike Hommey [:glandium] 2012-09-22 00:21:31 PDT
(In reply to Chris AtLee [:catlee] from comment #9)
> minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp

The dump is not very useful without the symbol files for the corresponding build.
Comment 23 Ryan VanderMeulen [:RyanVM] 2012-09-22 07:18:15 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15441080&tree=Firefox
s: bld-linux64-ec2-049
Comment 24 Ryan VanderMeulen [:RyanVM] 2012-09-22 07:41:27 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15441766&tree=Firefox
s: bld-linux64-ec2-034
Comment 25 Phil Ringnalda (:philor) 2012-09-22 10:19:41 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15443534&tree=Fx-Team
bld-linux64-ec2-036
Comment 26 Phil Ringnalda (:philor) 2012-09-22 11:06:39 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15445629&tree=Fx-Team
bld-linux64-ec2-052
Comment 28 Phil Ringnalda (:philor) 2012-09-22 11:51:24 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15446553&tree=Mozilla-Inbound
bld-linux64-ec2-008
Comment 31 Phil Ringnalda (:philor) 2012-09-22 18:29:22 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15450534&tree=Firefox
bld-centos6-hp-006
Comment 32 Phil Ringnalda (:philor) 2012-09-22 20:14:54 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15451521&tree=Mozilla-Inbound
bld-centos6-hp-009
Comment 33 Phil Ringnalda (:philor) 2012-09-22 20:57:26 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15452214&tree=Mozilla-Inbound
bld-linux64-ec2-048
Comment 34 Justin Wood (:Callek) 2012-09-22 21:01:11 PDT
possibly related?

--------------------------------------------------------------------------------
Exception in /builds/buildbot/try1/master/twistd.log:
2012-09-22 16:45:28-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
            self._runCallbacks()
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1072, in <lambda>
            d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 274, in createSummary
            if m:
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 75, in formatBytes
            bytes=float(str(bytes) + '.0')
        exceptions.ValueError: invalid literal for float(): __codesizeDiff.0
Comment 36 Phil Ringnalda (:philor) 2012-09-23 10:21:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15456564&tree=Firefox
bld-linux64-ec2-070
Comment 37 Phil Ringnalda (:philor) 2012-09-23 10:26:51 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15458017&tree=Mozilla-Inbound
bld-linux64-ec2-051
Comment 38 Phil Ringnalda (:philor) 2012-09-23 10:48:12 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15458944&tree=Firefox
bld-centos6-hp-006
Comment 39 Phil Ringnalda (:philor) 2012-09-23 11:00:17 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459037&tree=Firefox
 bld-linux64-ec2-048
Comment 40 Phil Ringnalda (:philor) 2012-09-23 11:19:15 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459523&tree=Mozilla-Inbound
bld-linux64-ec2-059
Comment 41 Phil Ringnalda (:philor) 2012-09-23 11:29:53 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459594&tree=Fx-Team
bld-linux64-ec2-058
Comment 42 Phil Ringnalda (:philor) 2012-09-23 15:08:25 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15463026&tree=Mozilla-Inbound
bld-linux64-ec2-008
Comment 43 Phil Ringnalda (:philor) 2012-09-23 15:19:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15463155&tree=Mozilla-Inbound
bld-linux64-ec2-059
Comment 44 Phil Ringnalda (:philor) 2012-09-23 17:16:28 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15464433&tree=Firefox
bld-centos6-hp-006
Comment 45 Phil Ringnalda (:philor) 2012-09-23 18:09:41 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15466015&tree=Mozilla-Inbound
bld-centos6-hp-019
Comment 46 Phil Ringnalda (:philor) 2012-09-23 18:55:17 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15467119&tree=Mozilla-Inbound
bld-linux64-ec2-031
Comment 47 Phil Ringnalda (:philor) 2012-09-23 19:58:30 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15468789&tree=Firefox
bld-linux64-ec2-006
Comment 48 Phil Ringnalda (:philor) 2012-09-23 20:06:04 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15469079&tree=Firefox
bld-centos6-hp-009
Comment 49 Phil Ringnalda (:philor) 2012-09-23 21:35:18 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15469866&tree=Mozilla-Inbound
bld-linux64-ec2-045
Comment 50 Phil Ringnalda (:philor) 2012-09-23 21:59:08 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471395&tree=Mozilla-Inbound
bld-linux64-ec2-020
Comment 51 Phil Ringnalda (:philor) 2012-09-23 22:02:18 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471459&tree=Mozilla-Inbound
bld-linux64-ec2-048
Comment 52 Phil Ringnalda (:philor) 2012-09-23 22:41:35 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471591&tree=Mozilla-Inbound
bld-linux64-ec2-022
Comment 53 Phil Ringnalda (:philor) 2012-09-24 08:08:38 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15478425&tree=Mozilla-Inbound
 bld-linux64-ec2-046
Comment 54 Phil Ringnalda (:philor) 2012-09-24 08:14:13 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15473262&tree=Mozilla-Inbound
bld-linux64-ec2-020
Comment 55 Phil Ringnalda (:philor) 2012-09-24 08:17:53 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15479030&tree=Firefox
bld-linux64-ec2-069
Comment 56 Phil Ringnalda (:philor) 2012-09-24 10:43:33 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15483879&tree=Mozilla-Inbound
bld-linux64-ec2-063
Comment 57 Phil Ringnalda (:philor) 2012-09-24 13:17:01 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15487266&tree=Mozilla-Inbound
bld-linux64-ec2-046
Comment 58 Phil Ringnalda (:philor) 2012-09-24 14:16:07 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15488914&tree=Mozilla-Inbound
bld-linux64-ec2-064
Comment 59 Phil Ringnalda (:philor) 2012-09-24 15:06:47 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15489605&tree=Mozilla-Inbound
bld-linux64-ec2-038
Comment 60 Phil Ringnalda (:philor) 2012-09-24 15:10:42 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15490382&tree=Mozilla-Inbound
bld-linux64-ec2-019
Comment 62 Phil Ringnalda (:philor) 2012-09-24 15:43:41 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15491799&tree=Mozilla-Inbound
bld-linux64-ec2-031
Comment 63 Phil Ringnalda (:philor) 2012-09-24 16:40:02 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15493559&tree=Mozilla-Inbound
bld-linux64-ec2-062
Comment 64 Phil Ringnalda (:philor) 2012-09-24 17:09:57 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15494473&tree=Mozilla-Inbound
bld-linux64-ec2-021
Comment 65 Phil Ringnalda (:philor) 2012-09-24 17:34:26 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15495269&tree=Mozilla-Inbound
bld-linux64-ec2-019
Comment 66 Phil Ringnalda (:philor) 2012-09-24 20:59:25 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15500612&tree=Mozilla-Inbound
bld-linux64-ec2-019
Comment 67 Phil Ringnalda (:philor) 2012-09-24 21:50:05 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15502565&tree=Mozilla-Inbound
bld-linux64-ec2-026
Comment 69 Phil Ringnalda (:philor) 2012-09-24 23:27:15 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=15505285&tree=Fx-Team
bld-centos6-hp-009
Comment 75 John Hopkins (:jhopkins) 2012-09-25 11:19:58 PDT
Created attachment 664586 [details] [diff] [review]
proposed mozilla-central workaround suggested by :ted
Comment 77 John Hopkins (:jhopkins) 2012-09-25 12:54:18 PDT
Comment on attachment 664586 [details] [diff] [review]
proposed mozilla-central workaround suggested by :ted

Landed in http://hg.mozilla.org/mozilla-central/rev/aa3266ba9cab
Comment 78 Chris AtLee [:catlee] 2012-09-26 10:50:48 PDT
Looks like this is fixed now!
Comment 79 Phil Ringnalda (:philor) 2012-10-05 15:33:01 PDT
Hi, I'm back.

https://tbpl.mozilla.org/php/getParsedLog.php?id=15844614&tree=Mozilla-Aurora

Note You need to log in before you can comment on or make changes to this bug.