Closed Bug 793016 Opened 12 years ago Closed 12 years ago

Very frequent hangs during leak tests on new linux build system

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(firefox17 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox17 --- fixed

People

(Reporter: philor, Assigned: jhopkins)

References

Details

(Keywords: intermittent-failure, Whiteboard: [qa-])

Attachments

(1 file)

proposed mozilla-central workaround suggested by :ted
836 bytes, patch
ted
: review+
catlee
: review+
jhopkins
: checked-in+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=15386072&tree=Mozilla-Inbound
Linux x86-64 mozilla-inbound leak test build on 2012-09-20 10:50:41 PDT for push 5543e5e5bcd3
slave: bld-linux64-ec2-041

localhost.localdomain - - [20/Sep/2012 14:46:12] "GET /grid.html HTTP/1.1" 200 -
++DOMWINDOW == 14 (0x3b9ff90) [serial = 19] [outer = 0x46a88d0]

command timed out: 3600 seconds without output, attempting to kill
program finished with exit code 247

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387455&tree=Mozilla-Inbound
bld-linux64-ec2-010

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388348&tree=Mozilla-Inbound
bld-linux64-ec2-033

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387848&tree=Mozilla-Inbound
bld-linux64-ec2-019

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389762&tree=Mozilla-Inbound
bld-linux64-ec2-021

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389924&tree=Mozilla-Inbound
bld-linux64-ec2-045

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388873&tree=Firefox
bld-linux64-ec2-056
Depends on: 793079
cltbld    1221  1208  0 16:35 ?        00:00:00 Xvfb :2 -screen 0 1280x1024x24
cltbld    1700     1  0 16:36 ?        00:00:08 /tools/buildbot-0.8.4-pre-moz2/bin/python2.7 /tools/buildbot/bin/twistd --no_save --logfile /builds/slave/twistd.log --python /builds/slave/buildbot.tac
cltbld    7402     1  0 20:36 ?        00:00:00 /usr/libexec/gam_server
cltbld    7531  7505  0 20:37 ?        00:00:00 python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7535  7531  0 20:37 ?        00:00:06 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7536  7531  0 20:37 ?        00:00:00 /usr/bin/perl /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/fix-linux-stack.pl
cltbld    7606  7535  0 20:37 ?        00:00:00 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log

minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp
No stack trace given in the log...

PROCESS-CRASH | Main app process exited normally | application crashed (minidump found)
Crash dump filename: /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/minidumps/010ac078-817a-80ff-51dee7bf-6e49c685.dmp
No symbols path given, can't process dump.
Summary: Very frequent hangs during leak tests on AWS → Very frequent hangs during leak tests on new linux build system
This is also happening on the HP machines.
Severity: normal → major
Assignee: nobody → jhopkins
(In reply to Chris AtLee [:catlee] from comment #9)
> minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp

The dump is not very useful without the symbol files for the corresponding build.
possibly related?

--------------------------------------------------------------------------------
Exception in /builds/buildbot/try1/master/twistd.log:
2012-09-22 16:45:28-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
            self._runCallbacks()
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1072, in <lambda>
            d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 274, in createSummary
            if m:
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 75, in formatBytes
            bytes=float(str(bytes) + '.0')
        exceptions.ValueError: invalid literal for float(): __codesizeDiff.0
Blocks: 793675
Depends on: 793782
Depends on: 794174

        Attachment #664586 -
        Flags: review?(ted.mielczarek)

        Attachment #664586 -
        Flags: review?(catlee)

        Attachment #664586 -
        Flags: review?(catlee) → review+

        Attachment #664586 -
        Flags: review?(ted.mielczarek) → review+

    
    

    
  
Comment on attachment 664586 [details] [diff] [review]
proposed mozilla-central workaround suggested by :ted

Landed in http://hg.mozilla.org/mozilla-central/rev/aa3266ba9cab

        Attachment #664586 -
        Flags: checked-in+

    
    

    
  
Looks like this is fixed now!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED

    
  
Depends on: 813039
Whiteboard: [orange][qa-] → [qa-]
Product: mozilla.org → Release Engineering
Component: General Automation → General

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: