Closed Bug 906782 Opened 13 years ago Closed 13 years ago

releng-puppet2.srv.releng.scl3.mozilla.com OOM'ing

Categories

(Infrastructure & Operations :: RelOps: Puppet, task)

Product:
Infrastructure & Operations
Component:
RelOps: Puppet
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

Details

(Whiteboard: [reit-ops] [closed-trees])

It went down a while back, panicing after OOM. Greg rebooted it, and while I was beginning to poke around at it, I saw more OOMs in the system log and then it disappeared. Greg has increased the RAM and we're booting it again.
This triggered an instance of bug 885780, and also bug 906785. Uppping the RAM from 2 to 4G seems to have cured what ails the host. I don't have much more explanation than that!
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Nope, it's doing it again.. Aug 19 11:37:40 releng-puppet2 kernel: ruby invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 In this case, it killed httpd, which was good and bad - it didn't panic, but of course it wasn't really doing its job anymore.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm trying a rollback: https://hg.mozilla.org/build/puppet/rev/21566f311bd9 Bug 906706, 906656, 895639: revert; a=bustage Possible one of these commits is causing a puppetmaster to OOM.
The backout didn't help.
This is now affecting other onsite masters, but not the AWS masters. The first OOM was Aug 19 10:41:24 releng-puppet2 kernel: ruby invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=-17, oom_score_adj=-1000 I don't see anything in syslog around that time that suggests what caused this to start. From watching a host go down in flames, it looks like one or two Rack processes will just eat RAM. lsof only shows connections back to the rack management processes, so i can't tell which client triggered it. Even in normal operation, ruby uses gobs of RAM, so it's hard to tell what's unusual. The AWS masters are all dead quiet. I can't tell why that'd be the case.
Stracing one of the blowing-up processes shows, among other things, select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) brk(0x62612000) = 0x62612000 select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) brk(0x62633000) = 0x62633000 select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) brk(0x62654000) = 0x62654000 select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) select(0, [], [], [], {0, 0}) = 0 (Timeout) where the brk() syscalls are increasing the process's memory footprint, and the selects are the equivalent of sleep(0).
The gdb backtrace of a failing process, at the point where it's just calling brk() all the time, is huge, and it's stuck in gc hell based on an mnew() call. #0 0x0000003d85c51223 in gc_mark (ptr=140455444360680, lev=1) at gc.c:804 #1 0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1012 #2 0x0000003d85c5130e in mark_entry (key=<value optimized out>, value=<value optimized out>, lev=<value optimized out>) at gc.c:712 #3 0x0000003d85c9942d in st_foreach (table=0x13184f30, func=0x3d85c51300 <mark_entry>, arg=0) at st.c:488 #4 0x0000003d85caabda in rb_mark_generic_ivar (obj=<value optimized out>) at variable.c:955 #5 0x0000003d85c50ecc in gc_mark_children (ptr=140455444358800, lev=2) at gc.c:848 #6 0x0000003d85c512e4 in mark_keyvalue (key=<value optimized out>, value=140455444358800, lev=1) at gc.c:764 #7 0x0000003d85c9942d in st_foreach (table=0x13182ed0, func=0x3d85c512c0 <mark_keyvalue>, arg=1) at st.c:488 #8 0x0000003d85c5107c in mark_hash (ptr=140455444356000, lev=1) at gc.c:774 #9 gc_mark_children (ptr=140455444356000, lev=1) at gc.c:1018 #10 0x0000003d85c5130e in mark_entry (key=<value optimized out>, value=<value optimized out>, lev=<value optimized out>) at gc.c:712 #11 0x0000003d85c9942d in st_foreach (table=0x13182ef0, func=0x3d85c51300 <mark_entry>, arg=0) at st.c:488 #12 0x0000003d85caabda in rb_mark_generic_ivar (obj=<value optimized out>) at variable.c:955 #13 0x0000003d85c50ecc in gc_mark_children (ptr=140455444355960, lev=3) at gc.c:848 #14 0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=2) at gc.c:1012 #15 0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1012 #16 0x0000003d85c50d4f in mark_locations_array (x=<value optimized out>, n=40969) at gc.c:690 #17 0x0000003d85c515f7 in garbage_collect () at gc.c:1493 #18 0x0000003d85c5226f in ruby_xmalloc (size=<value optimized out>) at gc.c:153 #19 0x0000003d85c34970 in mnew (klass=140455596193680, obj=140455595878120, id=55767, mklass=140455623223960) at eval.c:9367 #20 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455595878120, id=10233, oid=<value optimized out>, argc=1, argv=0x7fff7ceaa620, body=0x7fbe5f876638, flags=0) at eval.c:5968 #21 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455595878120, mid=10233, argc=1, argv=0x7fff7ceaa620, scope=0, self=<value optimized out>) at eval.c:6251 #22 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5dea1a78) at eval.c:3530 #23 0x0000003d85c368d8 in rb_eval (self=140455596493120, n=0x7fbe5dea1a50) at eval.c:3524 #24 0x0000003d85c368d8 in rb_eval (self=140455596493120, n=0x7fbe5dea19d8) at eval.c:3524 #25 0x0000003d85c3848f in rb_eval (self=140455596493120, n=0x7fbe5dea1780) at eval.c:3070 #26 0x0000003d85c38506 in rb_eval (self=140455596493120, n=0x7fbe5dea1730) at eval.c:3313 #27 0x0000003d85c376b2 in rb_eval (self=140455596493120, n=0x7fbe5dea1b18) at eval.c:3709 #28 0x0000003d85c3ae52 in rb_yield_0 (val=140455595878120, self=140455596493120, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #29 0x0000003d85c195a1 in rb_ary_each (ary=140455595878280) at array.c:1261 #30 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455595878280, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968 #31 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455595878280, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #32 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5dea1cd0) at eval.c:3530 #33 0x0000003d85c39f8f in rb_eval (self=140455596493120, n=0x7fbe5dea0b00) at eval.c:3255 #34 0x0000003d85c3c603 in rb_call0 (klass=140455596492840, recv=140455596493120, id=55841, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dea0a38, flags=0) at eval.c:6147 #35 0x0000003d85c3cb07 in rb_call (klass=140455596492840, recv=140455596493120, mid=55841, argc=3, argv=0x7fff7ceb08d0, scope=0, self=<value optimized out>) at eval.c:6251 #36 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5de9db80) at eval.c:3530 #37 0x0000003d85c3c603 in rb_call0 (klass=140455596492840, recv=140455596493120, id=55865, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de9d9c8, flags=0) at eval.c:6147 #38 0x0000003d85c3cb07 in rb_call (klass=140455596492840, recv=140455596493120, mid=55865, argc=4, argv=0x7fff7ceb18c0, scope=0, self=<value optimized out>) at eval.c:6251 #39 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de4a098) at eval.c:3530 #40 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56281, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de49788, flags=0) at eval.c:6147 #41 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56281, argc=1, argv=0x7fff7ceb2890, scope=1, self=<value optimized out>) at eval.c:6251 #42 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de57658) at eval.c:3545 #43 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147 #44 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7ceb3850, scope=1, self=<value optimized out>) at eval.c:6251 #45 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de53670) at eval.c:3545 #46 0x0000003d85c368d8 in rb_eval (self=140455565089240, n=0x7fbe5de535d0) at eval.c:3524 #47 0x0000003d85c3ae52 in rb_yield_0 (val=140455063282760, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #48 0x0000003d85c1c3d7 in rb_ary_select (ary=140455063283440) at array.c:1951 #49 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455063283440, id=3769, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f886dd0, flags=0) at eval.c:5968 #50 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455063283440, mid=3769, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #51 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de543e0) at eval.c:3530 #52 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de53580) at eval.c:3255 #53 0x0000003d85c3719e in rb_eval (self=140455565089240, n=0x7fbe5de546d8) at eval.c:3715 #54 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147 #55 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7ceb7de0, scope=1, self=<value optimized out>) at eval.c:6251 #56 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545 #57 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147 #58 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7ceb8da0, scope=1, self=<value optimized out>) at eval.c:6251 #59 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de506f0) at eval.c:3545 #60 0x0000003d85c37fb2 in rb_eval (self=140455565089240, n=0x7fbe5de508a8) at eval.c:3507 #61 0x0000003d85c3ae52 in rb_yield_0 (val=140455145297160, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #62 0x0000003d85c195a1 in rb_ary_each (ary=140455144421200) at array.c:1261 #63 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455144421200, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968 #64 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455144421200, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #65 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de50bf0) at eval.c:3530 #66 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de50498) at eval.c:3255 #67 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147 #68 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7cebc980, scope=1, self=<value optimized out>) at eval.c:6251 #69 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545 #70 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147 #71 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7cebd940, scope=1, self=<value optimized out>) at eval.c:6251 #72 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de506f0) at eval.c:3545 #73 0x0000003d85c37fb2 in rb_eval (self=140455565089240, n=0x7fbe5de508a8) at eval.c:3507 #74 0x0000003d85c3ae52 in rb_yield_0 (val=140455147134440, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #75 0x0000003d85c195a1 in rb_ary_each (ary=140455147131360) at array.c:1261 #76 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455147131360, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968 #77 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455147131360, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #78 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de50bf0) at eval.c:3530 #79 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de50498) at eval.c:3255 #80 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147 #81 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7cec1520, scope=1, self=<value optimized out>) at eval.c:6251 #82 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545 #83 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147 #84 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7cec24e0, scope=0, self=<value optimized out>) at eval.c:6251 #85 0x0000003d85c39642 in rb_eval (self=140455615755680, n=0x7fbe5dfb0720) at eval.c:3530 #86 0x0000003d85c36d7c in rb_eval (self=140455615755680, n=0x7fbe5dfb06d0) at eval.c:3474 #87 0x0000003d85c3c603 in rb_call0 (klass=140455615298120, recv=140455615755680, id=55201, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfb0860, flags=0) at eval.c:6147 #88 0x0000003d85c3cb07 in rb_call (klass=140455615298120, recv=140455615755680, mid=55201, argc=2, argv=0x7fff7cec3e80, scope=1, self=<value optimized out>) at eval.c:6251 #89 0x0000003d85c3953b in rb_eval (self=140455615755680, n=0x7fbe5dfba860) at eval.c:3545 #90 0x0000003d85c3c603 in rb_call0 (klass=140455615298120, recv=140455615755680, id=10081, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfbaea0, flags=0) at eval.c:6147 #91 0x0000003d85c3cb07 in rb_call (klass=140455615298120, recv=140455615755680, mid=10081, argc=2, argv=0x7fff7cec4e90, scope=0, self=<value optimized out>) at eval.c:6251 #92 0x0000003d85c39642 in rb_eval (self=140455564384120, n=0x7fbe5c064768) at eval.c:3530 #93 0x0000003d85c3719e in rb_eval (self=140455564384120, n=0x7fbe5c064998) at eval.c:3715 #94 0x0000003d85c3c603 in rb_call0 (klass=140455564384080, recv=140455564384120, id=4705, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c064588, flags=0) at eval.c:6147 #95 0x0000003d85c3cb07 in rb_call (klass=140455564384080, recv=140455564384120, mid=4705, argc=2, argv=0x7fff7cec6e28, scope=1, self=<value optimized out>) at eval.c:6251 #96 0x0000003d85c3d23b in rb_f_send (argc=2, argv=0x7fff7cec6e28, recv=140455564384120) at eval.c:6299 #97 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455564384120, id=4225, oid=<value optimized out>, argc=3, argv=0x7fff7cec6e20, body=0x7fbe5f8933c8, flags=0) at eval.c:5968 #98 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455564384120, mid=4225, argc=3, argv=0x7fff7cec6e20, scope=0, self=<value optimized out>) at eval.c:6251 #99 0x0000003d85c39642 in rb_eval (self=140455565092840, n=0x7fbe5c077ac0) at eval.c:3530 #100 0x0000003d85c39ab0 in rb_eval (self=140455565092840, n=0x7fbe5c077048) at eval.c:3341 #101 0x0000003d85c3c603 in rb_call0 (klass=140455564434840, recv=140455565092840, id=64753, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c076ff8, flags=0) at eval.c:6147 #102 0x0000003d85c3cb07 in rb_call (klass=140455564434840, recv=140455565092840, mid=64753, argc=2, argv=0x7fff7cec87d0, scope=1, self=<value optimized out>) at eval.c:6251 #103 0x0000003d85c3953b in rb_eval (self=140455565092840, n=0x7fbe5c076828) at eval.c:3545 #104 0x0000003d85c3ae52 in rb_yield_0 (val=140455565094880, self=140455565092840, klass=<value optimized out>, flags=3, avalue=2) at eval.c:5128 #105 0x0000003d85c3b462 in proc_invoke (proc=140455564434200, args=140455565094880, self=140455565092840, klass=<value optimized out>) at eval.c:8970 #106 0x0000003d85c3c094 in rb_call0 (klass=140455564434840, recv=140455565092840, id=4705, oid=<value optimized out>, argc=2, argv=0x7fff7ceca570, body=0x7fbe5c065af0, flags=0) at eval.c:6016 #107 0x0000003d85c3cb07 in rb_call (klass=140455564434840, recv=140455565092840, mid=4705, argc=2, argv=0x7fff7ceca570, scope=0, self=<value optimized out>) at eval.c:6251 #108 0x0000003d85c39642 in rb_eval (self=140455564336240, n=0x7fbe5c072570) at eval.c:3530 #109 0x0000003d85c3c603 in rb_call0 (klass=140455564335760, recv=140455564336240, id=64873, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c0724d0, flags=0) at eval.c:6147 #110 0x0000003d85c3cb07 in rb_call (klass=140455564335760, recv=140455564336240, mid=64873, argc=2, argv=0x7fff7cecb540, scope=0, self=<value optimized out>) at eval.c:6251 #111 0x0000003d85c39642 in rb_eval (self=140455558482320, n=0x7fbe5baf0a20) at eval.c:3530 #112 0x0000003d85c3719e in rb_eval (self=140455558482320, n=0x7fbe5baf0b60) at eval.c:3715 #113 0x0000003d85c3c603 in rb_call0 (klass=140455558538680, recv=140455558482320, id=104577, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5baf0d18, flags=0) at eval.c:6147 #114 0x0000003d85c3cb07 in rb_call (klass=140455558538680, recv=140455558482320, mid=104577, argc=5, argv=0x7fff7cecd508, scope=1, self=<value optimized out>) at eval.c:6251 #115 0x0000003d85c3d23b in rb_f_send (argc=5, argv=0x7fff7cecd508, recv=140455558482320) at eval.c:6299 #116 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455558482320, id=4225, oid=<value optimized out>, argc=6, argv=0x7fff7cecd500, body=0x7fbe5f8933c8, flags=0) at eval.c:5968 #117 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455558482320, mid=4225, argc=6, argv=0x7fff7cecd500, scope=1, self=<value optimized out>) at eval.c:6251 #118 0x0000003d85c3953b in rb_eval (self=140455558482320, n=0x7fbe5baf6678) at eval.c:3545 #119 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455558482320, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #120 0x0000003d85c36c9a in rb_eval (self=140455596849120, n=0x7fbe5df309f8) at eval.c:3330 #121 0x0000003d85c3c603 in rb_call0 (klass=140455596712480, recv=140455596849120, id=53361, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5df30b38, flags=0) at eval.c:6147 #122 0x0000003d85c3cb07 in rb_call (klass=140455596712480, recv=140455596849120, mid=53361, argc=1, argv=0x7fff7cecf7d0, scope=0, self=<value optimized out>) at eval.c:6251 #123 0x0000003d85c39642 in rb_eval (self=140455597135080, n=0x7fbe5df96d20) at eval.c:3530 #124 0x0000003d85c476dc in block_pass (self=140455597135080, node=0x7fbe5df96d48) at eval.c:9268 #125 0x0000003d85c38595 in rb_eval (self=140455597135080, n=0x7fbe5df96d48) at eval.c:3241 #126 0x0000003d85c3c603 in rb_call0 (klass=140455596729480, recv=140455597135080, id=53361, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5df97568, flags=0) at eval.c:6147 #127 0x0000003d85c3cb07 in rb_call (klass=140455596729480, recv=140455597135080, mid=53361, argc=1, argv=0x7fff7ced1680, scope=0, self=<value optimized out>) at eval.c:6251 #128 0x0000003d85c39642 in rb_eval (self=140455558482320, n=0x7fbe5baf7118) at eval.c:3530 #129 0x0000003d85c39f8f in rb_eval (self=140455558482320, n=0x7fbe5baf65d8) at eval.c:3255 #130 0x0000003d85c39ab0 in rb_eval (self=140455558482320, n=0x7fbe5baf6038) at eval.c:3341 #131 0x0000003d85c39c3a in rb_eval (self=140455558482320, n=0x7fbe5baf6010) at eval.c:3394 #132 0x0000003d85c3c603 in rb_call0 (klass=140455558538680, recv=140455558482320, id=65193, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5baf5fc0, flags=0) at eval.c:6147 #133 0x0000003d85c3cb07 in rb_call (klass=140455558538680, recv=140455558482320, mid=65193, argc=2, argv=0x7fff7ced43d0, scope=0, self=<value optimized out>) at eval.c:6251 #134 0x0000003d85c39642 in rb_eval (self=140455558536680, n=0x7fbe5bb7c3b8) at eval.c:3530 #135 0x0000003d85c39ab0 in rb_eval (self=140455558536680, n=0x7fbe5bb7acc0) at eval.c:3341 #136 0x0000003d85c3c603 in rb_call0 (klass=140455558538800, recv=140455558536680, id=5521, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5bb7d0b0, flags=0) at eval.c:6147 #137 0x0000003d85c3cb07 in rb_call (klass=140455558538800, recv=140455558536680, mid=5521, argc=1, argv=0x7fff7ced5d90, scope=0, self=<value optimized out>) at eval.c:6251 #138 0x0000003d85c39642 in rb_eval (self=140455558522960, n=0x7fbe5dfdc488) at eval.c:3530 #139 0x0000003d85c38506 in rb_eval (self=140455558522960, n=0x7fbe5dfdc460) at eval.c:3313 #140 0x0000003d85c376b2 in rb_eval (self=140455558522960, n=0x7fbe5dfdc528) at eval.c:3709 #141 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5dfda098) at eval.c:3394 #142 0x0000003d85c3c603 in rb_call0 (klass=140455597393600, recv=140455558522960, id=24433, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfda020, flags=4) at eval.c:6147 #143 0x0000003d85c3cb07 in rb_call (klass=140455597393600, recv=140455558522960, mid=24433, argc=4, argv=0x7fff7ced8bd0, scope=1, self=<value optimized out>) at eval.c:6251 #144 0x0000003d85c3953b in rb_eval (self=140455558522960, n=0x7fbe5e0030b0) at eval.c:3545 #145 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e002c78) at eval.c:3341 #146 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e002c50) at eval.c:3394 #147 0x0000003d85c38010 in rb_eval (self=140455558522960, n=0x7fbe5e002bd8) at eval.c:2979 #148 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e0015d0) at eval.c:3341 #149 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e0015a8) at eval.c:3394 #150 0x0000003d85c3c603 in rb_call0 (klass=140455597449480, recv=140455558522960, id=24281, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e001558, flags=2) at eval.c:6147 #151 0x0000003d85c3cb07 in rb_call (klass=140455597449480, recv=140455558522960, mid=24281, argc=3, argv=0x7fff7cedccd0, scope=1, self=<value optimized out>) at eval.c:6251 #152 0x0000003d85c3953b in rb_eval (self=140455558522960, n=0x7fbe5e00d4c0) at eval.c:3545 #153 0x0000003d85c3848f in rb_eval (self=140455558522960, n=0x7fbe5e00d240) at eval.c:3070 #154 0x0000003d85c37c2a in rb_eval (self=140455558522960, n=0x7fbe5e00d038) at eval.c:3179 #155 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e00b738) at eval.c:3341 #156 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e00b710) at eval.c:3394 #157 0x0000003d85c3c603 in rb_call0 (klass=140455597449480, recv=140455558522960, id=23713, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e00ec30, flags=0) at eval.c:6147 #158 0x0000003d85c3cb07 in rb_call (klass=140455597449480, recv=140455558522960, mid=23713, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #159 0x0000003d85c39642 in rb_eval (self=140455597393480, n=0x7fbe5e01edd8) at eval.c:3530 #160 0x0000003d85c39c3a in rb_eval (self=140455597393480, n=0x7fbe5e01e8b0) at eval.c:3394 #161 0x0000003d85c3c603 in rb_call0 (klass=140455597392280, recv=140455597393480, id=23617, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e01f9e0, flags=2) at eval.c:6147 #162 0x0000003d85c3cb07 in rb_call (klass=140455597392280, recv=140455597393480, mid=23617, argc=4, argv=0x7fff7cee23c8, scope=1, self=<value optimized out>) at eval.c:6251 #163 0x0000003d85c3d23b in rb_f_send (argc=4, argv=0x7fff7cee23c8, recv=140455597393480) at eval.c:6299 #164 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455597393480, id=4225, oid=<value optimized out>, argc=5, argv=0x7fff7cee23c0, body=0x7fbe5f8933c8, flags=0) at eval.c:5968 #165 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455597393480, mid=4225, argc=5, argv=0x7fff7cee23c0, scope=0, self=<value optimized out>) at eval.c:6251 #166 0x0000003d85c39642 in rb_eval (self=140455597206280, n=0x7fbe5e020e58) at eval.c:3530 #167 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5e0208b8) at eval.c:3341 #168 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455597206280, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #169 0x0000003d85c36c9a in rb_eval (self=140455597206280, n=0x7fbe5f0d1810) at eval.c:3330 #170 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0d0ff0) at eval.c:3341 #171 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0d0fc8) at eval.c:3394 #172 0x0000003d85c3c603 in rb_call0 (klass=140455616444560, recv=140455597206280, id=20761, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0d0a28, flags=4) at eval.c:6147 #173 0x0000003d85c3cb07 in rb_call (klass=140455616444560, recv=140455597206280, mid=20761, argc=2, argv=0x7fff7cee63a0, scope=1, self=<value optimized out>) at eval.c:6251 #174 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5e021448) at eval.c:3545 #175 0x0000003d85c39f8f in rb_eval (self=140455597206280, n=0x7fbe5e020818) at eval.c:3255 #176 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5e01fee0) at eval.c:3394 #177 0x0000003d85c3c603 in rb_call0 (klass=140455597393480, recv=140455597206280, id=19449, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e01fe90, flags=2) at eval.c:6147 #178 0x0000003d85c3cb07 in rb_call (klass=140455597393480, recv=140455597206280, mid=19449, argc=59, argv=0x7fff7cee8d18, scope=1, self=<value optimized out>) at eval.c:6251 #179 0x0000003d85c3d23b in rb_f_send (argc=59, argv=0x7fff7cee8d18, recv=140455597206280) at eval.c:6299 #180 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455597206280, id=4073, oid=<value optimized out>, argc=60, argv=0x7fff7cee8d10, body=0x7fbe5f893378, flags=0) at eval.c:5968 #181 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455597206280, mid=4073, argc=60, argv=0x7fff7cee8d10, scope=1, self=<value optimized out>) at eval.c:6251 #182 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0f5620) at eval.c:3545 #183 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0f5170) at eval.c:3394 #184 0x0000003d85c37c2a in rb_eval (self=140455597206280, n=0x7fbe5f0f4f18) at eval.c:3179 #185 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=20001, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0f4e78, flags=2) at eval.c:6147 #186 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=20001, argc=2, argv=0x7fff7ceeb260, scope=1, self=<value optimized out>) at eval.c:6251 #187 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0fb098) at eval.c:3545 #188 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0faf80) at eval.c:3394 #189 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0faa08) at eval.c:3341 #190 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0fa9e0) at eval.c:3394 #191 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=11161, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fa968, flags=0) at eval.c:6147 #192 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=11161, argc=4, argv=0x7fff7ceedf80, scope=1, self=<value optimized out>) at eval.c:6251 #193 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0fc088) at eval.c:3545 #194 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0fb9a8) at eval.c:3341 #195 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0fb980) at eval.c:3394 #196 0x0000003d85c38010 in rb_eval (self=140455597206280, n=0x7fbe5f0fb908) at eval.c:2979 #197 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=5313, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fd7f8, flags=0) at eval.c:6147 #198 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=5313, argc=0, argv=0x7fff7cef1758, scope=3, self=<value optimized out>) at eval.c:6251 #199 0x0000003d85c462c2 in rb_call_super (argc=<value optimized out>, argv=<value optimized out>) at eval.c:6419 #200 0x0000003d85c36bd3 in rb_eval (self=140455597206280, n=0x7fbe5e0233b0) at eval.c:3596 #201 0x0000003d85c3c603 in rb_call0 (klass=140455597393480, recv=140455597206280, id=5313, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e022910, flags=0) at eval.c:6147 #202 0x0000003d85c3cb07 in rb_call (klass=140455597393480, recv=140455597206280, mid=5313, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #203 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f106df8) at eval.c:3530 #204 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616224560, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #205 0x0000003d85c36c9a in rb_eval (self=140455616213360, n=0x7fbe5f1ee298) at eval.c:3330 #206 0x0000003d85c3719e in rb_eval (self=140455616213360, n=0x7fbe5f1ee428) at eval.c:3715 #207 0x0000003d85c3c603 in rb_call0 (klass=140455616229240, recv=140455616213360, id=19633, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f1ef210, flags=0) at eval.c:6147 #208 0x0000003d85c3cb07 in rb_call (klass=140455616229240, recv=140455616213360, mid=19633, argc=1, argv=0x7fff7cef4940, scope=0, self=<value optimized out>) at eval.c:6251 #209 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f1074b0) at eval.c:3530 #210 0x0000003d85c39f8f in rb_eval (self=140455616224560, n=0x7fbe5f106cb8) at eval.c:3255 #211 0x0000003d85c3719e in rb_eval (self=140455616224560, n=0x7fbe5f107550) at eval.c:3715 #212 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616224560, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #213 0x0000003d85c36c9a in rb_eval (self=140455616213360, n=0x7fbe5f1f34c8) at eval.c:3330 #214 0x0000003d85c39c3a in rb_eval (self=140455616213360, n=0x7fbe5f1f16a0) at eval.c:3394 #215 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616213360, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128 #216 0x0000003d85c2f75a in rb_ensure (b_proc=0x3d85c46ee0 <rb_yield>, data1=6, e_proc=0x7fbe5f427d50 <rb_mutex_unlock>, data2=140455616212760) at eval.c:5604 #217 0x0000003d85c3c6e5 in rb_call0 (klass=140455622990400, recv=140455616212760, id=13345, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f83d770, flags=0) at eval.c:5968 #218 0x0000003d85c3cb07 in rb_call (klass=140455622990400, recv=140455616212760, mid=13345, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #219 0x0000003d85c39642 in rb_eval (self=140455616213360, n=0x7fbe5f1f36a8) at eval.c:3530 #220 0x0000003d85c39f8f in rb_eval (self=140455616213360, n=0x7fbe5f1f0cf0) at eval.c:3255 #221 0x0000003d85c3c603 in rb_call0 (klass=140455616229240, recv=140455616213360, id=13345, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f1f0868, flags=0) at eval.c:6147 #222 0x0000003d85c3cb07 in rb_call (klass=140455616229240, recv=140455616213360, mid=13345, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251 #223 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f107708) at eval.c:3530 #224 0x0000003d85c39f8f in rb_eval (self=140455616224560, n=0x7fbe5f106740) at eval.c:3255 #225 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19537, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f107c58, flags=2) at eval.c:6147 #226 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19537, argc=1, argv=0x7fff7cefc870, scope=1, self=<value optimized out>) at eval.c:6251 #227 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f10c5c8) at eval.c:3545 #228 0x0000003d85c36d7c in rb_eval (self=140455616224560, n=0x7fbe5f10c578) at eval.c:3474 #229 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19441, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f10cff0, flags=0) at eval.c:6147 #230 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19441, argc=1, argv=0x7fff7cefe1f0, scope=1, self=<value optimized out>) at eval.c:6251 #231 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f105c28) at eval.c:3545 #232 0x0000003d85c3719e in rb_eval (self=140455616224560, n=0x7fbe5f105ca0) at eval.c:3715 #233 0x0000003d85c39ab0 in rb_eval (self=140455616224560, n=0x7fbe5f103a40) at eval.c:3341 #234 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19449, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f106010, flags=2) at eval.c:6147 #235 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19449, argc=59, argv=0x7fff7cf00b68, scope=1, self=<value optimized out>) at eval.c:6251 #236 0x0000003d85c3d23b in rb_f_send (argc=59, argv=0x7fff7cf00b68, recv=140455616224560) at eval.c:6299 #237 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455616224560, id=4073, oid=<value optimized out>, argc=60, argv=0x7fff7cf00b60, body=0x7fbe5f893378, flags=0) at eval.c:5968 #238 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455616224560, mid=4073, argc=60, argv=0x7fff7cf00b60, scope=1, self=<value optimized out>) at eval.c:6251 #239 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f0f5620) at eval.c:3545 #240 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0f5170) at eval.c:3394 #241 0x0000003d85c37c2a in rb_eval (self=140455616224560, n=0x7fbe5f0f4f18) at eval.c:3179 #242 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455616224560, id=20001, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0f4e78, flags=2) at eval.c:6147 #243 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455616224560, mid=20001, argc=2, argv=0x7fff7cf030b0, scope=1, self=<value optimized out>) at eval.c:6251 #244 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f0fb098) at eval.c:3545 #245 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0faf80) at eval.c:3394 #246 0x0000003d85c39ab0 in rb_eval (self=140455616224560, n=0x7fbe5f0faa08) at eval.c:3341 #247 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0fa9e0) at eval.c:3394 #248 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455616224560, id=11161, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fa968, flags=0) at eval.c:6147 #249 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455616224560, mid=11161, argc=4, argv=0x7fff7cf05dd0, scope=0, self=<value optimized out>) at eval.c:6251 #250 0x0000003d85c39642 in rb_eval (self=140455623348920, n=0x7fbe5f86f8d8) at eval.c:3530 #251 0x0000003d85c39ab0 in rb_eval (self=140455623348920, n=0x7fbe5f86f220) at eval.c:3341 #252 0x0000003d85c49bf1 in ruby_exec_internal () at eval.c:1658 #253 0x0000003d85c49c35 in ruby_exec () at eval.c:1678 #254 0x0000003d85c49c65 in ruby_run () at eval.c:1688 #255 0x0000000000400831 in main (argc=3, argv=0x7fff7cf07838, envp=<value optimized out>) at main.c:48
Whiteboard: [reit-ops] [closed-trees]
Ugh, I didn't see that trees closed, and assumed 14:11 < RyanVM|Sheriff> dustin: thanks, retriggering meant that everything was back to kosher. At any rate, these hosts are staying up well enough for the last hour, although I haven't found a root cause, so the trees should be open again.
Works for me.
..which means that the lack of any problem for the last hour or so may be due to a lack of demand. I've asked Greg to increase the RAM to 6G, which may give us some breathing room, but it won't take effect until a power cycle.
This still seems stable, but I'm going to avoid landing any puppet patches until I can watch things tomorrow.
This was due to bug 906706.
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.