906782 - releng-puppet2.srv.releng.scl3.mozilla.com OOM'ing

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

It went down a while back, panicing after OOM.  Greg rebooted it, and while I was beginning to poke around at it, I saw more OOMs in the system log and then it disappeared.

Greg has increased the RAM and we're booting it again.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This triggered an instance of bug 885780, and also bug 906785.

Uppping the RAM from 2 to 4G seems to have cured what ails the host.  I don't have much more explanation than that!

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Nope, it's doing it again..

Aug 19 11:37:40 releng-puppet2 kernel: ruby invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000

In this case, it killed httpd, which was good and bad - it didn't panic, but of course it wasn't really doing its job anymore.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I'm trying a rollback:

https://hg.mozilla.org/build/puppet/rev/21566f311bd9

    Bug 906706, 906656, 895639: revert; a=bustage

    Possible one of these commits is causing a puppetmaster to OOM.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

The backout didn't help.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

This is now affecting other onsite masters, but not the AWS masters.

The first OOM was

Aug 19 10:41:24 releng-puppet2 kernel: ruby invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=-17, oom_score_adj=-1000

I don't see anything in syslog around that time that suggests what caused this to start.

From watching a host go down in flames, it looks like one or two Rack processes will just eat RAM.  lsof only shows connections back to the rack management processes, so i can't tell which client triggered it.  Even in normal operation, ruby uses gobs of RAM, so it's hard to tell what's unusual.

The AWS masters are all dead quiet.  I can't tell why that'd be the case.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Stracing one of the blowing-up processes shows, among other things,

select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
brk(0x62612000)                         = 0x62612000
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
brk(0x62633000)                         = 0x62633000
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
brk(0x62654000)                         = 0x62654000
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)
select(0, [], [], [], {0, 0})           = 0 (Timeout)

where the brk() syscalls are increasing the process's memory footprint, and the selects are the equivalent of sleep(0).

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

The gdb backtrace of a failing process, at the point where it's just calling brk() all the time, is huge, and it's stuck in gc hell based on an mnew() call.

#0  0x0000003d85c51223 in gc_mark (ptr=140455444360680, lev=1) at gc.c:804
#1  0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1012
#2  0x0000003d85c5130e in mark_entry (key=<value optimized out>, value=<value optimized out>, lev=<value optimized out>) at gc.c:712
#3  0x0000003d85c9942d in st_foreach (table=0x13184f30, func=0x3d85c51300 <mark_entry>, arg=0) at st.c:488
#4  0x0000003d85caabda in rb_mark_generic_ivar (obj=<value optimized out>) at variable.c:955
#5  0x0000003d85c50ecc in gc_mark_children (ptr=140455444358800, lev=2) at gc.c:848
#6  0x0000003d85c512e4 in mark_keyvalue (key=<value optimized out>, value=140455444358800, lev=1) at gc.c:764
#7  0x0000003d85c9942d in st_foreach (table=0x13182ed0, func=0x3d85c512c0 <mark_keyvalue>, arg=1) at st.c:488
#8  0x0000003d85c5107c in mark_hash (ptr=140455444356000, lev=1) at gc.c:774
#9  gc_mark_children (ptr=140455444356000, lev=1) at gc.c:1018
#10 0x0000003d85c5130e in mark_entry (key=<value optimized out>, value=<value optimized out>, lev=<value optimized out>) at gc.c:712
#11 0x0000003d85c9942d in st_foreach (table=0x13182ef0, func=0x3d85c51300 <mark_entry>, arg=0) at st.c:488
#12 0x0000003d85caabda in rb_mark_generic_ivar (obj=<value optimized out>) at variable.c:955
#13 0x0000003d85c50ecc in gc_mark_children (ptr=140455444355960, lev=3) at gc.c:848
#14 0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=2) at gc.c:1012
#15 0x0000003d85c51050 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1012
#16 0x0000003d85c50d4f in mark_locations_array (x=<value optimized out>, n=40969) at gc.c:690
#17 0x0000003d85c515f7 in garbage_collect () at gc.c:1493
#18 0x0000003d85c5226f in ruby_xmalloc (size=<value optimized out>) at gc.c:153
#19 0x0000003d85c34970 in mnew (klass=140455596193680, obj=140455595878120, id=55767, mklass=140455623223960) at eval.c:9367
#20 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455595878120, id=10233, oid=<value optimized out>, argc=1, argv=0x7fff7ceaa620, body=0x7fbe5f876638, flags=0) at eval.c:5968
#21 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455595878120, mid=10233, argc=1, argv=0x7fff7ceaa620, scope=0, self=<value optimized out>) at eval.c:6251
#22 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5dea1a78) at eval.c:3530
#23 0x0000003d85c368d8 in rb_eval (self=140455596493120, n=0x7fbe5dea1a50) at eval.c:3524
#24 0x0000003d85c368d8 in rb_eval (self=140455596493120, n=0x7fbe5dea19d8) at eval.c:3524
#25 0x0000003d85c3848f in rb_eval (self=140455596493120, n=0x7fbe5dea1780) at eval.c:3070
#26 0x0000003d85c38506 in rb_eval (self=140455596493120, n=0x7fbe5dea1730) at eval.c:3313
#27 0x0000003d85c376b2 in rb_eval (self=140455596493120, n=0x7fbe5dea1b18) at eval.c:3709
#28 0x0000003d85c3ae52 in rb_yield_0 (val=140455595878120, self=140455596493120, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#29 0x0000003d85c195a1 in rb_ary_each (ary=140455595878280) at array.c:1261
#30 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455595878280, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968
#31 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455595878280, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#32 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5dea1cd0) at eval.c:3530
#33 0x0000003d85c39f8f in rb_eval (self=140455596493120, n=0x7fbe5dea0b00) at eval.c:3255
#34 0x0000003d85c3c603 in rb_call0 (klass=140455596492840, recv=140455596493120, id=55841, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dea0a38, flags=0) at eval.c:6147
#35 0x0000003d85c3cb07 in rb_call (klass=140455596492840, recv=140455596493120, mid=55841, argc=3, argv=0x7fff7ceb08d0, scope=0, self=<value optimized out>) at eval.c:6251
#36 0x0000003d85c39642 in rb_eval (self=140455596493120, n=0x7fbe5de9db80) at eval.c:3530
#37 0x0000003d85c3c603 in rb_call0 (klass=140455596492840, recv=140455596493120, id=55865, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de9d9c8, flags=0) at eval.c:6147
#38 0x0000003d85c3cb07 in rb_call (klass=140455596492840, recv=140455596493120, mid=55865, argc=4, argv=0x7fff7ceb18c0, scope=0, self=<value optimized out>) at eval.c:6251
#39 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de4a098) at eval.c:3530
#40 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56281, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de49788, flags=0) at eval.c:6147
#41 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56281, argc=1, argv=0x7fff7ceb2890, scope=1, self=<value optimized out>) at eval.c:6251
#42 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de57658) at eval.c:3545
#43 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147
#44 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7ceb3850, scope=1, self=<value optimized out>) at eval.c:6251
#45 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de53670) at eval.c:3545
#46 0x0000003d85c368d8 in rb_eval (self=140455565089240, n=0x7fbe5de535d0) at eval.c:3524
#47 0x0000003d85c3ae52 in rb_yield_0 (val=140455063282760, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#48 0x0000003d85c1c3d7 in rb_ary_select (ary=140455063283440) at array.c:1951
#49 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455063283440, id=3769, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f886dd0, flags=0) at eval.c:5968
#50 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455063283440, mid=3769, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#51 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de543e0) at eval.c:3530
#52 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de53580) at eval.c:3255
#53 0x0000003d85c3719e in rb_eval (self=140455565089240, n=0x7fbe5de546d8) at eval.c:3715
#54 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147
#55 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7ceb7de0, scope=1, self=<value optimized out>) at eval.c:6251
#56 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545
#57 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147
#58 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7ceb8da0, scope=1, self=<value optimized out>) at eval.c:6251
#59 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de506f0) at eval.c:3545
#60 0x0000003d85c37fb2 in rb_eval (self=140455565089240, n=0x7fbe5de508a8) at eval.c:3507
#61 0x0000003d85c3ae52 in rb_yield_0 (val=140455145297160, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#62 0x0000003d85c195a1 in rb_ary_each (ary=140455144421200) at array.c:1261
#63 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455144421200, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968
#64 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455144421200, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#65 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de50bf0) at eval.c:3530
#66 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de50498) at eval.c:3255
#67 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147
#68 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7cebc980, scope=1, self=<value optimized out>) at eval.c:6251
#69 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545
#70 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147
#71 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7cebd940, scope=1, self=<value optimized out>) at eval.c:6251
#72 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de506f0) at eval.c:3545
#73 0x0000003d85c37fb2 in rb_eval (self=140455565089240, n=0x7fbe5de508a8) at eval.c:3507
#74 0x0000003d85c3ae52 in rb_yield_0 (val=140455147134440, self=140455565089240, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#75 0x0000003d85c195a1 in rb_ary_each (ary=140455147131360) at array.c:1261
#76 0x0000003d85c3c6e5 in rb_call0 (klass=140455623294040, recv=140455147131360, id=4001, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f887410, flags=0) at eval.c:5968
#77 0x0000003d85c3cb07 in rb_call (klass=140455623294040, recv=140455147131360, mid=4001, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#78 0x0000003d85c39642 in rb_eval (self=140455565089240, n=0x7fbe5de50bf0) at eval.c:3530
#79 0x0000003d85c39f8f in rb_eval (self=140455565089240, n=0x7fbe5de50498) at eval.c:3255
#80 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=56265, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de556c8, flags=0) at eval.c:6147
#81 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=56265, argc=1, argv=0x7fff7cec1520, scope=1, self=<value optimized out>) at eval.c:6251
#82 0x0000003d85c3953b in rb_eval (self=140455565089240, n=0x7fbe5de59b10) at eval.c:3545
#83 0x0000003d85c3c603 in rb_call0 (klass=140455595737240, recv=140455565089240, id=55297, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5de5af88, flags=0) at eval.c:6147
#84 0x0000003d85c3cb07 in rb_call (klass=140455595737240, recv=140455565089240, mid=55297, argc=1, argv=0x7fff7cec24e0, scope=0, self=<value optimized out>) at eval.c:6251
#85 0x0000003d85c39642 in rb_eval (self=140455615755680, n=0x7fbe5dfb0720) at eval.c:3530
#86 0x0000003d85c36d7c in rb_eval (self=140455615755680, n=0x7fbe5dfb06d0) at eval.c:3474
#87 0x0000003d85c3c603 in rb_call0 (klass=140455615298120, recv=140455615755680, id=55201, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfb0860, flags=0) at eval.c:6147
#88 0x0000003d85c3cb07 in rb_call (klass=140455615298120, recv=140455615755680, mid=55201, argc=2, argv=0x7fff7cec3e80, scope=1, self=<value optimized out>) at eval.c:6251
#89 0x0000003d85c3953b in rb_eval (self=140455615755680, n=0x7fbe5dfba860) at eval.c:3545
#90 0x0000003d85c3c603 in rb_call0 (klass=140455615298120, recv=140455615755680, id=10081, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfbaea0, flags=0) at eval.c:6147
#91 0x0000003d85c3cb07 in rb_call (klass=140455615298120, recv=140455615755680, mid=10081, argc=2, argv=0x7fff7cec4e90, scope=0, self=<value optimized out>) at eval.c:6251
#92 0x0000003d85c39642 in rb_eval (self=140455564384120, n=0x7fbe5c064768) at eval.c:3530
#93 0x0000003d85c3719e in rb_eval (self=140455564384120, n=0x7fbe5c064998) at eval.c:3715
#94 0x0000003d85c3c603 in rb_call0 (klass=140455564384080, recv=140455564384120, id=4705, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c064588, flags=0) at eval.c:6147
#95 0x0000003d85c3cb07 in rb_call (klass=140455564384080, recv=140455564384120, mid=4705, argc=2, argv=0x7fff7cec6e28, scope=1, self=<value optimized out>) at eval.c:6251
#96 0x0000003d85c3d23b in rb_f_send (argc=2, argv=0x7fff7cec6e28, recv=140455564384120) at eval.c:6299
#97 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455564384120, id=4225, oid=<value optimized out>, argc=3, argv=0x7fff7cec6e20, body=0x7fbe5f8933c8, flags=0) at eval.c:5968
#98 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455564384120, mid=4225, argc=3, argv=0x7fff7cec6e20, scope=0, self=<value optimized out>) at eval.c:6251
#99 0x0000003d85c39642 in rb_eval (self=140455565092840, n=0x7fbe5c077ac0) at eval.c:3530
#100 0x0000003d85c39ab0 in rb_eval (self=140455565092840, n=0x7fbe5c077048) at eval.c:3341
#101 0x0000003d85c3c603 in rb_call0 (klass=140455564434840, recv=140455565092840, id=64753, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c076ff8, flags=0) at eval.c:6147
#102 0x0000003d85c3cb07 in rb_call (klass=140455564434840, recv=140455565092840, mid=64753, argc=2, argv=0x7fff7cec87d0, scope=1, self=<value optimized out>) at eval.c:6251
#103 0x0000003d85c3953b in rb_eval (self=140455565092840, n=0x7fbe5c076828) at eval.c:3545
#104 0x0000003d85c3ae52 in rb_yield_0 (val=140455565094880, self=140455565092840, klass=<value optimized out>, flags=3, avalue=2) at eval.c:5128
#105 0x0000003d85c3b462 in proc_invoke (proc=140455564434200, args=140455565094880, self=140455565092840, klass=<value optimized out>) at eval.c:8970
#106 0x0000003d85c3c094 in rb_call0 (klass=140455564434840, recv=140455565092840, id=4705, oid=<value optimized out>, argc=2, argv=0x7fff7ceca570, body=0x7fbe5c065af0, flags=0) at eval.c:6016
#107 0x0000003d85c3cb07 in rb_call (klass=140455564434840, recv=140455565092840, mid=4705, argc=2, argv=0x7fff7ceca570, scope=0, self=<value optimized out>) at eval.c:6251
#108 0x0000003d85c39642 in rb_eval (self=140455564336240, n=0x7fbe5c072570) at eval.c:3530
#109 0x0000003d85c3c603 in rb_call0 (klass=140455564335760, recv=140455564336240, id=64873, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5c0724d0, flags=0) at eval.c:6147
#110 0x0000003d85c3cb07 in rb_call (klass=140455564335760, recv=140455564336240, mid=64873, argc=2, argv=0x7fff7cecb540, scope=0, self=<value optimized out>) at eval.c:6251
#111 0x0000003d85c39642 in rb_eval (self=140455558482320, n=0x7fbe5baf0a20) at eval.c:3530
#112 0x0000003d85c3719e in rb_eval (self=140455558482320, n=0x7fbe5baf0b60) at eval.c:3715
#113 0x0000003d85c3c603 in rb_call0 (klass=140455558538680, recv=140455558482320, id=104577, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5baf0d18, flags=0) at eval.c:6147
#114 0x0000003d85c3cb07 in rb_call (klass=140455558538680, recv=140455558482320, mid=104577, argc=5, argv=0x7fff7cecd508, scope=1, self=<value optimized out>) at eval.c:6251
#115 0x0000003d85c3d23b in rb_f_send (argc=5, argv=0x7fff7cecd508, recv=140455558482320) at eval.c:6299
#116 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455558482320, id=4225, oid=<value optimized out>, argc=6, argv=0x7fff7cecd500, body=0x7fbe5f8933c8, flags=0) at eval.c:5968
#117 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455558482320, mid=4225, argc=6, argv=0x7fff7cecd500, scope=1, self=<value optimized out>) at eval.c:6251
#118 0x0000003d85c3953b in rb_eval (self=140455558482320, n=0x7fbe5baf6678) at eval.c:3545
#119 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455558482320, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#120 0x0000003d85c36c9a in rb_eval (self=140455596849120, n=0x7fbe5df309f8) at eval.c:3330
#121 0x0000003d85c3c603 in rb_call0 (klass=140455596712480, recv=140455596849120, id=53361, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5df30b38, flags=0) at eval.c:6147
#122 0x0000003d85c3cb07 in rb_call (klass=140455596712480, recv=140455596849120, mid=53361, argc=1, argv=0x7fff7cecf7d0, scope=0, self=<value optimized out>) at eval.c:6251
#123 0x0000003d85c39642 in rb_eval (self=140455597135080, n=0x7fbe5df96d20) at eval.c:3530
#124 0x0000003d85c476dc in block_pass (self=140455597135080, node=0x7fbe5df96d48) at eval.c:9268
#125 0x0000003d85c38595 in rb_eval (self=140455597135080, n=0x7fbe5df96d48) at eval.c:3241
#126 0x0000003d85c3c603 in rb_call0 (klass=140455596729480, recv=140455597135080, id=53361, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5df97568, flags=0) at eval.c:6147
#127 0x0000003d85c3cb07 in rb_call (klass=140455596729480, recv=140455597135080, mid=53361, argc=1, argv=0x7fff7ced1680, scope=0, self=<value optimized out>) at eval.c:6251
#128 0x0000003d85c39642 in rb_eval (self=140455558482320, n=0x7fbe5baf7118) at eval.c:3530
#129 0x0000003d85c39f8f in rb_eval (self=140455558482320, n=0x7fbe5baf65d8) at eval.c:3255
#130 0x0000003d85c39ab0 in rb_eval (self=140455558482320, n=0x7fbe5baf6038) at eval.c:3341
#131 0x0000003d85c39c3a in rb_eval (self=140455558482320, n=0x7fbe5baf6010) at eval.c:3394
#132 0x0000003d85c3c603 in rb_call0 (klass=140455558538680, recv=140455558482320, id=65193, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5baf5fc0, flags=0) at eval.c:6147
#133 0x0000003d85c3cb07 in rb_call (klass=140455558538680, recv=140455558482320, mid=65193, argc=2, argv=0x7fff7ced43d0, scope=0, self=<value optimized out>) at eval.c:6251
#134 0x0000003d85c39642 in rb_eval (self=140455558536680, n=0x7fbe5bb7c3b8) at eval.c:3530
#135 0x0000003d85c39ab0 in rb_eval (self=140455558536680, n=0x7fbe5bb7acc0) at eval.c:3341
#136 0x0000003d85c3c603 in rb_call0 (klass=140455558538800, recv=140455558536680, id=5521, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5bb7d0b0, flags=0) at eval.c:6147
#137 0x0000003d85c3cb07 in rb_call (klass=140455558538800, recv=140455558536680, mid=5521, argc=1, argv=0x7fff7ced5d90, scope=0, self=<value optimized out>) at eval.c:6251
#138 0x0000003d85c39642 in rb_eval (self=140455558522960, n=0x7fbe5dfdc488) at eval.c:3530
#139 0x0000003d85c38506 in rb_eval (self=140455558522960, n=0x7fbe5dfdc460) at eval.c:3313
#140 0x0000003d85c376b2 in rb_eval (self=140455558522960, n=0x7fbe5dfdc528) at eval.c:3709
#141 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5dfda098) at eval.c:3394
#142 0x0000003d85c3c603 in rb_call0 (klass=140455597393600, recv=140455558522960, id=24433, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5dfda020, flags=4) at eval.c:6147
#143 0x0000003d85c3cb07 in rb_call (klass=140455597393600, recv=140455558522960, mid=24433, argc=4, argv=0x7fff7ced8bd0, scope=1, self=<value optimized out>) at eval.c:6251
#144 0x0000003d85c3953b in rb_eval (self=140455558522960, n=0x7fbe5e0030b0) at eval.c:3545
#145 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e002c78) at eval.c:3341
#146 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e002c50) at eval.c:3394
#147 0x0000003d85c38010 in rb_eval (self=140455558522960, n=0x7fbe5e002bd8) at eval.c:2979
#148 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e0015d0) at eval.c:3341
#149 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e0015a8) at eval.c:3394
#150 0x0000003d85c3c603 in rb_call0 (klass=140455597449480, recv=140455558522960, id=24281, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e001558, flags=2) at eval.c:6147
#151 0x0000003d85c3cb07 in rb_call (klass=140455597449480, recv=140455558522960, mid=24281, argc=3, argv=0x7fff7cedccd0, scope=1, self=<value optimized out>) at eval.c:6251
#152 0x0000003d85c3953b in rb_eval (self=140455558522960, n=0x7fbe5e00d4c0) at eval.c:3545
#153 0x0000003d85c3848f in rb_eval (self=140455558522960, n=0x7fbe5e00d240) at eval.c:3070
#154 0x0000003d85c37c2a in rb_eval (self=140455558522960, n=0x7fbe5e00d038) at eval.c:3179
#155 0x0000003d85c39ab0 in rb_eval (self=140455558522960, n=0x7fbe5e00b738) at eval.c:3341
#156 0x0000003d85c39c3a in rb_eval (self=140455558522960, n=0x7fbe5e00b710) at eval.c:3394
#157 0x0000003d85c3c603 in rb_call0 (klass=140455597449480, recv=140455558522960, id=23713, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e00ec30, flags=0) at eval.c:6147
#158 0x0000003d85c3cb07 in rb_call (klass=140455597449480, recv=140455558522960, mid=23713, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#159 0x0000003d85c39642 in rb_eval (self=140455597393480, n=0x7fbe5e01edd8) at eval.c:3530
#160 0x0000003d85c39c3a in rb_eval (self=140455597393480, n=0x7fbe5e01e8b0) at eval.c:3394
#161 0x0000003d85c3c603 in rb_call0 (klass=140455597392280, recv=140455597393480, id=23617, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e01f9e0, flags=2) at eval.c:6147
#162 0x0000003d85c3cb07 in rb_call (klass=140455597392280, recv=140455597393480, mid=23617, argc=4, argv=0x7fff7cee23c8, scope=1, self=<value optimized out>) at eval.c:6251
#163 0x0000003d85c3d23b in rb_f_send (argc=4, argv=0x7fff7cee23c8, recv=140455597393480) at eval.c:6299
#164 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455597393480, id=4225, oid=<value optimized out>, argc=5, argv=0x7fff7cee23c0, body=0x7fbe5f8933c8, flags=0) at eval.c:5968
#165 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455597393480, mid=4225, argc=5, argv=0x7fff7cee23c0, scope=0, self=<value optimized out>) at eval.c:6251
#166 0x0000003d85c39642 in rb_eval (self=140455597206280, n=0x7fbe5e020e58) at eval.c:3530
#167 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5e0208b8) at eval.c:3341
#168 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455597206280, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#169 0x0000003d85c36c9a in rb_eval (self=140455597206280, n=0x7fbe5f0d1810) at eval.c:3330
#170 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0d0ff0) at eval.c:3341
#171 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0d0fc8) at eval.c:3394
#172 0x0000003d85c3c603 in rb_call0 (klass=140455616444560, recv=140455597206280, id=20761, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0d0a28, flags=4) at eval.c:6147
#173 0x0000003d85c3cb07 in rb_call (klass=140455616444560, recv=140455597206280, mid=20761, argc=2, argv=0x7fff7cee63a0, scope=1, self=<value optimized out>) at eval.c:6251
#174 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5e021448) at eval.c:3545
#175 0x0000003d85c39f8f in rb_eval (self=140455597206280, n=0x7fbe5e020818) at eval.c:3255
#176 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5e01fee0) at eval.c:3394
#177 0x0000003d85c3c603 in rb_call0 (klass=140455597393480, recv=140455597206280, id=19449, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e01fe90, flags=2) at eval.c:6147
#178 0x0000003d85c3cb07 in rb_call (klass=140455597393480, recv=140455597206280, mid=19449, argc=59, argv=0x7fff7cee8d18, scope=1, self=<value optimized out>) at eval.c:6251
#179 0x0000003d85c3d23b in rb_f_send (argc=59, argv=0x7fff7cee8d18, recv=140455597206280) at eval.c:6299
#180 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455597206280, id=4073, oid=<value optimized out>, argc=60, argv=0x7fff7cee8d10, body=0x7fbe5f893378, flags=0) at eval.c:5968
#181 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455597206280, mid=4073, argc=60, argv=0x7fff7cee8d10, scope=1, self=<value optimized out>) at eval.c:6251
#182 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0f5620) at eval.c:3545
#183 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0f5170) at eval.c:3394
#184 0x0000003d85c37c2a in rb_eval (self=140455597206280, n=0x7fbe5f0f4f18) at eval.c:3179
#185 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=20001, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0f4e78, flags=2) at eval.c:6147
#186 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=20001, argc=2, argv=0x7fff7ceeb260, scope=1, self=<value optimized out>) at eval.c:6251
#187 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0fb098) at eval.c:3545
#188 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0faf80) at eval.c:3394
#189 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0faa08) at eval.c:3341
#190 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0fa9e0) at eval.c:3394
#191 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=11161, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fa968, flags=0) at eval.c:6147
#192 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=11161, argc=4, argv=0x7fff7ceedf80, scope=1, self=<value optimized out>) at eval.c:6251
#193 0x0000003d85c3953b in rb_eval (self=140455597206280, n=0x7fbe5f0fc088) at eval.c:3545
#194 0x0000003d85c39ab0 in rb_eval (self=140455597206280, n=0x7fbe5f0fb9a8) at eval.c:3341
#195 0x0000003d85c39c3a in rb_eval (self=140455597206280, n=0x7fbe5f0fb980) at eval.c:3394
#196 0x0000003d85c38010 in rb_eval (self=140455597206280, n=0x7fbe5f0fb908) at eval.c:2979
#197 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455597206280, id=5313, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fd7f8, flags=0) at eval.c:6147
#198 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455597206280, mid=5313, argc=0, argv=0x7fff7cef1758, scope=3, self=<value optimized out>) at eval.c:6251
#199 0x0000003d85c462c2 in rb_call_super (argc=<value optimized out>, argv=<value optimized out>) at eval.c:6419
#200 0x0000003d85c36bd3 in rb_eval (self=140455597206280, n=0x7fbe5e0233b0) at eval.c:3596
#201 0x0000003d85c3c603 in rb_call0 (klass=140455597393480, recv=140455597206280, id=5313, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5e022910, flags=0) at eval.c:6147
#202 0x0000003d85c3cb07 in rb_call (klass=140455597393480, recv=140455597206280, mid=5313, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#203 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f106df8) at eval.c:3530
#204 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616224560, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#205 0x0000003d85c36c9a in rb_eval (self=140455616213360, n=0x7fbe5f1ee298) at eval.c:3330
#206 0x0000003d85c3719e in rb_eval (self=140455616213360, n=0x7fbe5f1ee428) at eval.c:3715
#207 0x0000003d85c3c603 in rb_call0 (klass=140455616229240, recv=140455616213360, id=19633, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f1ef210, flags=0) at eval.c:6147
#208 0x0000003d85c3cb07 in rb_call (klass=140455616229240, recv=140455616213360, mid=19633, argc=1, argv=0x7fff7cef4940, scope=0, self=<value optimized out>) at eval.c:6251
#209 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f1074b0) at eval.c:3530
#210 0x0000003d85c39f8f in rb_eval (self=140455616224560, n=0x7fbe5f106cb8) at eval.c:3255
#211 0x0000003d85c3719e in rb_eval (self=140455616224560, n=0x7fbe5f107550) at eval.c:3715
#212 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616224560, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#213 0x0000003d85c36c9a in rb_eval (self=140455616213360, n=0x7fbe5f1f34c8) at eval.c:3330
#214 0x0000003d85c39c3a in rb_eval (self=140455616213360, n=0x7fbe5f1f16a0) at eval.c:3394
#215 0x0000003d85c3ae52 in rb_yield_0 (val=6, self=140455616213360, klass=<value optimized out>, flags=0, avalue=0) at eval.c:5128
#216 0x0000003d85c2f75a in rb_ensure (b_proc=0x3d85c46ee0 <rb_yield>, data1=6, e_proc=0x7fbe5f427d50 <rb_mutex_unlock>, data2=140455616212760) at eval.c:5604
#217 0x0000003d85c3c6e5 in rb_call0 (klass=140455622990400, recv=140455616212760, id=13345, oid=<value optimized out>, argc=0, argv=0x0, body=0x7fbe5f83d770, flags=0) at eval.c:5968
#218 0x0000003d85c3cb07 in rb_call (klass=140455622990400, recv=140455616212760, mid=13345, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#219 0x0000003d85c39642 in rb_eval (self=140455616213360, n=0x7fbe5f1f36a8) at eval.c:3530
#220 0x0000003d85c39f8f in rb_eval (self=140455616213360, n=0x7fbe5f1f0cf0) at eval.c:3255
#221 0x0000003d85c3c603 in rb_call0 (klass=140455616229240, recv=140455616213360, id=13345, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f1f0868, flags=0) at eval.c:6147
#222 0x0000003d85c3cb07 in rb_call (klass=140455616229240, recv=140455616213360, mid=13345, argc=0, argv=0x0, scope=0, self=<value optimized out>) at eval.c:6251
#223 0x0000003d85c39642 in rb_eval (self=140455616224560, n=0x7fbe5f107708) at eval.c:3530
#224 0x0000003d85c39f8f in rb_eval (self=140455616224560, n=0x7fbe5f106740) at eval.c:3255
#225 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19537, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f107c58, flags=2) at eval.c:6147
#226 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19537, argc=1, argv=0x7fff7cefc870, scope=1, self=<value optimized out>) at eval.c:6251
#227 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f10c5c8) at eval.c:3545
#228 0x0000003d85c36d7c in rb_eval (self=140455616224560, n=0x7fbe5f10c578) at eval.c:3474
#229 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19441, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f10cff0, flags=0) at eval.c:6147
#230 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19441, argc=1, argv=0x7fff7cefe1f0, scope=1, self=<value optimized out>) at eval.c:6251
#231 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f105c28) at eval.c:3545
#232 0x0000003d85c3719e in rb_eval (self=140455616224560, n=0x7fbe5f105ca0) at eval.c:3715
#233 0x0000003d85c39ab0 in rb_eval (self=140455616224560, n=0x7fbe5f103a40) at eval.c:3341
#234 0x0000003d85c3c603 in rb_call0 (klass=140455616219080, recv=140455616224560, id=19449, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f106010, flags=2) at eval.c:6147
#235 0x0000003d85c3cb07 in rb_call (klass=140455616219080, recv=140455616224560, mid=19449, argc=59, argv=0x7fff7cf00b68, scope=1, self=<value optimized out>) at eval.c:6251
#236 0x0000003d85c3d23b in rb_f_send (argc=59, argv=0x7fff7cf00b68, recv=140455616224560) at eval.c:6299
#237 0x0000003d85c3c6e5 in rb_call0 (klass=140455623358800, recv=140455616224560, id=4073, oid=<value optimized out>, argc=60, argv=0x7fff7cf00b60, body=0x7fbe5f893378, flags=0) at eval.c:5968
#238 0x0000003d85c3cb07 in rb_call (klass=140455623358800, recv=140455616224560, mid=4073, argc=60, argv=0x7fff7cf00b60, scope=1, self=<value optimized out>) at eval.c:6251
#239 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f0f5620) at eval.c:3545
#240 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0f5170) at eval.c:3394
#241 0x0000003d85c37c2a in rb_eval (self=140455616224560, n=0x7fbe5f0f4f18) at eval.c:3179
#242 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455616224560, id=20001, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0f4e78, flags=2) at eval.c:6147
#243 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455616224560, mid=20001, argc=2, argv=0x7fff7cf030b0, scope=1, self=<value optimized out>) at eval.c:6251
#244 0x0000003d85c3953b in rb_eval (self=140455616224560, n=0x7fbe5f0fb098) at eval.c:3545
#245 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0faf80) at eval.c:3394
#246 0x0000003d85c39ab0 in rb_eval (self=140455616224560, n=0x7fbe5f0faa08) at eval.c:3341
#247 0x0000003d85c39c3a in rb_eval (self=140455616224560, n=0x7fbe5f0fa9e0) at eval.c:3394
#248 0x0000003d85c3c603 in rb_call0 (klass=140455616444920, recv=140455616224560, id=11161, oid=<value optimized out>, argc=<value optimized out>, argv=<value optimized out>, body=0x7fbe5f0fa968, flags=0) at eval.c:6147
#249 0x0000003d85c3cb07 in rb_call (klass=140455616444920, recv=140455616224560, mid=11161, argc=4, argv=0x7fff7cf05dd0, scope=0, self=<value optimized out>) at eval.c:6251
#250 0x0000003d85c39642 in rb_eval (self=140455623348920, n=0x7fbe5f86f8d8) at eval.c:3530
#251 0x0000003d85c39ab0 in rb_eval (self=140455623348920, n=0x7fbe5f86f220) at eval.c:3341
#252 0x0000003d85c49bf1 in ruby_exec_internal () at eval.c:1658
#253 0x0000003d85c49c35 in ruby_exec () at eval.c:1678
#254 0x0000003d85c49c65 in ruby_run () at eval.c:1688
#255 0x0000000000400831 in main (argc=3, argv=0x7fff7cf07838, envp=<value optimized out>) at main.c:48

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Ugh, I didn't see that trees closed, and assumed
  14:11 < RyanVM|Sheriff> dustin: thanks, retriggering
meant that everything was back to kosher.  At any rate, these hosts are staying up well enough for the last hour, although I haven't found a root cause, so the trees should be open again.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Works for me.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

..which means that the lack of any problem for the last hour or so may be due to a lack of demand.  I've asked Greg to increase the RAM to 6G, which may give us some breathing room, but it won't take effect until a power cycle.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

This still seems stable, but I'm going to avoid landing any puppet patches until I can watch things tomorrow.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

This was due to bug 906706.