1235436 - All Socorro Processors failing on memory error

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[K Lars Lohn [:lars] [:klohn]]

At approximately 7:30am PST, all the Socorro processors died within moments of each other.  Investigation showed that the python module ujson was core dumping and killing the entire processor process.  For a reason that is not clear, the automatic restart was failing.  Interestingly, while the problem is recurring every few minutes on all of the processor, the automatic restart is being successful 95% of the time.

The problem appears to happen in the processor rule "OutOfMemoryBinaryRule" when it attempts to read the json "memory report" submitted by the client.  ujson raises an unrecoverable "double free" or "memory corruption" error.  

There is no history of this problem happening in the past.  It began suddenly and continues to recur every few minutes. It is not the same single crash repeating over and over, each crash that triggers the problem is new.  Something changed on the Web that induces a crash in Firefox that in turn induces a crash in ujson, which brings the processors down.  

Experimenting with a workaround, substituting the 'json' module for 'json' forestalls the problem entirely.  PR pending...

collectors and crashmovers are not affected.  No crashes are being lost.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

Interestingly, the problem ceased almost exactly at noon, 12pm DST 2015-12-18.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

and then it came back for a couple hours on 12/28
   3 processors died of it
   7 recovered but gained no immunity

Comment 3

[Peter Bengtsson [:peterbe]]

How can we get access to these kinds of blobs for local testing/debugging?

Comment 4

[Peter Bengtsson [:peterbe]]

By the way we're running a release of ujson from April 2014. https://bugzilla.mozilla.org/show_bug.cgi?id=1237386

Comment 5

[Peter Bengtsson [:peterbe]]

Upgrading ujson is unlikely to solve this. We still need something to reproduce against. 

But I want to connect the bugs. Which we might want to re-evaluate later.

Comment 6

[Peter Bengtsson [:peterbe]]

It happened again. ujson 1.35 "solved it" but we don't want 1gb processed crash data in so we filed https://bugzilla.mozilla.org/show_bug.cgi?id=1248610 which took care of it.