1434131 - Segfault in cuckoo filter serialization

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P2)

Description

[Wesley Dawson [:whd]]

Attachment: hindsight.tar.gz

With the latest packages I see segfaults when restarting hindsight, presumably when serializing large bloom filters:

[427804.630654] hindsight[32750]: segfault at 7f4fe104b0d0 ip 00007f4fe104b0d0 sp 00007f4edfffe8d8 error 14
[427804.630716]  in cjson.so[7f4fe1265000+6000]
[427804.645008]  in cjson.so[7f4fe1265000+6000]
[427853.527252] hindsight[1051]: segfault at 7f90dc3d70d0 ip 00007f90dc3d70d0 sp 00007f9064ffb8d8 error 14
[427853.527259] hindsight[1052]: segfault at 7f90dc3d70d0 ip 00007f90dc3d70d0 sp 00007f903e6cc8d8 error 14
[427853.527261] hindsight[1054]: segfault at 7f90dc3d70d0 ip 00007f90dc3d70d0 sp 00007f903d6ca8d8 error 14
[427853.527265] hindsight[1055]: segfault at 7f90dc3d70d0 ip 00007f90dc3d70d0 sp 00007f903cec98d8 error 14
[427960.587078] hindsight[1317]: segfault at 7f7dee1220d0 ip 00007f7dee1220d0 sp 00007f7d69ffd8d8 error 14
[427960.587086] hindsight[1319]: segfault at 7f7dee1220d0 ip 00007f7dee1220d0 sp 00007f7d68ffb8d8 error 14
[427960.587090]  in libnss_dns-2.17.so[7f7deed45000+5000]
[427960.596232]  in libnss_dns-2.17.so[7f7deed45000+5000]

I've attached a hindsight config that can reproduce the problem. Some of the inputs aren't set to preserve data but if I remove them the segfault seems harder to trigger.

Comment 1

[Mike Trinkala [:trink]]

The working theory is that a malloc call is failing (since the cuckoo filters consume a lot of memory and double it during the deserialization process) and the returned NULL pointer is being written to. All the mallocs in the libluasandbox (no issues) and Hindsight have been audited (three untested malloc returns, now fixed).  However, even if these were the cause of the segfault the result would now be an error being logged and Hindsight exiting (so basically the same end result). If this is the case, it is not an issue with the cuckoo filter but any large plugin state that has been serialized.  However, I am still unable to reproduce it even on smaller instance type.

Comment 2

[Mike Trinkala [:trink]]

Fixed the error logging.  However, if malloc is failing and we need HUGE plugins with state preservation then we will have to redesign serialization.  The best solution, for now, is to turn off preservation (since when we crash we lose the state anyway and the duplicate rate isn't adversely affected).  The main reason we are using preservation on these plugins is due to the fact they are restarted to load the new pipeline schemas.  If something like this https://github.com/mozilla-services/lua_sandbox/issues/218 was implemented we could avoid the extra restarts.