slapd will occasionally segfault. This has been happening since 11-23 when we deployed to https://mozillians-dev.allizom.org/en-US/ During development, I noticed slapd will segfault for: * Being run in the forground and a SIG INT sent (Cntl-C) * If a file it relies on is changed, such as recompiling sasl-browserid and installing
Cleaned up syslogs of segfaults from today only. http://jason.pastebin.mozilla.org/1404677 I don't see anything interesting, but syslog isn't set to debug...
The segfault looks similar to the one I received during testing. Dec 13 16:29:18 mozillians1 kernel: slapd: segfault at 7f7d4a6d6680 ip 00007f7d4b1eba32 sp 00007fff95e17f78 error 4 in libc-2.12.so[7f7d4b16c000+186000] In my debugging the issue appeared to be from passing NULL to strlen() or a function that calls strlen(). The last gdb stack frame I got was for an optimized length intrinsic. Let me see if I can get the stack trace. If this is the case, the SASL library may be expecting the plugin to perform certain error checking.
(In reply to David Chan [:dchan] from comment #2) Do you recall the repro steps to cause a NULL input? I can bulletproof the code, but it would be great to repro before doing so.
Created attachment 581788 [details] valgrind output for kill segfault The attached valgrind output is created by: valgrind -v --leak-check=full --show-reachable=yes slapd -d 64 -f slapd.conf -h ldap://:1389 then sending `kill $pid`. This doesn't repro the mozillians-dev issue, but it is a known segfault mentioned earlier (Cntl-C).
Unable to repro with 30 concurrent requests of same assertion, same stale assertion, and same garbage assertion. Still digging.
I have access to mozillians-dev master slapd server. The segfaults look like shutdown segfaults, which llyod has identified a fix for in https://github.com/mozilla/sasl-browserid/issues/1 I'm now looking for patterns to explain why slapd was shutdown manually or via automated scripts (instead of web requests causing segfault). On Rackspace: We sent 1000~ concurrent requests with unique assertions/emails/password and couldn't repro segfault.
IT and I are pretty confident that: 1) There is a compatibility issue with sasl-browserid and doing replication over start-tls. 2) Segfaults were on server shutdown or restart, not serving traffic. #2 has been fixed and 1 will be fixed outside of this bug. There is a work-around in place for mozillians-dev for #1.
Bumping to verified per the passage of time, the [qa-] nature of the bug, and comment 7.