slapd will occasionally segfault. This has been happening since 11-23 when we deployed to https://mozillians-dev.allizom.org/en-US/
During development, I noticed slapd will segfault for:
* Being run in the forground and a SIG INT sent (Cntl-C)
* If a file it relies on is changed, such as recompiling sasl-browserid and installing
Cleaned up syslogs of segfaults from today only.
I don't see anything interesting, but syslog isn't set to debug...
The segfault looks similar to the one I received during testing.
Dec 13 16:29:18 mozillians1 kernel: slapd: segfault at 7f7d4a6d6680 ip 00007f7d4b1eba32 sp 00007fff95e17f78 error 4 in libc-2.12.so[7f7d4b16c000+186000]
In my debugging the issue appeared to be from passing NULL to strlen() or a function that calls strlen(). The last gdb stack frame I got was for an optimized length intrinsic. Let me see if I can get the stack trace.
If this is the case, the SASL library may be expecting the plugin to perform certain error checking.
(In reply to David Chan [:dchan] from comment #2)
Do you recall the repro steps to cause a NULL input?
I can bulletproof the code, but it would be great to repro before doing so.
Created attachment 581788 [details]
valgrind output for kill segfault
The attached valgrind output is created by:
valgrind -v --leak-check=full --show-reachable=yes slapd -d 64 -f slapd.conf -h ldap://:1389
then sending `kill $pid`.
This doesn't repro the mozillians-dev issue, but it is a known segfault mentioned earlier (Cntl-C).
Unable to repro with 30 concurrent requests of same assertion, same stale assertion, and same garbage assertion.
I have access to mozillians-dev master slapd server.
The segfaults look like shutdown segfaults, which llyod has identified a fix for in https://github.com/mozilla/sasl-browserid/issues/1
I'm now looking for patterns to explain why slapd was shutdown manually or via automated scripts (instead of web requests causing segfault).
We sent 1000~ concurrent requests with unique assertions/emails/password and couldn't repro segfault.
IT and I are pretty confident that:
1) There is a compatibility issue with sasl-browserid and doing replication over start-tls.
2) Segfaults were on server shutdown or restart, not serving traffic.
#2 has been fixed and 1 will be fixed outside of this bug. There is a work-around in place for mozillians-dev for #1.
Bumping to verified per the passage of time, the [qa-] nature of the bug, and comment 7.