710466 - sasl-browserid causing segfault in slapd

Status: VERIFIED FIXED

(Participation Infrastructure :: Phonebook, defect)

Description

[Austin King [:ozten]]

slapd will occasionally segfault. This has been happening since 11-23 when we deployed to https://mozillians-dev.allizom.org/en-US/

During development, I noticed slapd will segfault for:
* Being run in the forground and a SIG INT sent (Cntl-C)
* If a file it relies on is changed, such as recompiling sasl-browserid and installing

Comment 1

[Austin King [:ozten]]

Cleaned up syslogs of segfaults from today only.
http://jason.pastebin.mozilla.org/1404677

I don't see anything interesting, but syslog isn't set to debug...

Comment 2

[David Chan [:dchan]]

The segfault looks similar to the one I received during testing.

Dec 13 16:29:18 mozillians1 kernel: slapd[14391]: segfault at 7f7d4a6d6680 ip 00007f7d4b1eba32 sp 00007fff95e17f78 error 4 in libc-2.12.so[7f7d4b16c000+186000]

In my debugging the issue appeared to be from passing NULL to strlen() or a function that calls strlen(). The last gdb stack frame I got was for an optimized length intrinsic. Let me see if I can get the stack trace.

If this is the case, the SASL library may be expecting the plugin to perform certain error checking.

Comment 3

[Austin King [:ozten]]

(In reply to David Chan [:dchan] from comment #2)

Do you recall the repro steps to cause a NULL input?

I can bulletproof the code, but it would be great to repro before doing so.

Comment 4

[Austin King [:ozten]]

Attachment: valgrind output for kill segfault

The attached valgrind output is created by:

valgrind -v --leak-check=full --show-reachable=yes slapd -d 64 -f slapd.conf -h ldap://:1389

then sending `kill $pid`.

This doesn't repro the mozillians-dev issue, but it is a known segfault mentioned earlier (Cntl-C).

Comment 5

[Austin King [:ozten]]

Unable to repro with 30 concurrent requests of same assertion, same stale assertion, and same garbage assertion.

Still digging.

Comment 6

[Austin King [:ozten]]

I have access to mozillians-dev master slapd server.

The segfaults look like shutdown segfaults, which llyod has identified a fix for in https://github.com/mozilla/sasl-browserid/issues/1

I'm now looking for patterns to explain why slapd was shutdown manually or via automated scripts (instead of web requests causing segfault).

On Rackspace:
We sent 1000~ concurrent requests with unique assertions/emails/password and couldn't repro segfault.

Comment 7

[Austin King [:ozten]]

IT and I are pretty confident that:

1) There is a compatibility issue with sasl-browserid and doing replication over start-tls.
2) Segfaults were on server shutdown or restart, not serving traffic.

#2 has been fixed and 1 will be fixed outside of this bug. There is a work-around in place for mozillians-dev for #1.

Comment 8

[Matt Brandt [:mbrandt]]

Bumping to verified per the passage of time, the [qa-] nature of the bug, and comment 7.