sasl-browserid causing segfault in slapd

VERIFIED FIXED

Status

Participation Infrastructure
Phonebook
VERIFIED FIXED
6 years ago
5 years ago

People

(Reporter: ozten, Unassigned)

Tracking

Details

(Whiteboard: [qa-])

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
slapd will occasionally segfault. This has been happening since 11-23 when we deployed to https://mozillians-dev.allizom.org/en-US/

During development, I noticed slapd will segfault for:
* Being run in the forground and a SIG INT sent (Cntl-C)
* If a file it relies on is changed, such as recompiling sasl-browserid and installing
(Reporter)

Updated

6 years ago
Blocks: 665373
(Reporter)

Comment 1

6 years ago
Cleaned up syslogs of segfaults from today only.
http://jason.pastebin.mozilla.org/1404677

I don't see anything interesting, but syslog isn't set to debug...

Comment 2

6 years ago
The segfault looks similar to the one I received during testing.

Dec 13 16:29:18 mozillians1 kernel: slapd[14391]: segfault at 7f7d4a6d6680 ip 00007f7d4b1eba32 sp 00007fff95e17f78 error 4 in libc-2.12.so[7f7d4b16c000+186000]

In my debugging the issue appeared to be from passing NULL to strlen() or a function that calls strlen(). The last gdb stack frame I got was for an optimized length intrinsic. Let me see if I can get the stack trace.

If this is the case, the SASL library may be expecting the plugin to perform certain error checking.
(Reporter)

Comment 3

6 years ago
(In reply to David Chan [:dchan] from comment #2)

Do you recall the repro steps to cause a NULL input?

I can bulletproof the code, but it would be great to repro before doing so.
(Reporter)

Comment 4

6 years ago
Created attachment 581788 [details]
valgrind output for kill segfault

The attached valgrind output is created by:

valgrind -v --leak-check=full --show-reachable=yes slapd -d 64 -f slapd.conf -h ldap://:1389

then sending `kill $pid`.

This doesn't repro the mozillians-dev issue, but it is a known segfault mentioned earlier (Cntl-C).
(Reporter)

Comment 5

6 years ago
Unable to repro with 30 concurrent requests of same assertion, same stale assertion, and same garbage assertion.

Still digging.
(Reporter)

Comment 6

6 years ago
I have access to mozillians-dev master slapd server.

The segfaults look like shutdown segfaults, which llyod has identified a fix for in https://github.com/mozilla/sasl-browserid/issues/1

I'm now looking for patterns to explain why slapd was shutdown manually or via automated scripts (instead of web requests causing segfault).

On Rackspace:
We sent 1000~ concurrent requests with unique assertions/emails/password and couldn't repro segfault.
(Reporter)

Comment 7

6 years ago
IT and I are pretty confident that:

1) There is a compatibility issue with sasl-browserid and doing replication over start-tls.
2) Segfaults were on server shutdown or restart, not serving traffic.

#2 has been fixed and 1 will be fixed outside of this bug. There is a work-around in place for mozillians-dev for #1.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Updated

5 years ago
Component: mozillians.org → Phonebook
Product: Websites → Community Tools
QA Contact: mozillians-org → phonebook
Version: unspecified → other

Updated

5 years ago
Whiteboard: [qa-]
Bumping to verified per the passage of time, the [qa-] nature of the bug, and comment 7.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.