710467 - slapd segfault causes slapd to not use ca-bundle.crt

Status: VERIFIED INVALID

(Participation Infrastructure :: Phonebook, defect)

Description

[Austin King [:ozten]]

As noted in Bug#710466 after a segfault, a slapd server will start to experience SSL Peer verification errors.

Example logs:
Dec 13 16:38:19 mozillians1 browserid-server: curl_easy_perform failed [60] Peer certificate cannot be authenticated with known CA certificates
Dec 13 16:38:19 mozillians1 browserid-server: No dice, STATUS=[curl-error] REASON=[Peer certificate cannot be authenticated with known CA certificates]

Comment 1

[Austin King [:ozten]]

@jason or @jabba: Do we have a system in place to restart slapd after a segfault, or is it in a weird post-crash state?

Comment 2

[Austin King [:ozten]]

(In reply to Austin King [:ozten] from comment #1)

via dchan - is it possible slapd gets restart as another user who can't read the ca-bundle.crt?

Comment 3

[Justin Dow [:jabba]]

Puppet will start slapd if it isn't running, but it will only check into it once per half hour.

Comment 4

[Austin King [:ozten]]

(In reply to Justin Dow [:jabba] from comment #3)
mozillians-dev is having the same backend error, which I'm guessing is the SSL issue. It seems puppet isn't sufficient for resetting to a working state.

The site does work after jason does a manual restart.

Can we tell puppet to do a restart even if slapd looks "up"?

Comment 5

[Justin Dow [:jabba]]

Not really. Puppet isn't the right tool for this. You are probably looking for something like monit to watch the process and restart it if problems arise.

Comment 6

[Pete Fritchman [:petef]]

Maybe it's an environment thing?

Can we capture this from a working slapd and a failing slapd?

> xargs -0 -n1 < /proc/$slapd_pid/environ | sort

Comment 7

[Austin King [:ozten]]

(In reply to Pete Fritchman [:petef] from comment #6)
$ sudo xargs -0 -n1 < /proc/18166/environ | sort
-bash: /proc/18166/environ: Permission denied

boo.

Comment 8

[Austin King [:ozten]]

Based on logs, I don't think that this was actually happening.

master ldap only had 2 SSL failures in it's logs. slapd was running for very little of this week.

Comment 9

[Austin King [:ozten]]

Causing segfault with kill -11 and restarting via
* puppet
* /etc/init.d/slapd restart
* services slapd restart
* sudo slapd

I was not able to reproduce this behavior.

Comment 10

[Rebecca Billings [:rbillings]]

Closing as [qa-]