710480 - core files, syslog debug setting, and version of sasl-browserid

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Austin King [:ozten]]

To troubleshoot Bug#710467 I need help with:

1) What is the SHA hash for the version of sasl-browserid we deployed to mozillians-dev slapd servers? Does git log contain ' 35a8bf2f307dc6ef578fa4496382ed92185bf494' which fixes a segfault dchan found?

2) Can we set the OS to make core dumps for slapd

3) Can we set syslog to DEBUG for slapd? (I'm not seeing all the possible logging)

Comment 1

[Brian Hourigan [:digi]]

Hi Austin:

I looked at this with Jabba on mozillians1.dev.db.phx1 earlier this morning. I found some issues that may be contributing to the bizarre behavior witnessed from slapd when libbrowserid.so is loaded.

We noticed that slapd is compiled with threading (RedHat default), but the shared library dependencies of your module aren't thread safe. I did some quick research and compiled my findings below.

- MySQL is thread safe when linked against libmysqlclient_r
- Curl is thread safe as long as you don't share curl handles
- cyrus-sasl is thread safe provided you meet specific criteria [1]
- yajl thread safety is unknown

I believe most of the problems you're seeing with slapd are related to thread safety and/or lack of bounds checking. For example once the module is unloaded from slapd syslog begins logging activity immediately. We verified slapd isn't hung by invoking it with -d 256 and observing activity logs printed to stdout.

May I suggest the use of valgrind [2] or similar debugging toolkits to eliminate issues with your code. I have also written wrappers for malloc()/free() that I would be happy to share with you. They implement bounds checking by header/footer checks on free and keep track of class allocations per __FILE__ and __LINE__.

[1] http://cyrusimap.web.cmu.edu/docs/cyrus-sasl/2.1.25/readme.php
[2] http://valgrind.org/

Comment 2

[Austin King [:ozten]]

(In reply to Brian Hourigan [:digi] from comment #1)
This is really helpful, thanks. 

> cyrus-sasl is thread safe provided you meet specific criteria [1]

Those criteria would be met by slapd, since it's using cyrus-sasl, right?
(mutex, calling server_init, etc)

I'll keep digging for sasl plugin thread model notes.

I'll review the sasl-browserid code from a multi-thread perspective.

> May I suggest the use of valgrind [2] or similar debugging toolkits to eliminate issues with your code.

valgrind was used during development. Can we run mozillians-dev slapd servers under valgrind while troubleshooting? Valgrind output and core dumps would help me.

> I have also written wrappers for malloc()/free() that I would be happy to share with you.

Great, can you point me to your code? Currently we use sasl's wrappers for malloc and free.

I've setup a RHEL 64 bit machine and will try to repro thread issues under valgrind.

Comment 3

[Brian Hourigan [:digi]]

Attachment: Functions for statmalloc()/statfree() used in personal projects

Comment 4

[Brian Hourigan [:digi]]

Hi Austin:

I've attached the C code I use for statmalloc() and statfree(). I'm lacking documentation on this, but it's pretty straight forward. I would be happy to help answer any questions you may have about it's usage.

I don't see any problems running slapd on mozillians-dev under valgrind. In fact, I had asked :lerxst to make a snapshot of the VM before I started mucking with it as I had intended to rebuild everything with -g3.

I would encourage you to do whatever you need to do to facilitate debugging. We can always restore from a known state at any time.

Comment 5

[Austin King [:ozten]]

(In reply to Brian Hourigan [:digi] from comment #1)
> - yajl thread safety is unknown

yajl is thread safe and heavily tested in concurrent environments

Comment 6

[Justin Dow [:jabba]]

I think we are good here.