665927 - syncstorage is opening more than 1 ldap connection per worker

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[:Atoll]

using the config block below, each gunicorn worker child of the parent gunicorn process is opening more than one LDAP connection.  The LDAP pool size is 1, so this should not occur.  Some workers open as many as 18 workers, some as few as 2.  No workers open only 1 LDAP connection.

[auth]
backend = services.auth.ldapsql.LDAPAuth
ldapuri = ldap://ldap-slave.web.scl2.load.svc
sqluri = mysql://USERNAME:[e-mail]/weave0
ldap_timeout =  -1
ldap_use_pool = true
ldap_pool_size = 1
use_tls = false
bind_user = uid=binduser,ou=logins,dc=mozilla
bind_password = PASSWORD
pool_size = 10
pool_recycle = 3600
users_root = dc=mozilla
users_base_dn = ou=users,dc=mozilla
sreg_location =
sreg_scheme =
sreg_path =

Comment 1

[:Atoll]

[e-mail] sync]# while sleep 1; do date +%s; lsof -nP | grep :389 | cut -d\  -f3 | sort | uniq -c; echo; done
1308672283

1308672285

1308672286

1308672288
     1 23324
     1 23327
     1 23333
     1 23337

1308672290
     3 23323
     5 23324
     4 23325
     3 23326
     3 23327
     2 23328
     2 23329
     3 23330
     4 23331
     3 23332
     5 23333
     3 23334
    10 23335
     3 23336
     3 23337

1308672292
     5 23323
     9 23324
     4 23325
     5 23326
     5 23327
     5 23328
     6 23329
     5 23330
     6 23331
     6 23332
     5 23333
     5 23334
     5 23335
     4 23336
     5 23337

1308672293
     5 23323
    16 23327
     1 23328
     2 23330
     1 23331
     3 23332
     1 23333
     2 23334
     9 23336
     2 23337

1308672295
     1 23323
     2 23324
     2 23325
     4 23326
     3 23328
     7 23329
     4 23330
     3 23331
     5 23332
     5 23333
     7 23334
     4 23335
     6 23336
     2 23337
     7 8637

1308672297
     3 23323
     5 23324
     5 23325
     5 23326
     3 23327
     6 23328
    10 23329
     8 23330
     5 23331
     9 23332
     7 23333
     7 23334
     5 23335
     2 23336
     3 23337
     7 8637

1308672299
     3 23323
     5 23324
    11 23325
     6 23326
     3 23327
     7 23328
    10 23329
     8 23330
     5 23331
     9 23332
     7 23333
     8 23334
     5 23335
     5 23336
     3 23337
     7 8637

1308672302
     3 23323
     5 23324
    11 23325
     5 23326
     3 23327
     7 23328
    10 23329
     8 23330
     5 23331
     9 23332
     7 23333
     8 23334
     5 23335
     5 23336
     3 23337
     7 8637

1308672304
     3 23323
     5 23324
    11 23325
     6 23326
     3 23327
     7 23328
    10 23329
     8 23330
     5 23331
     9 23332
     7 23333
     8 23334
     5 23335
     5 23336
     3 23337
     7 8637

Comment 2

[:Atoll]

In comment #1, the load test was started against a fresh gunicorn at 1308672286 and the number of LDAP connections per worker increased each second until we stopped the loadtest.

Comment 3

[Tarek Ziadé (:tarek)]

We tracked down the issue with Pete. I am going to push a temp fix so Pete can continue the load testing, and will work on a full fix + tests

Comment 4

[Tarek Ziadé (:tarek)]

Attachment: temporary patch to fix the pool issue

Comment 5

[Tarek Ziadé (:tarek)]

Attachment: rebinds patch

Comment 6

[Raymond Forbes[:rforbes]]

I looked at the changes that change the connection behavior and I don't see any issues.  Infrasec is good with this.

Comment 7

[Tarek Ziadé (:tarek)]

Fixed in http://hg.mozilla.org/services/server-core/rev/6e2e30edfa44