679063 - need better handling of ldap.SERVER_DOWN

Status: VERIFIED FIXED

(Cloud Services :: Server: Core, defect)

Description

[Pete Fritchman [:petef]]

In ldapconnection:_bind, we retry when we get ldap.SERVER_DOWN. But in the case of the server actually going away, it just keeps throwing ldap.SERVER_DOWN.  In this case, we need to call conn.unbind_s() to release the old connection, and re-connect.

Repro:
* start server, make a couple calls that will pool an ldap connection or two
* restart LDAP server
* try to make requests to server (results in a BackendError).

Comment 1

[Tarek Ziadé (:tarek)]

Attachment: unbind on SERVER_DOWN

This patch will unbind if the last attempt resulted in a SERVER_DOWN error

Comment 2

[Pete Fritchman [:petef]]

Comment on attachment 557813 [details] [diff] [review]
unbind on SERVER_DOWN

I'll test this in test & stage environments next week.

Comment 3

[Tarek Ziadé (:tarek)]

pushed at https://hg.mozilla.org/services/server-core/rev/f07282b9faf3

Pete, I'll let you test & close this bug -- Thanks

Comment 4

[Pete Fritchman [:petef]]

tarek, afaict, ldappool.py isn't deployed to a basic sync-storage webhead.

[[e-mail] ~]$ find /usr/lib/python2.6 -name ldappool.py
[[e-mail] ~]$

Comment 5

[Tarek Ziadé (:tarek)]

reapplied at http://hg.mozilla.org/services/server-core/rev/e4e643dc51a6 for 2.6.1

we should never ever have duplicate modules in our projects. This is pure insanity for maintenance. Please never again. branches and versions ftw

Comment 6

[Pete Fritchman [:petef]]

still not fixed in the backported 2.6.x version. to repro:

* send load to frontend
* stop load
* restart ldap slave
* sleep(30)
* start load

and webheads throw this exception (just a couple times, then recover and get a new working connection):
2011-10-11 10:41:50,099 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 134, in authenticate_user
    user_id = self.backend.authenticate_user(user_name, password)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 255, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 164, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 144, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 272, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 216, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 150, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 190, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

{'info': '', 'desc': "Can't contact LDAP server"}

Comment 7

[Pete Fritchman [:petef]]

new exception after trying your patch from irc:

2011-10-11 12:23:15,873 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 134, in authenticate_user
    user_id = self.backend.authenticate_user(user_name, password)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 255, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 164, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 144, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 272, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 216, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 150, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 183, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

LDAP connection invalid

Comment 8

[Tarek Ziadé (:tarek)]

new version based on load tests - http://hg.mozilla.org/services/server-core/rev/0c4eba7cb621

Comment 9

[Tarek Ziadé (:tarek)]

note to myself - need it backport it to default

Comment 10

[Tarek Ziadé (:tarek)]

validated by pete, now on its own http://pypi.python.org/pypi/ldappool/

Comment 11

[James Bonacci [:jbonacci]]

OK. Marking as Verified.