Closed Bug 679063 Opened 10 years ago Closed 10 years ago

need better handling of ldap.SERVER_DOWN

Categories

(Cloud Services :: Server: Core, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: petef, Assigned: tarek)

References

Details

(Whiteboard: [qa+])

Attachments

(1 file)

In ldapconnection:_bind, we retry when we get ldap.SERVER_DOWN. But in the case of the server actually going away, it just keeps throwing ldap.SERVER_DOWN.  In this case, we need to call conn.unbind_s() to release the old connection, and re-connect.

Repro:
* start server, make a couple calls that will pool an ldap connection or two
* restart LDAP server
* try to make requests to server (results in a BackendError).
Assignee: nobody → tarek
Blocks: 680379
This patch will unbind if the last attempt resulted in a SERVER_DOWN error
Attachment #557813 - Flags: review?(rtilder)
Attachment #557813 - Flags: feedback?(petef)
Attachment #557813 - Flags: review?(rtilder) → review+
Comment on attachment 557813 [details] [diff] [review]
unbind on SERVER_DOWN

I'll test this in test & stage environments next week.
Attachment #557813 - Flags: feedback?(petef) → feedback+
pushed at https://hg.mozilla.org/services/server-core/rev/f07282b9faf3

Pete, I'll let you test & close this bug -- Thanks
Assignee: tarek → petef
tarek, afaict, ldappool.py isn't deployed to a basic sync-storage webhead.

[petef@sync1.web.scl2.stage.svc ~]$ find /usr/lib/python2.6 -name ldappool.py
[petef@sync1.web.scl2.stage.svc ~]$
Assignee: petef → tarek
Blocks: 692452
reapplied at http://hg.mozilla.org/services/server-core/rev/e4e643dc51a6 for 2.6.1

we should never ever have duplicate modules in our projects. This is pure insanity for maintenance. Please never again. branches and versions ftw
still not fixed in the backported 2.6.x version. to repro:

* send load to frontend
* stop load
* restart ldap slave
* sleep(30)
* start load

and webheads throw this exception (just a couple times, then recover and get a new working connection):
2011-10-11 10:41:50,099 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 134, in authenticate_user
    user_id = self.backend.authenticate_user(user_name, password)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 255, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 164, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 144, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 272, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 216, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 150, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 190, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

{'info': '', 'desc': "Can't contact LDAP server"}
new exception after trying your patch from irc:

2011-10-11 12:23:15,873 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 134, in authenticate_user
    user_id = self.backend.authenticate_user(user_name, password)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 255, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 164, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 144, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 272, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 216, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 150, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 183, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

LDAP connection invalid
new version based on load tests - http://hg.mozilla.org/services/server-core/rev/0c4eba7cb621
note to myself - need it backport it to default
validated by pete, now on its own http://pypi.python.org/pypi/ldappool/
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [qa+]
OK. Marking as Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.