668520 - one missing retry on a timeout

Status: RESOLVED FIXED

(Cloud Services :: Server: Core, defect)

Description

[Tarek Ziadé (:tarek)]

The ldap connector seem to miss a retry after the ldap server goes idle. note that everything's self healing after that call.



    2011-06-29 22:17:36,938 ERROR [syncserver] 116bb6c96ea884b2d793f2bc03a2eaf2
    2011-06-29 22:17:36,938 ERROR [syncserver] Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/services/util.py", line 500, in __call__
        return self.app(environ, start_response)
      File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
        return self.application(environ, replacement_start_response)
      File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
        resp = self.call_func(req, *args, **self.kwargs)
      File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
        return self.func(req, *args, **kwargs)
      File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 161, in __notified
        response = func(self, request)
      File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 206, in __call__
        self.auth.check(request, match)
      File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
        match.get('username'))
      File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 129, in authenticate_user
        user_id = self.backend.authenticate_user(user_name, password)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 250, in authenticate_user
        dn = self._username2dn(user_name)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 162, in _username2dn
        return self._get_dn_by_filter('(uid=%s)' % user_name)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 142, in _get_dn_by_filter
        with self._conn() as conn:
      File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
        return self.gen.next()
      File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 243, in connection
        conn = self._get_connection(bind, passwd)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 192, in _get_connection
        conn = self._match(bind, passwd)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 126, in _match
        self._bind(conn, bind, passwd)
      File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 166, in _bind
        raise BackendError(str(e))
    BackendError: None

Comment 1

[Tarek Ziadé (:tarek)]

Attachment: retry on timeout

Now any timeout will be handled like a server down error: it will be retried n time before failure.

Comment 2

[Ryan Tilder [:rtilder]]

Comment on attachment 557809 [details] [diff] [review]
retry on timeout

Review of attachment 557809 [details] [diff] [review]:
-----------------------------------------------------------------

::: services/ldappool.py
@@ +146,3 @@
>                      # the server seems down, we can retry
>                      time.sleep(self.retry_delay)
>                      tries += 1

There's no feedback for if/when tries >= self.retry_max even when the state of connected is checked right after this loop. Might be good to add it while making changes to this block anyway. See next comment.

@@ +162,1 @@
>                  else:

Somewhere in this "if not connected:" block we should be checking to see if self.retry_max has been exceeded.

Comment 3

[Tarek Ziadé (:tarek)]

(In reply to Ryan Tilder [:rtilder] from comment #2)
...
> Somewhere in this "if not connected:" block we should be checking to see if
> self.retry_max has been exceeded.

Right now the output is a successful or unsuccessful connection, no matter how many attempts where made. For what reason do you want to check if retry_max has been reached in the 'not connected' block ?

Comment 4

[Ryan Tilder [:rtilder]]

(In reply to Tarek Ziadé (:tarek) from comment #3)
> 
> Right now the output is a successful or unsuccessful connection, no matter
> how many attempts where made. For what reason do you want to check if
> retry_max has been reached in the 'not connected' block ?
>

Logging purposes, really.  It's a specific case we're testing for so we might as well report it to the operator of the sync server.

Comment 5

[Tarek Ziadé (:tarek)]

pushed in https://hg.mozilla.org/services/server-core/rev/7f95019a6cb9

Created Bug 684685 for the logging improvements