674640 - BackendError: BackendError on localhost

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[:Atoll]

sync in test environment broke since yesterday.  on first sync attempt:

2011-07-27 12:40:56,253 ERROR [syncserver] d3c516343f459a555121f993f8030a53
2011-07-27 12:40:56,253 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 128, in authenticate_user
    environ.get('HTTP_HOST'))
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 250, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 162, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 142, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 243, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 192, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 126, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 166, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

Comment 1

[Toby Elliott [:telliott]]

The only way to get to the lines you reach is multiple ldap.SERVER_DOWN responses from the server.

Filing Bug 674648 to make that error message a lot better.

Comment 3

[Pete Fritchman [:petef]]

I think this exception also bubbles up if there is a stale (i.e. since terminated) connection in the ldap connection pool.

2011-08-04 23:02:37,900 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 194, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 255, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 129, in authenticate_user
    environ.get('HTTP_HOST'))
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 250, in authenticate_user
    dn = self._username2dn(user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 162, in _username2dn
    return self._get_dn_by_filter('(uid=%s)' % user_name)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 142, in _get_dn_by_filter
    with self._conn() as conn:
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 243, in connection
    conn = self._get_connection(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 192, in _get_connection
    conn = self._match(bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 126, in _match
    self._bind(conn, bind, passwd)
  File "/usr/lib/python2.6/site-packages/services/auth/ldapconnection.py", line 166, in _bind
    raise BackendError(str(e))
BackendError: BackendError on localhost

Comment 4

[:Atoll]

Another undiagnosable backtrace from account-portal, with the same error message.  Included below with new and current line numbers from production.  Raising priority since this bug is keeping us from resolving a-p crashes in production right now.  Tarek, when can we expect this to be resolved?

2011-08-21 17:41:12,422 ERROR [accountportal] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/beaker/middleware.py", line 152, in __call__
    return self.wrap_app(environ, session_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/accountportal/wsgiapp.py", line 178, in __call__
    result = function(request, **params)
  File "/usr/lib/python2.6/site-packages/accountportal/controllers/account.py", line 187, in delete_account
    if not self.auth.delete_user(request.user, pwd):
  File "/usr/lib/python2.6/site-packages/services/user/sreg.py", line 130, in delete_user
    raise BackendError()
BackendError: BackendError on localhost

Comment 5

[:Atoll]

Oops, I read back through the bug and the new stack trace is a-p, but the old traces are sync storage.

Tarek, should I file a separate bug, or are these all the same core library issue?

Comment 6

[Tarek Ziadé (:tarek)]

@Richard: it's a different issue.

The core lib has now ways to give more information when a BackendError() is raised, but it's up to every project to give extensive info when they raise.

So please, create a new bug on the corresponding project when you get such an error. Thanks!

Comment 7

[Tarek Ziadé (:tarek)]

(In reply to Richard Soderberg [:atoll] from comment #4)
> Another undiagnosable backtrace from account-portal, with the same error
> message.  Included below with new and current line numbers from production. 
> Raising priority since this bug is keeping us from resolving a-p crashes in
> production right now.  Tarek, when can we expect this to be resolved?
> 
> 2011-08-21 17:41:12,422 ERROR [accountportal] Traceback (most recent call
> last):
>   File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in
> __call__
>     return self.app(environ, start_response)
>   File "/usr/lib/python2.6/site-packages/beaker/middleware.py", line 152, in
> __call__
>     return self.wrap_app(environ, session_start_response)
>   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
>     resp = self.call_func(req, *args, **self.kwargs)
>   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in
> call_func
>     return self.func(req, *args, **kwargs)
>   File "/usr/lib/python2.6/site-packages/accountportal/wsgiapp.py", line
> 178, in __call__
>     result = function(request, **params)
>   File
> "/usr/lib/python2.6/site-packages/accountportal/controllers/account.py",
> line 187, in delete_account
>     if not self.auth.delete_user(request.user, pwd):
>   File "/usr/lib/python2.6/site-packages/services/user/sreg.py", line 130,
> in delete_user
>     raise BackendError()
> BackendError: BackendError on localhost

I have filed Bug 680882 for this one.

Comment 8

[Tarek Ziadé (:tarek)]

see ldappool