700818 - test-auth returning 503 for meta/global

Status: RESOLVED FIXED

(Cloud Services :: Server: Core, defect)

Description

[Richard Newman [:rnewman]]

1320789918805	Sync.RecordManager	TRACE	Importing record: https://test-sync1.services.mozilla.com/1.1/aaxbccvqiluvsoutdiqrx3waszwfyrp7/storage/meta/global
1320789918858	Sync.Resource	DEBUG	mesg: GET fail 503 https://test-sync1.services.mozilla.com/1.1/aaxbccvqiluvsoutdiqrx3waszwfyrp7/storage/meta/global
1320789918859	Sync.Resource	DEBUG	GET fail 503 https://test-sync1.services.mozilla.com/1.1/aaxbccvqiluvsoutdiqrx3waszwfyrp7/storage/meta/global

Causing TPS failures.

Comment 1

[:Atoll]

Found this in the sync error logs.  Please escalate to services-eng for assistance.  It looks like the configs on sync2.web.mtv1.test.svc.m.c are no longer valid for the storage server codebase, and we're somehow not using LDAP anymore?

2011-11-08 14:06:00,543 ERROR [syncserver] e68730edad9271b47b6a042c3826fcd2
2011-11-08 14:06:00,544 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 495, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 191, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 254, in __call__
    self.auth.check(request, match)
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 66, in check
    match.get('username'))
  File "/usr/lib/python2.6/site-packages/services/wsgiauth.py", line 134, in authenticate_user
    user_id = self.backend.authenticate_user(user_name, password)
  File "/usr/lib/python2.6/site-packages/services/auth/sql.py", line 132, in authenticate_user
    user_name=user_name).fetchone()
  File "/usr/lib/python2.6/site-packages/services/util.py", line 635, in safe_execute
    raise BackendError(str(exc))
BackendError: BackendError on localhost

(OperationalError) no such table: users u'SELECT users.id, users.password_hash, users.status \nFROM users \nWHERE users.username = ?' ('aaxbccvqiluvsoutdiqrx3waszwfyrp7',)

Comment 2

[:Atoll]

Note that anyone with ssh access to a dev server that we run also should have access to the test CI environment.

For this specific task, the web server at:

sync2.web.mtv1.test.svc.mozilla.com

Is accessible through the office.m.o VPN and everyone should have sudo already to look at logs.

I expect that a config change will result from this task, let us know so that we can commit it to puppet.

Comment 3

[Richard Newman [:rnewman]]

Throwing this at Tarek.

Comment 4

[Toby Elliott [:telliott]]

Cleaning out old bugs. We presumably fixed this one.