Closed Bug 774810 Opened 13 years ago Closed 13 years ago

dev-token.services.mozilla.com not responding to HTTP requests

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rmiller, Assigned: jlaz)

Details

(Whiteboard: [qa+])

I get the following when I try to run the AitC functional tests in the dev environment, seems to show the dev-token server not responding to authentication requests: spire git:master ~/venv/server-aitc $ ./bin/python aitc/tests/functional/test_aitc.py --use-token-server https://dev-token.services.mozilla.com/1.0/aitc/1.0 Traceback (most recent call last): File "aitc/tests/functional/test_aitc.py", line 432, in <module> res = run_live_functional_tests(TestAITC, sys.argv) File "/Users/rob/venv/server-aitc/deps/server-syncstorage/syncstorage/tests/functional/support.py", line 170, in run_live_functional_tests creds = authenticate_to_token_server(url, opts.email, opts.audience) File "/Users/rob/venv/server-aitc/deps/server-syncstorage/syncstorage/tests/functional/support.py", line 112, in authenticate_to_token_server "Authorization": "Browser-ID " + assertion, File "/Users/rob/venv/server-aitc/lib/python2.6/site-packages/requests/api.py", line 54, in get return request('get', url, **kwargs) File "/Users/rob/venv/server-aitc/lib/python2.6/site-packages/requests/safe_mode.py", line 37, in wrapped return function(method, url, **kwargs) File "/Users/rob/venv/server-aitc/lib/python2.6/site-packages/requests/api.py", line 42, in request return s.request(method=method, url=url, **kwargs) File "/Users/rob/venv/server-aitc/lib/python2.6/site-packages/requests/sessions.py", line 230, in request r.send(prefetch=prefetch) File "/Users/rob/venv/server-aitc/lib/python2.6/site-packages/requests/models.py", line 609, in send raise ConnectionError(e) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='dev-token.services.mozilla.com', port=443): Max retries exceeded with url: /1.0/aitc/1.0
Repeated test above from qa1. I get all failures for this testing, but see something a bit different... $ ./bin/python aitc/tests/functional/test_aitc.py --use-token-server https://dev-token.services.mozilla.com/1.0/aitc/1.0 EEEEEEEEEEEEEEEEEEE ====================================================================== ERROR: test_creating_an_app_with_x_if_unmodified_since_equal_to_zero (syncstorage.tests.functional.support.LiveTestCases) ---------------------------------------------------------------------- Traceback (most recent call last): File "aitc/tests/functional/test_aitc.py", line 63, in setUp apps = self.app.get(self.root + "/apps/").json["apps"] File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 749, in get expect_errors=expect_errors) File "/opt/aitc1/server-aitc/deps/server-syncstorage/syncstorage/tests/functional/support.py", line 39, in new_do_request return orig_do_request(req, *args, **kwds) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1062, in do_request self._check_status(status, res) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1098, in _check_status res) AppError: Bad response: 500 Internal Server Error (not 200 OK or 3xx redirect for https://dev-aitc1.services.mozilla.com/1.0/11212/apps/) '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8">\n<title>Service Unavailable</title>\n<style type="text/css">\nbody, p, h1 {\n font-family: Verdana, Arial, Helvetica, sans-serif;\n}\nh2 {\n font-family: Arial, Helvetica, sans-serif;\n color: #b10b29;\n}\n</style>\n</head>\n<body>\n<h2>Service Unavailable</h2>\n<p>The service is temporarily unavailable. Please try again later.</p>\n</body>\n</html>\n' Love the HTML by the way.... Every test failed with the same traceback and error information. Will dig around on the Dev token* boxes next...
Whiteboard: [qa+]
That 500 Server Error with a #b10b29 heading is a Zeus "no backend nodes available" error page, which means they stopped responding to health checks for a bit.
Dev token1 Nothing interesting in /var/log/token /var/log/nginx/error.log is full of these: 2012/07/17 13:25:58 [emerg] 5229#0: still could not bind() 2012/07/17 13:26:01 [emerg] 5232#0: bind() to 0.0.0.0:80 failed (98: Address already in use) 2012/07/17 13:26:01 [emerg] 5232#0: bind() to 0.0.0.0:80 failed (98: Address already in use) 2012/07/17 13:26:01 [emerg] 5232#0: bind() to 0.0.0.0:80 failed (98: Address already in use) 2012/07/17 13:26:01 [emerg] 5232#0: bind() to 0.0.0.0:80 failed (98: Address already in use) 2012/07/17 13:26:01 [emerg] 5232#0: bind() to 0.0.0.0:80 failed (98: Address already in use) /var/log/nginx/access*.log looks ok to me for 7/17/2012 (all 200s) Dev token2 and token3 Nothing interesting in /var/log/token Can not access /var/log/nginx/error.log (640 on token2 vs. 644 on token1) Dev zeus zlb1.mtv1.svc.mozilla.com zlb2.mtv1.svc.mozilla.com I do not have SSH access
Fixed by killing an older nginx process
Assignee: nobody → jlaz
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
OK, so we still see issues, similar to Comment 1. :RaFromBRC is poking around Dev AITC...
I think this is OK now.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.