628972 - 500 error on available_nodes exhaustion?

Status: RESOLVED FIXED

(Cloud Services :: Server: Registration, defect)

Description

[:Atoll]

found this in the sreg traceback


@400000004d4026d303092efc 2011-01-26 05:51:05,050 INFO  [syncserver] 10.250.64.1, 10.250.64.84, 10.250.64.78 - - [26/Jan/2011:05:51:05 -0700] "PUT /user/1
/weaveunittest_gimqfjerdr HTTP/1.0" 200 24 "-" "Python-urllib/2.6"
@400000004d4026d3030936cc 10.250.64.1, 10.250.64.84, 10.250.64.78 - - [26/Jan/2011:05:51:05 -0700] "PUT /user/1/weaveunittest_gimqfjerdr HTTP/1.0" 200 24 
"-" "Python-urllib/2.6"
@400000004d4026d3062c026c 2011-01-26 05:51:05,102 DEBUG [syncserver] Unable to get a node for user id: 224
@400000004d4026d3062c0654 Unable to get a node for user id: 224
@400000004d4026d3062c0a3c 2011-01-26 05:51:05,103 ERROR [syncserver] Traceback (most recent call last):
@400000004d4026d3062c0a3c   File "/usr/lib/python2.6/site-packages/services/util.py", line 424, in __call__
@400000004d4026d3062c0e24     return self.app(environ, start_response)
@400000004d4026d3062c0e24   File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
@400000004d4026d3062c297c     return self.application(environ, replacement_start_response)
@400000004d4026d3062c297c   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
@400000004d4026d3062c2d64     resp = self.call_func(req, *args, **self.kwargs)
@400000004d4026d3062c2d64   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
@400000004d4026d3062c314c     return self.func(req, *args, **kwargs)
@400000004d4026d3062c3534   File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 167, in __call__
@400000004d4026d3062c391c     result = function(request, **params)
@400000004d4026d3062c391c   File "/usr/lib/python2.6/site-packages/syncreg/controllers/user.py", line 94, in user_node
@400000004d4026d3062c3d04     location = self.auth.get_user_node(user_id)
@400000004d4026d3062c3d04   File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 510, in get_user_node
@400000004d4026d3062c48bc     raise NodeAttributionError(user_id)
@400000004d4026d3062c48bc NodeAttributionError: 224
@400000004d4026d3062c4ca4 
@400000004d4026d3062c4ca4 Traceback (most recent call last):
@400000004d4026d3062c4ca4   File "/usr/lib/python2.6/site-packages/services/util.py", line 424, in __call__
@400000004d4026d3062c508c     return self.app(environ, start_response)
@400000004d4026d3062c508c   File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
@400000004d4026d3062c585c     return self.application(environ, replacement_start_response)
@400000004d4026d3062c585c   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
@400000004d4026d3062c5c44     resp = self.call_func(req, *args, **self.kwargs)
@400000004d4026d3062c5c44   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
@400000004d4026d3062c602c     return self.func(req, *args, **kwargs)
@400000004d4026d30832cb04   File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 167, in __call__
@400000004d4026d30832ceec     result = function(request, **params)
@400000004d4026d30832d2d4   File "/usr/lib/python2.6/site-packages/syncreg/controllers/user.py", line 94, in user_node
@400000004d4026d30832d2d4     location = self.auth.get_user_node(user_id)
@400000004d4026d30832d6bc   File "/usr/lib/python2.6/site-packages/services/auth/ldapsql.py", line 510, in get_user_node
@400000004d4026d308330984     raise NodeAttributionError(user_id)
@400000004d4026d308330984 NodeAttributionError: 224

and the accompanying reg log entry, perhaps


current:@400000004d4026d30651a724 2011-01-26 05:51:05,105 INFO  [syncserver] 10.250.64.1, 10.250.64.84 - - [26/Jan/2011:05:51:05 -0700] "GET /user/1/weaveunittest_gimqfjerdr/node/weave HTTP/1.0" 500 37 "-" "Python-urllib/2.6"
current:@400000004d4026d30651ab0c 10.250.64.1, 10.250.64.84 - - [26/Jan/2011:05:51:05 -0700] "GET /user/1/weaveunittest_gimqfjerdr/node/weave HTTP/1.0" 500 37 "-" "Python-urllib/2.6"

Comment 1

[Tarek Ziadé (:tarek)]

That error is raise in case there are no more node available.

It should return a 404 here, even if the user exists.

Comment 2

[Tarek Ziadé (:tarek)]

Also, if the user does not exists, the 404 body is WEAVE_ERROR_INVALID_USERNAME

Comment 3

[Mike Connor [:mconnor]]

Let's make sure we get these semantics right.  404 if the user exists will break existing clients.  IIRC, PHP has these semantics (undocumented, because we suck at that):

200 + node = success
200 + null = no nodes for now, please try again later
404 = you're doing it wrong.

Toby, please confirm I'm not on crack. Also, please document this in the API docs!

Comment 4

[Tarek Ziadé (:tarek)]

Unless I read it wrong, I was describing the current PHP behavior from http://hg.mozilla.org/services/reg-server/file/2a73b7376e2c/1.0/weave_user/mozilla.php#l276

Comment 5

[Mike Connor [:mconnor]]

you're missing the piece at http://mxr.mozilla.org/services-central/source/reg-server/1.0/index.php#73 which returns "null" if get_user_location returns false

Comment 6

[Tarek Ziadé (:tarek)]

No I did not miss it, but I read it too quickly and misread the chaining - the Exception(WEAVE_ERROR_INVALID_USERNAME, 404) here seems like it's never called, so probably a historical leftover.

Anyways, I'll 200 - and that's what the doc says.

Comment 7

[Tarek Ziadé (:tarek)]

This is fixed at https://hg.mozilla.org/services/server-reg/file/tip/syncreg/controllers/user.py#l99  (also in sreg)

Comment 8

[:Atoll]

verified in staging today during a bonus QA pass, this fix will coincidentally ship with next Monday's train.  tagged [qa?] for tchung followup.

1306532101337   Net.Resource       DEBUG  GET success 200 https://stage-auth.services.mozilla.com/user/1.0/v55kxyyncmbt5j5ggh5pmdbsyx6jhjd4/node/weave
1306532101337   Service.Main       DEBUG  cluster value = null