1190422 - Better error handling of HTTP 500 errors in funsize scheduler

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

Balrog sometimes returns 500 errors, probably should retry on those.

Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler 2015-08-03 08:33:59,372 - funsize.worker - ERROR - Failed to process message
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler Traceback (most recent call last):
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/funsize/worker.py", line 105, in process_message
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     self.dispatch_message(body)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/funsize/worker.py", line 145, in dispatch_message
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     locale=locale, revision=properties["revision"])
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/funsize/worker.py", line 185, in create_partial
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     platform, locale)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/funsize/balrog.py", line 52, in get_build
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     req = requests.get(url, auth=self.auth, verify=self.verify)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/requests/api.py", line 60, in get
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     return request('get', url, **kwargs)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/requests/api.py", line 49, in request
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     return session.request(method=method, url=url, **kwargs)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     resp = self.send(prep, **send_kwargs)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     r = adapter.send(request, **kwargs)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler   File "/builds/funsize/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler     raise ConnectionError(err, request=request)
Aug 03 08:33:59 buildbot-master89.bb.releng.scl3.mozilla.com python: funsize_scheduler ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Comment 1

[Rail Aliiev [:rail]]

Attachment: retry on 500

Comment 2

[Rail Aliiev [:rail]]

merged