1052916 - Retry bouncer submissions

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

During 32.0b3, some machines on the server side of bouncer were faulty (bug 1047190), so sometimes our API calls in the bouncer_submitter job would fail. Here's a sample error:

19:11:40     INFO - #####
19:11:40     INFO - ##### Running submit step.
19:11:40     INFO - #####
19:11:40     INFO - Running main action method: submit
19:11:40     INFO - Adding complete-mar...
19:11:40     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_add/
19:11:40     INFO - POST data: languages=ach&languages=af&language...=zu&product=Firefox-32.0b3-Complete
19:11:41     INFO - Server response
19:11:41     INFO - <?xml version="1.0" encoding="utf-8"?>....
19:11:41     INFO - Adding paths...
19:11:41     INFO - linux (linux): /firefox/releases/32.0b3/update/linux-i686/:lang/firefox-32.0b3.complete.mar
19:11:41     INFO - Submitting to https://bounceradmin.mozilla.com/api/location_add/
19:11:41     INFO - POST data: path=%2Ffirefox%2Freleases%2F32.0b3%2Fupdate%2Flinux-i686%2F%3Alang%2Ffirefox-32.0b3.complete.mar&product=Firefox-32.0b3-Complete&os=linux
19:12:14    FATAL - Uncaught exception: Traceback (most recent call last):
19:12:14    FATAL -   File "/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/mozharness/base/script.py", line 1237, in run
19:12:14    FATAL -     self.run_action(action)
19:12:14    FATAL -   File "/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/mozharness/base/script.py", line 1179, in run_action
19:12:14    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
19:12:14    FATAL -   File "/builds/slave/rel-m-beta-bncr_sub-0000000000/scripts/mozharness/base/script.py", line 1120, in _possibly_run_method
19:12:14    FATAL -     return getattr(self, method_name)()
19:12:14    FATAL -   File "scripts/scripts/bouncer_submitter.py", line 184, in submit
19:12:14    FATAL -     self.api_add_location(product_name, bouncer_platform, path)
19:12:14    FATAL -   File "scripts/scripts/bouncer_submitter.py", line 168, in api_add_location
19:12:14    FATAL -     self.api_call("location_add/", data)
19:12:14    FATAL -   File "scripts/scripts/bouncer_submitter.py", line 137, in api_call
19:12:14    FATAL -     res = urllib2.urlopen(request, timeout=60).read()
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
19:12:14    FATAL -     return _opener.open(url, data, timeout)
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 391, in open
19:12:14    FATAL -     response = self._open(req, data)
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
19:12:14    FATAL -     '_open', req)
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
19:12:14    FATAL -     result = func(*args)
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
19:12:14    FATAL -     return self.do_open(httplib.HTTPSConnection, req)
19:12:14    FATAL -   File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
19:12:14    FATAL -     r = h.getresponse()
19:12:14    FATAL -   File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
19:12:14    FATAL -     response.begin()
19:12:14    FATAL -   File "/usr/lib64/python2.6/httplib.py", line 391, in begin
19:12:14    FATAL -     version, status, reason = self._read_status()
19:12:14    FATAL -   File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
19:12:14    FATAL -     raise BadStatusLine(line)
19:12:14    FATAL - BadStatusLine
19:12:14    FATAL - Running post_fatal callback...
19:12:14    FATAL - Exiting -1

I'm not sure what code was being returned, and it might have been a timeout after 30 seconds. A retry of the failed post would have work well here.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

P3 but should be quick.

Comment 2

[Massimo Gervasini [:massimo]]

Attachment: [mozharness] Bug 1052916 - Retry bouncer submissions.patch

Added retry for api_call(), using the same mechanism provided by mozharness/base/script.py (_download_file). 

api_call() catches also the following exceptions: socket.timeout, socket.error and httplib.BadStatusLine which was raised in the log (comment #1)

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8498243 [details] [diff] [review]
[mozharness] Bug 1052916 - Retry bouncer submissions.patch

Review of attachment 8498243 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/bouncer_submitter.py
@@ +124,4 @@
>                              local_dict["tuxedoPassword"])
>          return self.credentials
>  
> +    def api_call(self, route, data, error_level, retry_config=None):

Might as well set error_level=FATAL in this definition, and leave it off the two calls later. r+ with that.

Comment 4

[Massimo Gervasini [:massimo]]

Comment on attachment 8498243 [details] [diff] [review]
[mozharness] Bug 1052916 - Retry bouncer submissions.patch

updated as requested in comment #3

Comment 5

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.