1191478 - Set RETRY when the archiver connection times out

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Apologies if I got the summary wrong based on the failure in the log. Regardless, we see these from time to time and an auto-retry would be handy since it doesn't seem likely there's much we can do about them.

https://treeherder.mozilla.org/logviewer.html#?job_id=12531561&repo=mozilla-inbound

2015-08-05 12:20:49,544 truncating revision to first 12 chars
2015-08-05 12:20:49,545 Setting DEBUG logging.
2015-08-05 12:20:49,545 attempt 1/10
2015-08-05 12:20:49,545 Getting archive location from https://api.pub.build.mozilla.org/archiver/hgmo/integration/mozilla-inbound/a7860794b00e?&preferred_region=us-west-2&suffix=tar.gz&subdir=testing/mozharness
Traceback (most recent call last):
File "archiver_client.py", line 306, in <module>
main()
File "archiver_client.py", line 300, in main
archiver(api_url=api_url, config_key=config, options=options)
File "archiver_client.py", line 187, in archiver
response = get_url_response(api_url, options)
File "archiver_client.py", line 123, in get_url_response
response = urllib2.urlopen(api_url, timeout=60)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.7/socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
ssl.SSLError: The read operation timed out
program finished with exit code 1

Comment 1

[Jordan Lund (:jlund)]

Attachment: 150805_adds_sslerror_to_archiver_client_retry-tools.patch

I'm not sure the exact reason this fails but we should just retry from the client when we do get errors like this.

this patch should do that

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8644040 [details] [diff] [review]
150805_adds_sslerror_to_archiver_client_retry-tools.patch

looks good and safe to me

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8644040 [details] [diff] [review]
150805_adds_sslerror_to_archiver_client_retry-tools.patch

thanks! remote:   https://hg.mozilla.org/build/tools/rev/7ffff0e13950