1177251 - release-sanity timed out talking to hg.m.o

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Probably should retry harder.

2015-06-24 13:16:50,759 : INFO : Checking for existence l10n changeset bn-IN e4dd4d7f827a in repo https://hg.mozilla.org/releases/l10n/mozilla-release/bn-IN/file/e4dd4d7f827a ...
2015-06-24 13:16:50,979 : INFO : Checking for existence l10n changeset br bcf57cc518e0 in repo https://hg.mozilla.org/releases/l10n/mozilla-release/br/file/bcf57cc518e0 ...
2015-06-24 13:16:51,170 : INFO : Checking for existence l10n changeset bs 8fc6fcbbced3 in repo https://hg.mozilla.org/releases/l10n/mozilla-release/bs/file/8fc6fcbbced3 ...
Traceback (most recent call last):
  File "tools/buildbot-helpers/release_sanity.py", line 461, in <module>
    l10nRevisionFile):
  File "tools/buildbot-helpers/release_sanity.py", line 151, in verify_l10n_changesets
    sleeptime=1, retry_exceptions=(urllib2.HTTPError,))
  File "/builds/releaserunner/tools/buildfarm/release/tools/lib/python/vendor/redo-1.4.1/redo/__init__.py", line 152, in retry
    return action(*args, **kwargs)
  File "tools/buildbot-helpers/release_sanity.py", line 123, in get_l10n_changesets
    urllib2.urlopen(locale_url)
  File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/tools/python27/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/tools/python27/lib/python2.7/urllib2.py", line 1180, in do_open
    r = h.getresponse(buffering=True)
  File "/tools/python27/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/tools/python27/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/tools/python27/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/tools/python27/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/tools/python27/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/tools/python27/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
socket.error: [Errno 110] Connection timed out
2015-06-24 13:32:22,998 - INFO - command: ERROR
Traceback (most recent call last):
  File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-esr38', '--username', '[e-mail]', '--version', '38.1.0esr', '--build-number', '1', '--release-config', 'release-firefox-mozilla-esr38.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--bypass-l10n-dashboard-check', '--dry-run']' returned non-zero exit status 1
2015-06-24 13:32:23,006 - INFO - command: END (942.13s elapsed)

Comment 1

[Hal Wine [:hwine] use NI!]

Status:
 - connection successful on retry
 - no network issues know to #moc at this time

Checking with #webops to confirm this is issue with zlb. I believe the socket connect error is to zlb, which will fully open the connection before checking the back end, as zlb returns a valid http 5xx code if backend is unavailable.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Hit this again today with 40.0b6:
2015-07-20 17:08:11,692 : INFO : Checking for existence l10n changeset es-AR 86a1a98fd56d in repo https://hg.mozilla.org/releases/l10n/mozilla-beta/es-AR/file/86a1a98fd56d ...
2015-07-20 17:08:11,974 : INFO : Checking for existence l10n changeset es-CL f7560219a7bf in repo https://hg.mozilla.org/releases/l10n/mozilla-beta/es-CL/file/f7560219a7bf ...
2015-07-20 17:08:12,132 : INFO : Checking for existence l10n changeset es-ES c165563f84b3 in repo https://hg.mozilla.org/releases/l10n/mozilla-beta/es-ES/file/c165563f84b3 ...

Gonna patch release-runner to add timeouts.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Dust with 10s timeouts

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8636412 [details] [diff] [review]
[tools] Dust with 10s timeouts

https://hg.mozilla.org/build/tools/rev/7e1127a34df2

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Deployed to production; from tools rev 47804cd5cf4b.