Closed Bug 1177251 Opened 8 years ago Closed 8 years ago

release-sanity timed out talking to hg.m.o


(Release Engineering :: Release Automation: Other, defect)

Not set


(Not tracked)



(Reporter: rail, Assigned: nthomas)



(1 file)

Probably should retry harder.

2015-06-24 13:16:50,759 : INFO : Checking for existence l10n changeset bn-IN e4dd4d7f827a in repo ...
2015-06-24 13:16:50,979 : INFO : Checking for existence l10n changeset br bcf57cc518e0 in repo ...
2015-06-24 13:16:51,170 : INFO : Checking for existence l10n changeset bs 8fc6fcbbced3 in repo ...
Traceback (most recent call last):
  File "tools/buildbot-helpers/", line 461, in <module>
  File "tools/buildbot-helpers/", line 151, in verify_l10n_changesets
    sleeptime=1, retry_exceptions=(urllib2.HTTPError,))
  File "/builds/releaserunner/tools/buildfarm/release/tools/lib/python/vendor/redo-1.4.1/redo/", line 152, in retry
    return action(*args, **kwargs)
  File "tools/buildbot-helpers/", line 123, in get_l10n_changesets
  File "/tools/python27/lib/python2.7/", line 126, in urlopen
    return, data, timeout)
  File "/tools/python27/lib/python2.7/", line 400, in open
    response = self._open(req, data)
  File "/tools/python27/lib/python2.7/", line 418, in _open
    '_open', req)
  File "/tools/python27/lib/python2.7/", line 378, in _call_chain
    result = func(*args)
  File "/tools/python27/lib/python2.7/", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/tools/python27/lib/python2.7/", line 1180, in do_open
    r = h.getresponse(buffering=True)
  File "/tools/python27/lib/python2.7/", line 1030, in getresponse
  File "/tools/python27/lib/python2.7/", line 407, in begin
    version, status, reason = self._read_status()
  File "/tools/python27/lib/python2.7/", line 365, in _read_status
    line = self.fp.readline()
  File "/tools/python27/lib/python2.7/", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/tools/python27/lib/python2.7/", line 241, in recv
  File "/tools/python27/lib/python2.7/", line 160, in read
socket.error: [Errno 110] Connection timed out
2015-06-24 13:32:22,998 - INFO - command: ERROR
Traceback (most recent call last):
  File "/builds/releaserunner/tools/lib/python/util/", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'tools/buildbot-helpers/', '--branch', 'mozilla-esr38', '--username', '', '--version', '38.1.0esr', '--build-number', '1', '--release-config', '', '--skip-verify-configs', '--masters-json-file', '', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--bypass-l10n-dashboard-check', '--dry-run']' returned non-zero exit status 1
2015-06-24 13:32:23,006 - INFO - command: END (942.13s elapsed)
 - connection successful on retry
 - no network issues know to #moc at this time

Checking with #webops to confirm this is issue with zlb. I believe the socket connect error is to zlb, which will fully open the connection before checking the back end, as zlb returns a valid http 5xx code if backend is unavailable.
Hit this again today with 40.0b6:
2015-07-20 17:08:11,692 : INFO : Checking for existence l10n changeset es-AR 86a1a98fd56d in repo ...
2015-07-20 17:08:11,974 : INFO : Checking for existence l10n changeset es-CL f7560219a7bf in repo ...
2015-07-20 17:08:12,132 : INFO : Checking for existence l10n changeset es-ES c165563f84b3 in repo ...

Gonna patch release-runner to add timeouts.
Assignee: nobody → nthomas
Attachment #8636412 - Flags: review?(bhearsum)
Attachment #8636412 - Flags: review?(bhearsum) → review+
Deployed to production; from tools rev 47804cd5cf4b.
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.