Closed Bug 1486224 Opened Last year Closed Last year

Intermittent urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

Categories

(Firefox Build System :: Task Configuration, task, P5)

Tracking

(firefox-esr60 fixed, firefox63 fixed, firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox-esr60 --- fixed
firefox63 --- fixed
firefox64 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: tomprince)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=195911812&repo=mozilla-central

https://queue.taskcluster.net/v1/task/etCKWvV6Txen0IsM1zQqDQ/runs/0/artifacts/public/logs/live_backing.log

Renaming to /builds/worker/fetches/target.tar.gz
Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1253, in connect
    super().connect()
  File "/usr/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 712, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 345, in <module>
    sys.exit(main())
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 341, in main
    return args.func(args)
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 307, in command_task_artifacts
    fetch_urls(downloads)
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 258, in fetch_urls
    f.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 237, in fetch_and_extract
    download_to_path(url, dest_path, sha256=sha256, size=size)
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 116, in download_to_path
    for chunk in stream_download(url, sha256=sha256, size=size):
  File "/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content", line 64, in stream_download
    with urllib.request.urlopen(url) as fh:
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 504, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 696, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 504, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 696, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
[fetches 2018-08-25T11:06:17.471Z] removing /builds/worker/fetches
[fetches 2018-08-25T11:06:17.480Z] finished
Traceback (most recent call last):
  File "/builds/worker/bin/run-task", line 754, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/builds/worker/bin/run-task", line 742, in main
    fetch_artifacts()
  File "/builds/worker/bin/run-task", line 496, in fetch_artifacts
    subprocess.run(cmd, check=True, env=os.environ)
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/builds/worker/workspace/build/src/taskcluster/scripts/misc/fetch-content', 'task-artifacts']' returned non-zero exit status 1
[taskcluster 2018-08-25 11:06:17.836Z] === Task Finished ===
[taskcluster 2018-08-25 11:06:17.972Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2018-08-25 11:06:18.060Z] Artifact "public/build/sq/target.complete.mar" not found at "/builds/worker/workspace/build/outputs/sq/target.complete.mar"
[taskcluster 2018-08-25 11:06:18.164Z] Artifact "public/build/sq/target.dmg" not found at "/builds/worker/workspace/build/outputs/sq/target.dmg"
[taskcluster 2018-08-25 11:06:18.741Z] Unsuccessful task run with exit code: 1 completed in 137.635 seconds
Product: Firefox Build System → Release Engineering
QA Contact: catlee
Flags: needinfo?(gps)
fetch-content definitely needs some retry logic. If it had some, 99.9% of these errors would likely go away.
Component: General → Task Configuration
Flags: needinfo?(gps)
Product: Release Engineering → Firefox Build System
QA Contact: catlee
Comment on attachment 9011532 [details]
Bug 1486224: [fetch-content] Retry downloads when fetching content; r?gps

Gregory Szorc [:gps] has approved the revision.
Attachment #9011532 - Flags: review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/569670a6d205
[fetch-content] Retry downloads when fetching content; r=gps
https://hg.mozilla.org/mozilla-central/rev/569670a6d205
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Assignee: nobody → mozilla
Duplicate of this bug: 1490709
See Also: → 1489598
You need to log in before you can comment on or make changes to this bug.