1417719 - gecko decision tasks failing on Try with ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out. (read timeout=5)

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try is closed for this.

https://public-artifacts.taskcluster.net/ZmkXwQXQR7GPo2s6R1RuNQ/0/public/logs/live_backing.log

[task 2017-11-15T23:28:48.743066Z] Querying version control for metadata: https://hg.mozilla.org/try/json-automationrelevance/7f8b9195ffb8158ee9eb2103288cccf5ead439a0
[task 2017-11-15T23:28:48.743120Z] attempt 1/2
[task 2017-11-15T23:28:48.743246Z] retry: calling get_automationrelevance, attempt #1
[task 2017-11-15T23:28:48.751118Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2017-11-15T23:28:53.878244Z] retry: Caught exception: 
[task 2017-11-15T23:28:53.878434Z] sleeping for 9.00s (attempt 1/2)
[task 2017-11-15T23:29:02.887788Z] attempt 2/2
[task 2017-11-15T23:29:02.887893Z] retry: calling get_automationrelevance, attempt #2
[task 2017-11-15T23:29:02.889131Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2017-11-15T23:29:07.919757Z] retry: Caught exception: 
[task 2017-11-15T23:29:07.919853Z] retry: Giving up on get_automationrelevance
[task 2017-11-15T23:29:07.919938Z] Traceback (most recent call last):
[task 2017-11-15T23:29:07.920009Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 165, in taskgraph_decision
[task 2017-11-15T23:29:07.921058Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-11-15T23:29:07.921139Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 128, in taskgraph_decision
[task 2017-11-15T23:29:07.921248Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-11-15T23:29:07.921350Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 213, in morphed_task_graph
[task 2017-11-15T23:29:07.921443Z]     return self._run_until('morphed_task_graph')
[task 2017-11-15T23:29:07.921545Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 319, in _run_until
[task 2017-11-15T23:29:07.921625Z]     k, v = self._run.next()
[task 2017-11-15T23:29:07.921722Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 306, in _run
[task 2017-11-15T23:29:07.921810Z]     existing_tasks=existing_tasks)
[task 2017-11-15T23:29:07.921916Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 55, in optimize_task_graph
[task 2017-11-15T23:29:07.921998Z]     do_not_optimize=do_not_optimize)
[task 2017-11-15T23:29:07.922097Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 126, in remove_tasks
[task 2017-11-15T23:29:07.922182Z]     if opt.should_remove_task(task, params, arg):
[task 2017-11-15T23:29:07.922279Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 353, in should_remove_task
[task 2017-11-15T23:29:07.922363Z]     changed = files_changed.check(params, file_patterns)
[task 2017-11-15T23:29:07.922465Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 58, in check
[task 2017-11-15T23:29:07.922549Z]     changed_files = get_changed_files(repository, revision)
[task 2017-11-15T23:29:07.922646Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 33, in get_changed_files
[task 2017-11-15T23:29:07.922738Z]     contents = retry(get_automationrelevance, attempts=2, sleeptime=10)
[task 2017-11-15T23:29:07.922829Z]   File "/builds/worker/checkouts/gecko/third_party/python/redo/redo/__init__.py", line 162, in retry
[task 2017-11-15T23:29:07.922899Z]     return action(*args, **kwargs)
[task 2017-11-15T23:29:07.923034Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 31, in get_automationrelevance
[task 2017-11-15T23:29:07.923136Z]     response = requests.get(url, timeout=5)
[task 2017-11-15T23:29:07.923236Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/api.py", line 67, in get
[task 2017-11-15T23:29:07.923328Z]     return request('get', url, params=params, **kwargs)
[task 2017-11-15T23:29:07.923433Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/api.py", line 53, in request
[task 2017-11-15T23:29:07.923521Z]     return session.request(method=method, url=url, **kwargs)
[task 2017-11-15T23:29:07.923635Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 468, in request
[task 2017-11-15T23:29:07.923717Z]     resp = self.send(prep, **send_kwargs)
[task 2017-11-15T23:29:07.923819Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 576, in send
[task 2017-11-15T23:29:07.923899Z]     r = adapter.send(request, **kwargs)
[task 2017-11-15T23:29:07.923998Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 449, in send
[task 2017-11-15T23:29:07.924079Z]     raise ReadTimeout(e, request=request)
[task 2017-11-15T23:29:07.924174Z] ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out. (read timeout=5)

Comment 1

[Gregory Szorc [:gps]]

Details in bug 1417720. fubar banned ActiveData temporarily. Service should be back to normal. We'll use the other bug to track follow-ups to ActiveData so this doesn't happen again.