1770035 - Intermittent ERROR: Could not install packages due to an OSError: : Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known' | TypeError: warning() takes 2 positional arguments but 3 were given

Status: NEW

(Infrastructure & Operations :: RelOps: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=378524171&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/E15rt0zASh2BR3H09_4cAg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-05-18T17:49:14.837Z] 17:49:14     INFO -  Successfully installed vcversioner-2.16.0.0
[task 2022-05-18T17:49:14.880Z] 17:49:14     INFO - Return code: 0
[task 2022-05-18T17:49:14.880Z] 17:49:14     INFO - Installing psutil>=5.6.3 into virtualenv /opt/worker/tasks/task_165289543096576/build/venv
[task 2022-05-18T17:49:14.906Z] 17:49:14     INFO - find_links: connection checks passed for https://pypi.pub.build.mozilla.org/pub/, adding.
[task 2022-05-18T17:49:14.906Z] 17:49:14     INFO - retry: Calling run_command with args: [['/opt/worker/tasks/task_165289543096576/build/venv/bin/pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub/', 'psutil>=5.6.3']], kwargs: {'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': re.compile('\\d+: error: '), 'level': 'error'}, {'regex': re.compile('\\d+: warning: '), 'level': 'warning'}, {'regex': re.compile('Downloading .* \\(.*\\): *([0-9]+%)? *[0-9\\.]+[kmKM]b'), 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': re.compile('Warning:.*Error: '), 'level': 'warning'}, {'regex': re.compile('package.*> Error:'), 'level': 'error'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': re.compile('raise \\w*Exception: '), 'level': 'critical'}, {'regex': re.compile('raise \\w*Error: '), 'level': 'critical'}], 'cwd': '/opt/worker/tasks/task_165289543096576/build', 'env': {'SCCACHE_DISABLE': '1', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/autoland', 'MOZ_AUTOMATION': '1', 'RUN_ID': '0', 'TMPDIR': '/var/folders/41/jjgywl1j5y521hm6flvcmfq0000014/T/', 'LOGNAME': 'cltbld', 'LANG': 'en_US.UTF-8', 'MOZ_FETCHES_DIR': '/opt/worker/tasks/task_165289543096576/fetches', 'MOZ_SCM_LEVEL': '3', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.ZJOOA2QDve/Listeners', 'EXTRA_MOZHARNESS_CONFIG': '{"installer_url": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/P_YGjxqvS5Gb1XpwlJfT0g/artifacts/public/build/target.dmg", "test_packages_url": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/P_YGjxqvS5Gb1XpwlJfT0g/artifacts/public/build/target.test_packages.json"}', 'TASK_ID': 'E15rt0zASh2BR3H09_4cAg', 'USER': 'cltbld', 'SHLVL': '1', 'GECKO_HEAD_REV': '2bc2f5be60849fd02c087f130bc4126eb20bdba2', 'MOZ_FETCHES': '[{"artifact": "public/build/fix-stacks.tar.zst", "extract": true, "task": "RsHJxf26TJWCvfHGk4aR_A"}, {"artifact": "public/build/minidump-stackwalk.tar.zst", "extract": true, "task": "ILi5ZZ4-QpKht36ZXbUc9g"}]', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin', 'HOME': '/Users/cltbld', 'LC_ALL': 'en_US.UTF-8', 'PYTHON': '/usr/local/bin/python3', 'SHELL': '/bin/bash', 'PWD': '/opt/worker', 'XPC_FLAGS': '0x0', 'XPC_SERVICE_NAME': '0', '_': '/usr/local/bin/start-worker', 'MOZHARNESS_TEST_PATHS': '{"mochitest-browser-chrome": ["browser/base/content/test/alerts/browser.ini", "browser/base/content/test/static/browser.ini", "browser/components/newtab/test/browser/browser.ini", "browser/components/preferences/tests/browser.ini"]}', 'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com', 'MOZ_NODE_PATH': '/usr/local/bin/node', '__CF_USER_TEXT_ENCODING': '0x24:0x0:0x0'}, 'error_level': 'warning'}, attempt #1
[task 2022-05-18T17:49:14.906Z] 17:49:14     INFO - Running command: ['/opt/worker/tasks/task_165289543096576/build/venv/bin/pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub/', 'psutil>=5.6.3'] in /opt/worker/tasks/task_165289543096576/build
[task 2022-05-18T17:49:14.906Z] 17:49:14     INFO - Copy/paste: /opt/worker/tasks/task_165289543096576/build/venv/bin/pip install --timeout 120 --no-index --find-links https://pypi.pub.build.mozilla.org/pub/ psutil>=5.6.3
[task 2022-05-18T17:49:14.907Z] 17:49:14     INFO - Using env: (same as previous command)
[task 2022-05-18T17:49:15.208Z] 17:49:15     INFO -  Looking in links: https://pypi.pub.build.mozilla.org/pub/
[task 2022-05-18T17:49:16.049Z] 17:49:16     INFO -  Collecting psutil>=5.6.3
[task 2022-05-18T17:49:34.021Z] 17:49:34     INFO -    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.pub.build.mozilla.org', port=443): Read timed out. (read timeout=120.0)")': /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl
[task 2022-05-18T17:49:34.673Z] 17:49:34     INFO -    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fab8065db10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl
[task 2022-05-18T17:49:35.721Z] 17:49:35     INFO -    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fab8286e290>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl
[task 2022-05-18T17:49:37.820Z] 17:49:37     INFO -    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fab81010250>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl
[task 2022-05-18T17:49:41.881Z] 17:49:41     INFO -    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fab81010590>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl
[task 2022-05-18T17:49:41.882Z] 17:49:41     INFO -  ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='pypi.pub.build.mozilla.org', port=443): Max retries exceeded with url: /pub/psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fab810108d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
[task 2022-05-18T17:49:41.923Z] 17:49:41  WARNING - Return code: 1
[task 2022-05-18T17:49:41.924Z] 17:49:41  WARNING - Could not install python package: failed all attempts.
[task 2022-05-18T17:49:41.924Z] 17:49:41     INFO - Installing mozsystemmonitor==0.4 into virtualenv /opt/worker/tasks/task_165289543096576/build/venv
[task 2022-05-18T17:49:41.924Z] 17:49:41  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:49:52.037Z] 17:49:52  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:02.066Z] 17:50:02  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:12.130Z] 17:50:12  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:22.252Z] 17:50:22  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:32.296Z] 17:50:32  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:42.300Z] 17:50:42  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:50:52.301Z] 17:50:52  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:02.330Z] 17:51:02  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:12.375Z] 17:51:12  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:22.386Z] 17:51:22  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:32.389Z] 17:51:32  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:42.402Z] 17:51:42  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:51:52.503Z] 17:51:52  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:02.571Z] 17:52:02  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:12.713Z] 17:52:12  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:22.746Z] 17:52:22  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:32.880Z] 17:52:32  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:42.970Z] 17:52:42  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:52:53.075Z] 17:52:53  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:03.109Z] 17:53:03  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:13.140Z] 17:53:13  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:23.247Z] 17:53:23  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:33.303Z] 17:53:33  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:43.395Z] 17:53:43  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:53:53.531Z] 17:53:53  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:54:03.560Z] 17:54:03  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:54:13.697Z] 17:54:13  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:54:23.827Z] 17:54:23  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:54:33.921Z] 17:54:33  WARNING - find_links: dns check failed for pypi.pub.build.mozilla.org, sleeping 10s and retrying...
[task 2022-05-18T17:54:43.924Z] 17:54:43     INFO - Running post-action listener: _resource_record_post_action
[task 2022-05-18T17:54:43.924Z] 17:54:43     INFO - Running post-action listener: _start_resource_monitoring
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - Unable to start resource monitor: Traceback (most recent call last):
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/script.py", line 2292, in run_action
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/script.py", line 2244, in _possibly_run_method
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -     return getattr(self, method_name)()
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/python.py", line 595, in create_virtualenv
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -     editable=editable,
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/python.py", line 357, in install_module
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING -     ", but max retries reached. continuing...",
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - TypeError: warning() takes 2 positional arguments but 3 were given
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - 
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - During handling of the above exception, another exception occurred:
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - 
[task 2022-05-18T17:54:43.928Z] 17:54:43  WARNING - Traceback (most recent call last):
[task 2022-05-18T17:54:43.929Z] 17:54:43  WARNING -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/python.py", line 690, in _start_resource_monitoring
[task 2022-05-18T17:54:43.929Z] 17:54:43  WARNING -     from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
[task 2022-05-18T17:54:43.929Z] 17:54:43  WARNING - ModuleNotFoundError: No module named 'mozsystemmonitor'
[task 2022-05-18T17:54:43.929Z] 17:54:43     INFO - [mozharness: 2022-05-18 17:54:43.929088Z] Finished create-virtualenv step (failed)
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/script.py", line 2358, in run
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -     self.run_action(action)
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/script.py", line 2292, in run_action
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/script.py", line 2244, in _possibly_run_method
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -     return getattr(self, method_name)()
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/python.py", line 595, in create_virtualenv
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -     editable=editable,
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -   File "/opt/worker/tasks/task_165289543096576/mozharness/mozharness/base/python.py", line 357, in install_module
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL -     ", but max retries reached. continuing...",
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL - TypeError: warning() takes 2 positional arguments but 3 were given
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL - Running post_fatal callback...
[task 2022-05-18T17:54:43.929Z] 17:54:43    FATAL - Exiting -1
[task 2022-05-18T17:54:43.929Z] 17:54:43     INFO - Running post-run listener: _resource_record_post_run
[taskcluster 2022-05-18T17:54:43.951Z]    Exit Code: 255
[taskcluster 2022-05-18T17:54:43.951Z]    User Time: 10.663231s
[taskcluster 2022-05-18T17:54:43.951Z]  Kernel Time: 6.206193s
[taskcluster 2022-05-18T17:54:43.951Z]    Wall Time: 5m47.987828s
[taskcluster 2022-05-18T17:54:43.951Z]       Result: FAILED

Comment 1

[Mike Hommey [:glandium]]

Looks like an infra problem with pypi.pub.build.mozilla.org

Comment 2

[Intermittent Failures Robot]

3 failures in 2694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• macosx1015-64-qr: 3
  • opt: 2
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770035&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 3336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows7-32-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770035&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 1328 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• windows10-64-2004-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770035&startday=2022-12-26&endday=2023-01-01&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 3914 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770035&startday=2023-01-23&endday=2023-01-29&tree=all

Comment 6

[Intermittent Failures Robot]

2 failures in 4004 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 2

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1
• windows10-64-2004-mingwclang-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1770035&startday=2023-04-17&endday=2023-04-23&tree=all