1733419 - Intermittent ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=353255277&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BZsQAeQ0Tg-SMzPEYxh6wQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-09-30T15:53:38.434Z] 15:53:38     INFO -  INFO - ...failed to fetch 'tp5n.zip' from https://tooltool.mozilla-releng.net/
[task 2021-09-30T15:53:38.435Z] 15:53:38    ERROR -  Traceback (most recent call last):
[task 2021-09-30T15:53:38.435Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1318, in do_open
[task 2021-09-30T15:53:38.436Z] 15:53:38     INFO -      encode_chunked=req.has_header('Transfer-encoding'))
[task 2021-09-30T15:53:38.436Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1239, in request
[task 2021-09-30T15:53:38.436Z] 15:53:38     INFO -      self._send_request(method, url, body, headers, encode_chunked)
[task 2021-09-30T15:53:38.437Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1285, in _send_request
[task 2021-09-30T15:53:38.437Z] 15:53:38     INFO -      self.endheaders(body, encode_chunked=encode_chunked)
[task 2021-09-30T15:53:38.438Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1234, in endheaders
[task 2021-09-30T15:53:38.438Z] 15:53:38     INFO -      self._send_output(message_body, encode_chunked=encode_chunked)
[task 2021-09-30T15:53:38.439Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1026, in _send_output
[task 2021-09-30T15:53:38.439Z] 15:53:38     INFO -      self.send(msg)
[task 2021-09-30T15:53:38.439Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 964, in send
[task 2021-09-30T15:53:38.439Z] 15:53:38     INFO -      self.connect()
[task 2021-09-30T15:53:38.440Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1400, in connect
[task 2021-09-30T15:53:38.440Z] 15:53:38     INFO -      server_hostname=server_hostname)
[task 2021-09-30T15:53:38.440Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\ssl.py", line 407, in wrap_socket
[task 2021-09-30T15:53:38.441Z] 15:53:38     INFO -      _context=self, _session=session)
[task 2021-09-30T15:53:38.441Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\ssl.py", line 814, in __init__
[task 2021-09-30T15:53:38.441Z] 15:53:38     INFO -      self.do_handshake()
[task 2021-09-30T15:53:38.442Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\ssl.py", line 1068, in do_handshake
[task 2021-09-30T15:53:38.442Z] 15:53:38     INFO -      self._sslobj.do_handshake()
[task 2021-09-30T15:53:38.442Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\ssl.py", line 689, in do_handshake
[task 2021-09-30T15:53:38.443Z] 15:53:38     INFO -      self._sslobj.do_handshake()
[task 2021-09-30T15:53:38.443Z] 15:53:38     INFO -  ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
[task 2021-09-30T15:53:38.443Z] 15:53:38     INFO -  During handling of the above exception, another exception occurred:
[task 2021-09-30T15:53:38.444Z] 15:53:38    ERROR -  Traceback (most recent call last):
[task 2021-09-30T15:53:38.444Z] 15:53:38     INFO -    File "C:\Users\task_1633009644\mozharness\external_tools\tooltool.py", line 932, in fetch_file
[task 2021-09-30T15:53:38.444Z] 15:53:38     INFO -      with request(url, auth_file) as f, open(temp_path, mode="wb") as out:
[task 2021-09-30T15:53:38.445Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\contextlib.py", line 81, in __enter__
[task 2021-09-30T15:53:38.445Z] 15:53:38     INFO -      return next(self.gen)
[task 2021-09-30T15:53:38.445Z] 15:53:38     INFO -    File "C:\Users\task_1633009644\mozharness\external_tools\tooltool.py", line 911, in request
[task 2021-09-30T15:53:38.446Z] 15:53:38     INFO -      with closing(urllib2.urlopen(req)) as f:
[task 2021-09-30T15:53:38.446Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 223, in urlopen
[task 2021-09-30T15:53:38.446Z] 15:53:38     INFO -      return opener.open(url, data, timeout)
[task 2021-09-30T15:53:38.447Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 526, in open
[task 2021-09-30T15:53:38.447Z] 15:53:38     INFO -      response = self._open(req, data)
[task 2021-09-30T15:53:38.447Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 544, in _open
[task 2021-09-30T15:53:38.447Z] 15:53:38     INFO -      '_open', req)
[task 2021-09-30T15:53:38.448Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 504, in _call_chain
[task 2021-09-30T15:53:38.448Z] 15:53:38     INFO -      result = func(*args)
[task 2021-09-30T15:53:38.448Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1361, in https_open
[task 2021-09-30T15:53:38.448Z] 15:53:38     INFO -      context=self._context, check_hostname=self._check_hostname)
[task 2021-09-30T15:53:38.449Z] 15:53:38     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1320, in do_open
[task 2021-09-30T15:53:38.449Z] 15:53:38     INFO -      raise URLError(err)
[task 2021-09-30T15:53:38.450Z] 15:53:38     INFO -  urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)>
[task 2021-09-30T15:53:38.450Z] 15:53:38    ERROR -  ERROR - The following files failed: 'tp5n.zip'
[task 2021-09-30T15:53:38.483Z] 15:53:38    ERROR - Return code: 1
[task 2021-09-30T15:53:38.484Z] 15:53:38     INFO - retry: Failed, sleeping 60 seconds before retrying
[task 2021-09-30T15:54:38.492Z] 15:54:38     INFO - retry: Calling run_command with args: (['c:\\mozilla-build\\python3\\python3.exe', '-u', 'C:\\Users\\task_1633009644\\mozharness\\external_tools\\tooltool.py', 'fetch', '-m', 'C:\\Users\\task_1633009644\\build\\tests\\talos\\tp5n-pageset.manifest', '-o', '-c', 'c:\\build\\tooltool_cache'],), kwargs: {'cwd': 'C:\\Users\\task_1633009644\\build\\tests\\talos\\talos\\tests', 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': re.compile('Warning:.*Error: '), 'level': 'warning'}, {'regex': re.compile('package.*> Error:'), 'level': 'error'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': re.compile('raise \\w*Exception: '), 'level': 'critical'}, {'regex': re.compile('raise \\w*Error: '), 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'privileged': False, 'output_timeout': 600}, attempt #2

Comment 1

[Alexandru Michis [:malexandru]]

Also seems to be affecting browsertime jobs (log).

Comment 2

[Intermittent Failures Robot]

10 failures in 3376 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10

Platform and build breakdown:

• windows10-64-shippable-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1733419&startday=2021-09-27&endday=2021-10-03&tree=all

Comment 3

[Intermittent Failures Robot]

3 failures in 3030 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 3

Platform and build breakdown:

• windows10-64: 1
  • opt: 1
• windows10-64-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1733419&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 2723 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• windows10-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1733419&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.