1837647 - Perma Win7 runtests.py | Timed out while waiting for server startup.

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=418743854&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OQ3vNo7SRLOPjbkTvNtPQQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-06-09T14:33:32.660Z] 14:33:32     INFO - TEST-START | accessible/tests/browser/tree/browser_aria_owns.js
[task 2023-06-09T14:33:32.660Z] 14:33:32     INFO - TEST-SKIP | accessible/tests/browser/tree/browser_aria_owns.js | took 0ms
[task 2023-06-09T14:33:32.660Z] 14:33:32     INFO - Running manifest: accessible\tests\browser\bounds\browser.ini
[task 2023-06-09T14:33:32.660Z] 14:33:32     INFO - The following extra prefs will be set:
[task 2023-06-09T14:33:32.660Z] 14:33:32     INFO -   javascript.options.asyncstack_capture_debuggee_only=false
[task 2023-06-09T14:33:33.614Z] 14:33:33     INFO - PID 1132 | C:\Users\task_168631978783692\build\tests\bin\pk12util.exe: PKCS12 IMPORT SUCCESSFUL
[task 2023-06-09T14:33:33.771Z] 14:33:33     INFO - MochitestServer : launching ['C:\\Users\\task_168631978783692\\build\\tests\\bin\\xpcshell.exe', '-g', 'C:\\Users\\task_168631978783692\\build\\application\\firefox', '-f', 'C:\\Users\\task_168631978783692\\build\\tests\\bin\\components\\httpd.js', '-e', "const _PROFILE_PATH = 'C:\\\\Users\\\\task_168631978783692\\\\AppData\\\\Local\\\\Temp\\\\tmpkz6icwok.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', 'C:\\Users\\task_168631978783692\\build\\tests\\mochitest\\server.js']
[task 2023-06-09T14:33:33.771Z] 14:33:33     INFO - runtests.py | Server pid: 776
[task 2023-06-09T14:33:33.771Z] 14:33:33     INFO - runtests.py | Websocket server pid: 3780
[task 2023-06-09T14:33:33.771Z] 14:33:33     INFO - runtests.py | SSL tunnel pid: 3620
[task 2023-06-09T14:35:26.806Z] 14:35:26    ERROR - TEST-UNEXPECTED-FAIL | runtests.py | Timed out while waiting for server startup.
[task 2023-06-09T14:35:27.821Z] 14:35:27     INFO - Failed to stop web server on http://127.0.0.1:8888/server/shutdown
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1317, in do_open
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      encode_chunked=req.has_header('Transfer-encoding'))
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1229, in request
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      self._send_request(method, url, body, headers, encode_chunked)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1275, in _send_request
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      self.endheaders(body, encode_chunked=encode_chunked)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1224, in endheaders
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      self._send_output(message_body, encode_chunked=encode_chunked)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1016, in _send_output
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      self.send(msg)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 956, in send
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      self.connect()
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 928, in connect
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      (self.host,self.port), self.timeout, self.source_address)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\socket.py", line 727, in create_connection
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      raise err
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\socket.py", line 716, in create_connection
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      sock.connect(sa)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -  ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -  During handling of the above exception, another exception occurred:
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 615, in stop
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      with closing(urlopen(self.shutdownURL)) as c:
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 222, in urlopen
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      return opener.open(url, data, timeout)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 525, in open
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      response = self._open(req, data)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 543, in _open
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -      '_open', req)
[task 2023-06-09T14:35:27.837Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 503, in _call_chain
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -      result = func(*args)
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1345, in http_open
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -      return self.do_open(http.client.HTTPConnection, req)
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1319, in do_open
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -      raise URLError(err)
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO -  urllib.error.URLError: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO - Web server killed.
[task 2023-06-09T14:35:27.853Z] 14:35:27     INFO - Stopping web server
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO - Failed to stop web server on http://127.0.0.1:8888/server/shutdown
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 3592, in doTests
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self.startServers(options, debuggerInfo)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 1465, in startServers
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self.server.ensureReady(self.SERVER_STARTUP_TIMEOUT)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 611, in ensureReady
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      sys.exit(1)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  SystemExit: 1
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  During handling of the above exception, another exception occurred:
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1317, in do_open
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      encode_chunked=req.has_header('Transfer-encoding'))
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1229, in request
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self._send_request(method, url, body, headers, encode_chunked)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1275, in _send_request
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self.endheaders(body, encode_chunked=encode_chunked)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1224, in endheaders
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self._send_output(message_body, encode_chunked=encode_chunked)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 1016, in _send_output
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self.send(msg)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 956, in send
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      self.connect()
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\http\client.py", line 928, in connect
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      (self.host,self.port), self.timeout, self.source_address)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\socket.py", line 727, in create_connection
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      raise err
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\socket.py", line 716, in create_connection
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      sock.connect(sa)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  During handling of the above exception, another exception occurred:
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 615, in stop
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      with closing(urlopen(self.shutdownURL)) as c:
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 222, in urlopen
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      return opener.open(url, data, timeout)
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 525, in open
[task 2023-06-09T14:35:28.853Z] 14:35:28     INFO -      response = self._open(req, data)
[task 2023-06-09T14:35:28.868Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 543, in _open
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -      '_open', req)
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 503, in _call_chain
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -      result = func(*args)
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1345, in http_open
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -      return self.do_open(http.client.HTTPConnection, req)
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -    File "c:\mozilla-build\python3\lib\urllib\request.py", line 1319, in do_open
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -      raise URLError(err)
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO -  urllib.error.URLError: <urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO - Web server killed.
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO - Stopping web socket server
[task 2023-06-09T14:35:28.884Z] 14:35:28     INFO - Stopping ssltunnel
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 3592, in doTests
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      self.startServers(options, debuggerInfo)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 1465, in startServers
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      self.server.ensureReady(self.SERVER_STARTUP_TIMEOUT)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 611, in ensureReady
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      sys.exit(1)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -  SystemExit: 1
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -  During handling of the above exception, another exception occurred:
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -  Traceback (most recent call last):
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 4102, in <module>
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      sys.exit(cli())
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 4098, in cli
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      return run_test_harness(parser, options)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 4083, in run_test_harness
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      result = runner.runTests(options)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 3438, in runTests
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      res = self.runMochitests(options, tests_in_manifest, manifestToFilter=m)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 3123, in runMochitests
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      result = self.doTests(options, testsToRun, manifestToFilter)
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 3738, in doTests
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      self.stopServers()
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -    File "C:\Users\task_168631978783692\build\tests\mochitest\runtests.py", line 1505, in stopServers
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -      if self.http3Server is not None:
[task 2023-06-09T14:35:28.915Z] 14:35:28     INFO -  AttributeError: 'MochitestDesktop' object has no attribute 'http3Server'
[task 2023-06-09T14:35:29.009Z] 14:35:29     INFO - Return code: 1
[task 2023-06-09T14:35:29.009Z] 14:35:29    ERROR - No suite end message was emitted by this harness.
[task 2023-06-09T14:35:29.009Z] 14:35:29     INFO - TinderboxPrint: mochitest-mochitest-browser-a11y<br/>49/0/0
[task 2023-06-09T14:35:29.009Z] 14:35:29  WARNING - setting return code to 2

Comment 1

[Michael Froman [:mjf]]

Try run to verify this is the culprit and a temp fix.

Comment 2

[Michael Froman [:mjf]]

Attachment: Bug 1837647 - limit GetDpiForMonitor to return default value to fix Win7 tests. r?dbaker!

Upstream commit 60795e8c7a added a call to ::GetDpiForMonitor which is only
supported in Win8 and newer. This is causing perma-orange on try, so we'll
disable it for now and fix properly in Bug 1837667.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1acdb1e345a7
limit GetDpiForMonitor to return default value to fix Win7 tests. r=dbaker,webrtc-reviewers

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/501ade4b55d9
limit GetDpiForMonitor to return default value to fix Win7 tests. r=dbaker,webrtc-reviewers

Comment 5

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/501ade4b55d9

Comment 6

[Intermittent Failures Robot]

110 failures in 738 pushes (0.149 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 62
• autoland: 48

Platform and build breakdown:

• windows7-32-shippable-qr: 70
  • opt: 70
• windows7-32-qr: 40
  • opt: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1837647&startday=2023-06-09&endday=2023-06-09&tree=all

Comment 7

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/1acdb1e345a7

Comment 8

[Intermittent Failures Robot]

110 failures in 3885 pushes (0.028 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 62
• autoland: 48

Platform and build breakdown:

• windows7-32-shippable-qr: 70
  • opt: 70
• windows7-32-qr: 40
  • opt: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1837647&startday=2023-06-05&endday=2023-06-11&tree=all