Closed Bug 1206613 Opened 9 years ago Closed 8 years ago

Intermittent failure in test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update| IOError: Process has been closed (Exit code: 0) (Reason: [Errno 10054] An existing connection was forcibly closed by the re

Categories

(Testing :: Firefox UI Tests, defect)

Product:
Testing
Component:
Firefox UI Tests
Version:
43 Branch
Platform:
x86_64
Windows
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(firefox43 wontfix, firefox44 wontfix, firefox45 affected, firefox46 wontfix, firefox47 wontfix, firefox49 wontfix, firefox50 affected, firefox51 affected)

Status:
RESOLVED DUPLICATE of bug 1314627
Tracking Flags:
Tracking Status
firefox43 --- wontfix
firefox44 --- wontfix
firefox45 --- affected
firefox46 --- wontfix
firefox47 --- wontfix
firefox49 --- wontfix
firefox50 --- affected
firefox51 --- affected

People

(Reporter: whimboo, Unassigned)

References

Details

(Keywords: intermittent-failure)

After a restart of the application a lot of update tests were failing on Saturday across platforms due to a disconnect of Marionette: 07:45:49 3:05.45 LOG: MainThread WARNING Failed to gather test failure debug. 07:45:49 Traceback (most recent call last): 07:45:49 07:45:49 File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette\runner\base.py", line 529, in gather_debug 07:45:49 with marionette.using_context(marionette.CONTEXT_CHROME): 07:45:49 07:45:49 File "C:\Python27\Lib\contextlib.py", line 17, in __enter__ 07:45:49 return self.gen.next() 07:45:49 07:45:49 File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\marionette.py", line 1212, in using_context 07:45:49 scope = self._send_message("getContext", key="value") 07:45:49 07:45:49 File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\decorators.py", line 36, in _ 07:45:49 return func(*args, **kwargs) 07:45:49 07:45:49 File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_driver\marionette.py", line 681, in _send_message 07:45:49 resp = self.client.send(packet) 07:45:49 07:45:49 File "c:\jenkins\workspace\mozilla-central_update\venv\lib\site-packages\marionette_transport\transport.py", line 107, in send 07:45:49 raise e 07:45:49 07:45:49 error: [Errno 10054] An existing connection was forcibly closed by the remote host Maybe Marionette didn't wait long enough until it tried to reconnect and Firefox was still shutting down.
Actually it seems to be Windows only.
OS: Unspecified → Windows
Summary: Intermittent failure in test_direct_update.py TestDirectUpdate.test_update | error: [Errno 10054] An existing connection was forcibly closed by the remote host → Intermittent failure in test_direct_update.py and test_fallback_update.py | error: [Errno 10054] An existing connection was forcibly closed by the remote host
Product: Mozilla QA → Testing
Summary: Intermittent failure in test_direct_update.py and test_fallback_update.py | error: [Errno 10054] An existing connection was forcibly closed by the remote host → Intermittent failure in test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update| error: [Errno 10054] An existing connection was forcibly closed by the remote host
Lately this seems to be a problem with the x86_64 build only.
status-firefox43: affected → wontfix
status-firefox44: affected → wontfix
status-firefox46: --- → affected
status-firefox47: --- → affected
Hardware: Unspecified → x86_64
For Firefox beta we had this failure permanently on Windows. So I had a look at the VM while the tests were running and have seen that we get a yellow screen of dead. So this bug seems to be dependent on bug 1255811.
Depends on: 1255811
Looks like this only affects builds on mozilla-central and mozilla-aurora. Machines where this is happening most of the time: * mm-win-7-32-4.qa.scl3.mozilla.com * mm-win-7-64-3.qa.scl3.mozilla.com * mm-win-7-64-4.qa.scl3.mozilla.com * mm-win-xp-32-4.qa.scl3.mozilla.com
I was finally able to see that problem by running the mozharness update script on mm-win-7-64-3.qa.scl3.mozilla.com. As what happened is similar to bug 1238002. We had a very slow download of the partial mar file around 100KB after ~3 minutes or so - I actually didn't measured the timing. Then Firefox may have been killed by Marionette. Not sure why because our timeout for downloading an update is around 6 minutes. I could only imagine that absolutely no data was received/sent through the channel. I will try to reproduce the issue outside of mozharness now.
See Also: → 1238002
Here some timings: 02:08:52 INFO - MARIONETTE LOG: INFO: TEST-START: c:\jenkins\workspace\mozi lla-aurora_update\build\tests\firefox-ui\tests\update\fallback\test_fallback_upd ate.py:test_update 02:08:52 INFO - *** AUS:SVC Creating UpdateService 02:11:55 WARNING - Failed to gather test failure debug. It looks like we have a 120s timeout somewhere in Marionette before the session gets killed.
Here a case from mozilla-aurora with an interesting detail: https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&filter-searchStr=Firefox%20UI&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3578368 > IOError: Process has been closed (Exit code: 0) It means Firefox shutdown but did not restart. It might be related to bug 974971.
status-firefox50: --- → affected
(In reply to Henrik Skupin (:whimboo) from comment #72) > > IOError: Process has been closed (Exit code: 0) > > It means Firefox shutdown but did not restart. It might be related to bug > 974971. I can see two more of those unexpected Firefox shutdowns during an expected restart for applying updates lately on aurora. There were a couple in the following commit: https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=5dddbefdf759f09b1411f33fa0920835b919fc81&filter-searchStr=firefox%20update%20windows&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=3591639
status-firefox46: affected → wontfix
status-firefox47: affected → wontfix
status-firefox49: --- → wontfix
status-firefox51: --- → affected
Summary: Intermittent failure in test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update| error: [Errno 10054] An existing connection was forcibly closed by the remote host → Intermittent failure in test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update| IOError: Process has been closed (Exit code: 0) (Reason: [Errno 10054] An existing connection was forcibly closed by the re
I'm going to assume that this bug is the same as bug 1314627 which got fixed recently.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.