Intermittent test_profile_management.py TestLog.test_in_app_restart_the_browser | error: [Errno 61] Connection refused

RESOLVED FIXED in Firefox 38

Status

RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: chmanchester, Assigned: chmanchester)

Tracking

({intermittent-failure, pi-marionette-intermittent})

unspecified
mozilla39
x86
Mac OS X
intermittent-failure, pi-marionette-intermittent
Points:
---

Firefox Tracking Flags

(firefox37 unaffected, firefox38 fixed, firefox39 fixed, firefox-esr31 unaffected)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment, 1 obsolete attachment)

(Assignee)

Description

4 years ago
This is like bug 1142344, but different.

21:46:31 INFO - TEST-START | test_profile_management.py TestLog.test_in_app_restart_the_browser
21:46:35 INFO - Failed to gather test failure debug.
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 551, in gather_debug
21:46:35 INFO - marionette.set_context(marionette.CONTEXT_CHROME)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1132, in set_context
21:46:35 INFO - return self._send_message('setContext', 'ok', value=context)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - error: [Errno 61] Connection refused
21:46:35 ERROR - TEST-UNEXPECTED-ERROR | test_profile_management.py TestLog.test_in_app_restart_the_browser | error: [Errno 61] Connection refused
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 296, in run
21:46:35 INFO - testMethod()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit/test_profile_management.py", line 44, in test_in_app_restart_the_browser
21:46:35 INFO - self.marionette.restart(in_app=True)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 918, in restart
21:46:35 INFO - self.start_session(session_id=self.session_id)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 942, in start_session
21:46:35 INFO - self.session = self._send_message('newSession', 'value', capabilities=desired_capabilities, sessionId=session_id)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - TEST-INFO took 3918ms
21:46:35 INFO - Failed to gather test failure debug.
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 551, in gather_debug
21:46:35 INFO - marionette.set_context(marionette.CONTEXT_CHROME)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1132, in set_context
21:46:35 INFO - return self._send_message('setContext', 'ok', value=context)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - error: [Errno 61] Connection refused
21:46:35 INFO - test_end for test_profile_management.py TestLog.test_in_app_restart_the_browser logged while not in progress. Logged with data: {"status": "ERROR", "extra": {"class_name": "test_profile_management.TestLog", "method_name": "test_in_app_restart_the_browser"}, "expected": "PASS", "test": "test_profile_management.py TestLog.test_in_app_restart_the_browser", "message": "error: [Errno 61] Connection refused", "stack": "Traceback (most recent call last):\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py\", line 326, in run\n self.tearDown()\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py\", line 648, in tearDown\n self.marionette.set_context(\"content\")\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 1132, in set_context\n return self._send_message('setContext', 'ok', value=context)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py\", line 36, in _\n return func(*args, **kwargs)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 692, in _send_message\n response = self.client.send(message, ignore_response=ignore_response)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py\", line 89, in send\n self.connect()\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py\", line 69, in connect\n self.sock.connect((self.addr, self.port))\n File \"/tools/python27/lib/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\n"}
(Assignee)

Updated

4 years ago
Keywords: intermittent-failure
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 6

4 years ago
These always die after trying to start the new session after the restart, maybe we have a case where we actually succeed in reconnecting to the browser as it's shutting down, and then fail to start the session because it's then as some stage of disconnecting.

I'll see if I can reproduce on try, but I think all the mac builds there are still broken. This is probably going to get a lot worse when this merges around to other integration branches, so we might have to disable the test on osx temporarily.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
More issues with uncaught crashes too :(
Keywords: ateam-marionette-intermittent
(Assignee)

Comment 10

4 years ago
(In reply to Chris Manchester [:chmanchester] from comment #6)
> These always die after trying to start the new session after the restart,
> maybe we have a case where we actually succeed in reconnecting to the
> browser as it's shutting down, and then fail to start the session because
> it's then as some stage of disconnecting.
> 
> I'll see if I can reproduce on try, but I think all the mac builds there are
> still broken. This is probably going to get a lot worse when this merges
> around to other integration branches, so we might have to disable the test
> on osx temporarily.

I reproduced this on try with a lot of logging. The theory isn't contradicted at least, I'll push a possible fix when try opens.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 16

4 years ago
(In reply to Chris Manchester [:chmanchester] from comment #10)
> (In reply to Chris Manchester [:chmanchester] from comment #6)
> > These always die after trying to start the new session after the restart,
> > maybe we have a case where we actually succeed in reconnecting to the
> > browser as it's shutting down, and then fail to start the session because
> > it's then as some stage of disconnecting.
> > 
> > I'll see if I can reproduce on try, but I think all the mac builds there are
> > still broken. This is probably going to get a lot worse when this merges
> > around to other integration branches, so we might have to disable the test
> > on osx temporarily.
> 
> I reproduced this on try with a lot of logging. The theory isn't
> contradicted at least, I'll push a possible fix when try opens.

Adding a (2 second) delay before attempting to reconnect appears to fix the problem. I'm going to try a few things that aren't quite that hacky today.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 20

4 years ago
I have a fix that doesn't add an arbitrary sleep, but closes the listener in the marionette server before quitting to avoid this issue. Looks good on osx, I'm running it on more platforms now.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 38

4 years ago
Created attachment 8578098 [details]
MozReview Request: bz://1142404/chmanchester

/r/5435 - Bug 1142404 - Fix marionette's restarts to keep from attempting to reconnect to the browser as it's shutting down.

Pull down this commit:

hg pull review -r 7376dbbba74fcde7d19f32a6b2d564e3408e97bf
Attachment #8578098 - Flags: review?(dburns)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment on attachment 8578098 [details]
MozReview Request: bz://1142404/chmanchester

https://reviewboard.mozilla.org/r/5433/#review4429

Ship It!
Attachment #8578098 - Flags: review?(dburns) → review+
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://hg.mozilla.org/mozilla-central/rev/52887e9b87d5
Assignee: nobody → cmanchester
Status: NEW → RESOLVED
Last Resolved: 4 years ago
status-firefox39: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
If bug 1141679 gets backported to 38 at some point, please remember to make sure that this is as well.
status-firefox37: --- → unaffected
status-firefox38: --- → unaffected
status-firefox-esr31: --- → unaffected
We are going to backport bug 1141679 in the near future, so marking this back as needed on Aurora as well. Setting ni? for David, who is most likely going to do this backport.
status-firefox38: unaffected → affected
Flags: needinfo?(dburns)
status-firefox38: affected → fixed
(Assignee)

Comment 58

3 years ago
Comment on attachment 8578098 [details]
MozReview Request: bz://1142404/chmanchester
Attachment #8578098 - Attachment is obsolete: true
Attachment #8619736 - Flags: review+
(Assignee)

Comment 59

3 years ago
Created attachment 8619736 [details]
MozReview Request: Bug 1142404 - Fix marionette's restarts to keep from attempting to reconnect to the browser as it's shutting down.
You need to log in before you can comment on or make changes to this bug.