Closed Bug 1142404 Opened 10 years ago Closed 9 years ago

Intermittent test_profile_management.py TestLog.test_in_app_restart_the_browser | error: [Errno 61] Connection refused

Categories

(Testing :: Marionette Client and Harness, defect)

Product:
Testing
Component:
Marionette Client and Harness
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox37 unaffected, firefox38 fixed, firefox39 fixed, firefox-esr31 unaffected)

Status:
RESOLVED FIXED
Milestone:
mozilla39
Tracking Flags:
Tracking Status
firefox37 --- unaffected
firefox38 --- fixed
firefox39 --- fixed
firefox-esr31 --- unaffected

People

(Reporter: chmanchester, Assigned: chmanchester)

References

Details

(Keywords: intermittent-failure, pi-marionette-intermittent)

Attachments

(1 file, 1 obsolete file)

MozReview Request: Bug 1142404 - Fix marionette's restarts to keep from attempting to reconnect to the browser as it's shutting down.
39 bytes, text/x-review-board-request
automatedtester
: review+
Details 
This is like bug 1142344, but different.

21:46:31 INFO - TEST-START | test_profile_management.py TestLog.test_in_app_restart_the_browser
21:46:35 INFO - Failed to gather test failure debug.
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 551, in gather_debug
21:46:35 INFO - marionette.set_context(marionette.CONTEXT_CHROME)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1132, in set_context
21:46:35 INFO - return self._send_message('setContext', 'ok', value=context)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - error: [Errno 61] Connection refused
21:46:35 ERROR - TEST-UNEXPECTED-ERROR | test_profile_management.py TestLog.test_in_app_restart_the_browser | error: [Errno 61] Connection refused
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 296, in run
21:46:35 INFO - testMethod()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/tests/testing/marionette/client/marionette/tests/unit/test_profile_management.py", line 44, in test_in_app_restart_the_browser
21:46:35 INFO - self.marionette.restart(in_app=True)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 918, in restart
21:46:35 INFO - self.start_session(session_id=self.session_id)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 942, in start_session
21:46:35 INFO - self.session = self._send_message('newSession', 'value', capabilities=desired_capabilities, sessionId=session_id)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - TEST-INFO took 3918ms
21:46:35 INFO - Failed to gather test failure debug.
21:46:35 INFO - Traceback (most recent call last):
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/tests/marionette/marionette/runner/base.py", line 551, in gather_debug
21:46:35 INFO - marionette.set_context(marionette.CONTEXT_CHROME)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1132, in set_context
21:46:35 INFO - return self._send_message('setContext', 'ok', value=context)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
21:46:35 INFO - return func(*args, **kwargs)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 692, in _send_message
21:46:35 INFO - response = self.client.send(message, ignore_response=ignore_response)
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 89, in send
21:46:35 INFO - self.connect()
21:46:35 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py", line 69, in connect
21:46:35 INFO - self.sock.connect((self.addr, self.port))
21:46:35 INFO - File "/tools/python27/lib/python2.7/socket.py", line 224, in meth
21:46:35 INFO - return getattr(self._sock,name)(*args)
21:46:35 INFO - error: [Errno 61] Connection refused
21:46:35 INFO - test_end for test_profile_management.py TestLog.test_in_app_restart_the_browser logged while not in progress. Logged with data: {"status": "ERROR", "extra": {"class_name": "test_profile_management.TestLog", "method_name": "test_in_app_restart_the_browser"}, "expected": "PASS", "test": "test_profile_management.py TestLog.test_in_app_restart_the_browser", "message": "error: [Errno 61] Connection refused", "stack": "Traceback (most recent call last):\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py\", line 326, in run\n self.tearDown()\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py\", line 648, in tearDown\n self.marionette.set_context(\"content\")\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 1132, in set_context\n return self._send_message('setContext', 'ok', value=context)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py\", line 36, in _\n return func(*args, **kwargs)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 692, in _send_message\n response = self.client.send(message, ignore_response=ignore_response)\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py\", line 89, in send\n self.connect()\n File \"/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/marionette_transport/transport.py\", line 69, in connect\n self.sock.connect((self.addr, self.port))\n File \"/tools/python27/lib/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\n"}
These always die after trying to start the new session after the restart, maybe we have a case where we actually succeed in reconnecting to the browser as it's shutting down, and then fail to start the session because it's then as some stage of disconnecting.

I'll see if I can reproduce on try, but I think all the mac builds there are still broken. This is probably going to get a lot worse when this merges around to other integration branches, so we might have to disable the test on osx temporarily.
More issues with uncaught crashes too :(
Keywords: ateam-marionette-intermittent
(In reply to Chris Manchester [:chmanchester] from comment #6)
> These always die after trying to start the new session after the restart,
> maybe we have a case where we actually succeed in reconnecting to the
> browser as it's shutting down, and then fail to start the session because
> it's then as some stage of disconnecting.
> 
> I'll see if I can reproduce on try, but I think all the mac builds there are
> still broken. This is probably going to get a lot worse when this merges
> around to other integration branches, so we might have to disable the test
> on osx temporarily.

I reproduced this on try with a lot of logging. The theory isn't contradicted at least, I'll push a possible fix when try opens.
(In reply to Chris Manchester [:chmanchester] from comment #10)
> (In reply to Chris Manchester [:chmanchester] from comment #6)
> > These always die after trying to start the new session after the restart,
> > maybe we have a case where we actually succeed in reconnecting to the
> > browser as it's shutting down, and then fail to start the session because
> > it's then as some stage of disconnecting.
> > 
> > I'll see if I can reproduce on try, but I think all the mac builds there are
> > still broken. This is probably going to get a lot worse when this merges
> > around to other integration branches, so we might have to disable the test
> > on osx temporarily.
> 
> I reproduced this on try with a lot of logging. The theory isn't
> contradicted at least, I'll push a possible fix when try opens.

Adding a (2 second) delay before attempting to reconnect appears to fix the problem. I'm going to try a few things that aren't quite that hacky today.
I have a fix that doesn't add an arbitrary sleep, but closes the listener in the marionette server before quitting to avoid this issue. Looks good on osx, I'm running it on more platforms now.
/r/5435 - Bug 1142404 - Fix marionette's restarts to keep from attempting to reconnect to the browser as it's shutting down.

Pull down this commit:

hg pull review -r 7376dbbba74fcde7d19f32a6b2d564e3408e97bf
Attachment #8578098 - Flags: review?(dburns)
Comment on attachment 8578098 [details]
MozReview Request: bz://1142404/chmanchester

https://reviewboard.mozilla.org/r/5433/#review4429

Ship It!
Attachment #8578098 - Flags: review?(dburns) → review+
https://hg.mozilla.org/mozilla-central/rev/52887e9b87d5
Assignee: nobody → cmanchester
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox39: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
If bug 1141679 gets backported to 38 at some point, please remember to make sure that this is as well.
status-firefox37: --- → unaffected
status-firefox38: --- → unaffected
status-firefox-esr31: --- → unaffected
We are going to backport bug 1141679 in the near future, so marking this back as needed on Aurora as well. Setting ni? for David, who is most likely going to do this backport.
status-firefox38: unaffected → affected
Flags: needinfo?(dburns)

        Attachment #8578098 -
        Attachment is obsolete: true

        Attachment #8619736 -
        Flags: review+

    
  
Product: Testing → Remote Protocol

    
    

    
  
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: