Permafail test_places.py TestPlaces.test_bookmarks | IOError: Process has been unexpectedly closed (Exit code: -15) (Reason: [Errno 111] Connection refused)

RESOLVED FIXED in Firefox 52

Status

Testing
Firefox UI Tests
--
major
RESOLVED FIXED
2 years ago
10 months ago

People

(Reporter: Treeherder Bug Filer, Unassigned)

Tracking

({intermittent-failure})

Version 3
mozilla53
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox51 unaffected, firefox52 fixed, firefox53 fixed)

Details

Happened for all en-US Nightly builds yesterday for OS X and Linux when we call navigate():

06:24:58     INFO -  1480256698242	Marionette	TRACE	conn14 -> [0,50,"get",{"url":"http://10.22.73.172:40881/layout/mozilla_governance.html"}]
06:25:58     INFO -  1480256758302	Marionette	DEBUG	Closed connection conn14
06:27:04     INFO - TEST-UNEXPECTED-ERROR | test_places.py TestPlaces.test_bookmarks | IOError: Process has been unexpectedly closed (Exit code: -15) (Reason: [Errno 111] Connection refused)

As you can see in the host ip the local test pages we no longer serve via 127.0.0.1 but through the public IP of the host. I strongly believe that this and the abort of Firefox is a regression from bug 1103196 which landed earlier on Sunday.
Blocks: 1103196
Flags: needinfo?(ato)
I have filed the public IP part under Marionette as bug 1320643.
Depends on: 1320643
(In reply to Henrik Skupin (:whimboo) from comment #1)
> As you can see in the host ip the local test pages we no longer serve via
> 127.0.0.1 but through the public IP of the host. I strongly believe that
> this and the abort of Firefox is a regression from bug 1103196 which landed
> earlier on Sunday.

That it’s intermittent suggests the assert isn’t deterministic.
Flags: needinfo?(ato)
As I have said in comment 0 this is a hang in navigate/get and most likely related to your httpd changes. I will wait for todays Nightly builds but so far it seems to be a perma failure for those builds on Linux and OS X.
It's actually not the only test which is failing on OS X nightly builds permanently. Here an example:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=Firefox%20&filter-tier=1&filter-tier=2&filter-tier=3&bugfiler&selectedJob=5715589
Severity: normal → major
Summary: Intermittent test_places.py TestPlaces.test_bookmarks | IOError: Process has been unexpectedly closed (Exit code: -15) (Reason: [Errno 111] Connection refused) → Permafail test_places.py TestPlaces.test_bookmarks | IOError: Process has been unexpectedly closed (Exit code: -15) (Reason: [Errno 111] Connection refused)

Comment 6

2 years ago
25 failures in 694 pushes (0.036 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-central: 25

Platform breakdown:
* osx-10-9: 5
* osx-10-11: 5
* osx-10-10: 5
* linux64: 5
* linux32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2016-11-28&endday=2016-12-04&tree=all

Comment 7

2 years ago
25 failures in 35 pushes (0.714 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-aurora: 20
* mozilla-central: 5

Platform breakdown:
* osx-10-9: 5
* osx-10-11: 5
* osx-10-10: 5
* linux64: 5
* linux32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2016-12-05&endday=2016-12-05&tree=all
I discovered that the underlying problem here is a missing noproxy setting for our Linux and Mac machines for `10.*.*.*`, which is the local network the machines run in. I will have to get this added and the timeout for navigate() with 300s will not cause our jobs to get hard-killed after 60 minutes.
Depends on: 1322199
Given that the underlying proxy is not configured correctly, a page load for a locally served page will not timeout but interrupted by a socket timeout. Reason is that the latter kicks in after 60s, and a call to get() will return after its default timeout of 300s. As such we see this failure.

I proposed to dynamically adjust the socket timeout based on the expected command duration over on bug 1322277.
Depends on: 1322277

Comment 10

2 years ago
29 failures in 42 pushes (0.69 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-aurora: 29

Platform breakdown:
* osx-10-11: 8
* linux64: 8
* linux32: 8
* osx-10-10: 4
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2016-12-07&endday=2016-12-07&tree=all
Fixed by bug 1322199 which includes the Proxy updates for our Linux and OS X machines. Beside that bug 1320643 would also have fixed the failure.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox52: --- → fixed
status-firefox53: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla53

Comment 12

2 years ago
16 failures in 52 pushes (0.308 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-aurora: 16

Platform breakdown:
* linux64: 5
* osx-10-9: 4
* osx-10-11: 4
* osx-10-10: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2016-12-08&endday=2016-12-08&tree=all

Comment 13

2 years ago
93 failures in 289 pushes (0.322 failures/push) were associated with this bug in the last 7 days. 

This is the #1 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-aurora: 80
* mozilla-central: 13

Platform breakdown:
* linux64: 24
* osx-10-11: 23
* linux32: 19
* osx-10-10: 15
* osx-10-9: 11
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2016-12-05&endday=2016-12-11&tree=all
status-firefox51: --- → unaffected

Comment 14

10 months ago
1 failures in 924 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1320642&startday=2017-09-04&endday=2017-09-10&tree=all
You need to log in before you can comment on or make changes to this bug.