Closed Bug 1274120 Opened 10 years ago Closed 9 years ago

Intermittent test_refresh_firefox.py TestFirefoxRefresh.testReset | IOError: Process has been closed (Exit code: 0) (Reason: Timed out waiting for port 2828!)

Categories

(Testing :: Marionette Client and Harness, defect, P5)

Product:
Testing
Component:
Marionette Client and Harness
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1340621

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

It's a new test as written in bug 888624. It's interesting that it fails immediately after starting the test. Maybe one of the dependent bugs on bug 1258982 could be the cause. Especially inapp restarts are not that trustworthy right now.
Blocks: 888624
(In reply to Henrik Skupin (:whimboo) from comment #1) > It's a new test as written in bug 888624. It's interesting that it fails > immediately after starting the test. Maybe one of the dependent bugs on bug > 1258982 could be the cause. Especially inapp restarts are not that > trustworthy right now. What's the timeline on those bugs?
Flags: needinfo?(hskupin)
Nothing I can actually answer. I wait for that stuff myself for some firefox-ui tests. If it turns out to be a real problem you might want to talk to David Burns. For firefox-ui tests we have a workaround in place (https://dxr.mozilla.org/mozilla-central/source/testing/puppeteer/firefox/firefox_puppeteer/testcases/base.py#69) which at least makes it a bit more reliable.
Flags: needinfo?(hskupin)
(In reply to Henrik Skupin (:whimboo) from comment #3) > Nothing I can actually answer. I wait for that stuff myself for some > firefox-ui tests. If it turns out to be a real problem you might want to > talk to David Burns. > > For firefox-ui tests we have a workaround in place > (https://dxr.mozilla.org/mozilla-central/source/testing/puppeteer/firefox/ > firefox_puppeteer/testcases/base.py#69) which at least makes it a bit more > reliable. AIUI the issue is that marionette does not reconnect successfully to the new process (within whatever timeout it uses). How does the quit-application-requested stuff help with that? Or do I misunderstand the issue? Or something else? :-) FWIW, as it is it doesn't seem like this is toooo frequent, assuming Phil filed soon after the first instance of this. bug 888624 landed 5 days ago, and orange once every 5 days isn't tooooo terrible. Still, would be nice if it were more reliable, also because I expect that cleanup is imperfect in cases like these...
Flags: needinfo?(hskupin)
(In reply to :Gijs Kruitbosch from comment #4) > AIUI the issue is that marionette does not reconnect successfully to the new > process (within whatever timeout it uses). How does the > quit-application-requested stuff help with that? Or do I misunderstand the > issue? Or something else? :-) AFAIR it was added to help with dataloss. Marionette force quit the browser for a restart so some tasks (eg. sessionstore) might still not have successfully saved the data. > FWIW, as it is it doesn't seem like this is toooo frequent, assuming Phil > filed soon after the first instance of this. bug 888624 landed 5 days ago, > and orange once every 5 days isn't tooooo terrible. Still, would be nice if > it were more reliable, also because I expect that cleanup is imperfect in > cases like these... Right. It was just a FYI from my side given that we experience similar issues. Lets have a look again if it turns out to be happening more often.
Flags: needinfo?(hskupin)
Priority: -- → P5
I will try to make it part of my work for bug 1283906. Maybe one of the other already listed bugs might fix this one.
Blocks: 1283906
Looks like we had a spike of those failures in the last 2 days. As it looks like the system which is affected most of the times is windows7-32-vm and then specifically for debug builds.
Interestingly this nearly permafail started with the following changeset: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c6d147ba37b52dd368bc213580e1bec9f3f71e0d&filter-searchStr=mn Which is "Bug 1281259 - Port DXVA to gfxConfig".
Oh, Wes already backed-out the causing patch so that we failure rate dropped.
The signature of this failure has been changed meanwhile. Going to update the summary of this bug. Also to note a lot of failing Mn jobs in Treeherder have been wrongly starred against this bug. :( The last one which actually covers this failure is from Aug 24th: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=649c62d13b57bf6966bc70f4da603d52c92070de&selectedJob=34564951 The interesting part is in the gecko.log. Specifically the last two lines: https://public-artifacts.taskcluster.net/WaauBypDSoiF4DqZEmuT8Q/0/public/test_info//gecko.log > MARIONETTE LOG: INFO: TEST-START: /home/worker/workspace/build/tests/marionette/tests/browser/components/migration/tests/marionette/test_refresh_firefox.py:testReset > Inconsistency detected by ld.so: dl-open.c: 667: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed! Then the log ends. Given the actual failure message we got: > IOError: Process has been closed (Exit code: 127) (Reason: [Errno 111] Connection refused) So the Firefox process died with exit code 127. Given that older failures also show an exit code of 0, we should have different reasons for a quit in this test.
Summary: Intermittent test_refresh_firefox.py TestFirefoxRefresh.testReset | IOError: Timed out waiting for port! → Intermittent test_refresh_firefox.py TestFirefoxRefresh.testReset | IOError: Process has been closed (Exit code: 0) (Reason: Timed out waiting for port 2828!)
Given the very rare occurrence of this test failure I will not make it part of my work for bug 1283906. But maybe it get fixed by one of the other changes.
No longer blocks: 1283906
See Also: → 1330744
:whimboo, can you look at resolving this bug? possibly the resolution is the same as bug 1330744, both of these contribute to many failures ni myself until :whimboo can take ni requests next week
Flags: needinfo?(jmaher)
See Also: → 1340621
I haven't written this test and I also don't maintain it. This is all in the hands of Firefox developers like Gijs. Also I do not see a relation to bug 1330744 at all. This bug is about a completely different issue.
(In reply to Henrik Skupin (:whimboo) from comment #19) > I haven't written this test and I also don't maintain it. This is all in the > hands of Firefox developers like Gijs. Also I do not see a relation to bug > 1330744 at all. This bug is about a completely different issue. There's no debug output for "Firefox developers like [me]" to help here. Marionette fails to reconnect to Firefox... why? There's no screenshot, there's no log output from Firefox, there's no indication whether the process is alive at that point or not, there's no crash dumps, it's not even clear what emits this "we couldn't connect" output - the python handler of Firefox or the Firefox process itself? Or something else? Successful runs of this test on win8 debug (the only affected platform) take approximately 30 seconds or thereabouts, and the orange ones seem to take 120s, so it seems to me that just extending the timeout won't make a difference. Other than that I have no idea what to suggest because it's completely not clear to me what the issue is. It's also not clear to me why this started spiking on the 17th of February and is now not happening at all anymore. Doesn't seem worth investigating unless/until there's more info about why this spiked on Feb 17.
(In reply to Henrik Skupin (:whimboo) from comment #19) > Also I do not see a relation to bug > 1330744 at all. This bug is about a completely different issue. I only noted bug 1330744 as a see-also because it's the same test and it started failing frequently at the same time (~Feb 15), on the same platform. It looks to me like many of the Feb 16-18 failures in bug 1330744 should have been reported against this bug or bug 1340621.
this is not showing up anymore, although I do question the use of marionette as debugging has been difficult and we end up with many questions and little answers usually :(
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) from comment #22) > this is not showing up anymore, although I do question the use of marionette > as debugging has been difficult and we end up with many questions and little > answers usually :( There's no other framework that lets us have an automated test here, to my knowledge, as we need to be able to (re)start the browser (with custom options/params).
Btw, I still wonder why we see an exit code of 0 for a restart here. In our checks it would mean that the process exited and didn't get restarted. Which itself reminds me again on bug 974971. Can someone please check which of the restarts is actually failing nowadays? (In reply to Geoff Brown [:gbrown] from comment #21) > It looks to me like many of the Feb 16-18 failures in bug 1330744 should > have been reported against this bug or bug 1340621. Shall we better dupe bug 1340621 into this one? What is the reason to keep two bugs open for the same failure message? (In reply to Joel Maher ( :jmaher) from comment #22) > this is not showing up anymore, although I do question the use of marionette > as debugging has been difficult and we end up with many questions and little > answers usually :( Really? I'm not with you on that in general cases because compared to other test harnesses we have all the debug info in the logs. Also here we have a kinda special test.
Whiteboard: [stockwell unknown]
Fixed by changing the output so that we refiled the bug.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing
You need to log in before you can comment on or make changes to this bug.