1324059 - Intermittent test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been closed (Exit code: 0) (Reason: Timed out waiting for connection on localhost:2828!) - No relaunch of Firefox

Status: RESOLVED DUPLICATE of bug 1293969

(Testing :: Firefox UI Tests, defect)

Description

[Treeherder Bug Filer]

Filed by: fmezei [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=2093238&repo=mozilla-beta

https://firefox-ui-tests.s3.amazonaws.com/ccffcbf8-e8a6-40f2-9186-93ea55f450e1/log_info.log

Encountered once while running ondemand update tests for Firefox 51 Beta 8. Passed on re-build.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The signature seems to have been changed a bit, but I think it's still the same underlying issue. It increased in its occurrence a lot during the last couple of days.

Here an example from the aurora branch:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=260ad3e3f8970a637c8acef0b83fde9da19d1044&filter-searchStr=Firefox+UI&filter-tier=1&filter-tier=2&filter-tier=3

The underlying issue is that Firefox does NOT restart as requested but simply quit. The exit code is 0.

05:20:50     INFO -  1491135650061	Marionette	TRACE	conn4 -> [0,1798,"quitApplication",{"flags":["eRestart","eAttemptQuit"]}]
05:20:50     INFO -  1491135650061	Marionette	INFO	New connections will no longer be accepted
05:20:50     INFO -  1491135650061	Marionette	TRACE	conn4 <- [1,1798,null,{}]
05:20:50     INFO -  1491135650295	Marionette	DEBUG	Closed connection conn4
05:20:51     INFO -  *** UTM:SVC TimerManager:registerTimer - id: telemetry_modules_ping
05:23:57     INFO - TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | IOError: Process has been unexpectedly closed (Exit code: 0) (Reason: Timed out waiting for connection on localhost:2828!)
05:23:57     INFO - Traceback (most recent call last):
05:23:57     INFO -   File "c:\jenkins\workspace\mozilla-aurora_update\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 166, in run
05:23:57     INFO -     testMethod()
05:23:57     INFO -   File "c:\jenkins\workspace\mozilla-aurora_update\build\tests\firefox-ui\tests\update\fallback\test_fallback_update.py", line 20, in test_update
05:23:57     INFO -     self.download_and_apply_available_update(force_fallback=True)
05:23:57     INFO -   File "c:\jenkins\workspace\mozilla-aurora_update\build\venv\lib\site-packages\firefox_ui_harness\testcases.py", line 288, in download_and_apply_available_update
05:23:57     INFO -     self.restart()

This reminds me on bug 974971. I really wonder what's wrong here.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Andreas patch for bug 1337743 landed between the two nightlies when the problems are starting to appear. So this most likely looks to be a regression due to this patch.

Maybe the call to Puppeteer's `restart()` method does not correctly use the parameters?

https://dxr.mozilla.org/mozilla-central/source/testing/marionette/puppeteer/firefox/firefox_puppeteer/mixins.py#63

This might be identical to the connection issues with update tests on OS X which started at the same time. This is tracked via bug 1326237.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I also checked central and it has the same regression range. So clearly a regression from bug 1326237.

But interestingly update tests for newer builds on both branches do not show this issue anymore. I don't remember that we pushed any follow-up, so I might want to observe it the next days.

Clearing ni? from Andreas for now.

Comment 4

[Andreas Tolfsen ❲:ato❳]

There doesn’t appear to be anything suspicious with the payload the client is sending:

> 05:20:50     INFO -  1491135650061	Marionette	TRACE	conn4 -> [0,1798,"quitApplication",{"flags":["eRestart","eAttemptQuit"]}]

This is the expected input.  In case the client were to send two accidental eAttemptQuits, e.g. ["eAttemptQuit", "eAttemptQuit"], the client would remove the duplicate from the list in http://searchfox.org/mozilla-central/source/testing/marionette/client/marionette_driver/marionette.py#1159.

The server will also complain in case the client doesn’t remove duplicates as a secondary security check in http://searchfox.org/mozilla-central/source/testing/marionette/driver.js#2749.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The command from the client to the server seems fine, yes. But at this time something else caused Firefox to quit instead of restart. As long as we do not have a fix landed for bug 1350897 we don't know what actually happened. This might still be a Firefox bug, as mentioned earlier (bug 974971) but somehow got repeatedly triggered by your landed patch.

All the update tests from today for central and aurora look fine and still do not show this failure. It means that it fixed itself, and it might not be worth the time figuring out why.

I will close the bug as WFM for now. Lets reopen it if we see it again.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Happened again:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=Firefox%20&filter-tier=1&filter-tier=2&filter-tier=3&bugfiler&selectedJob=102808930

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bumped up kinda heavily yesterday with update tests across all flavors of Windows:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8f80d594c08d5c7a112e5d4b9eb44ffca717eb7b&filter-searchStr=firefox update&filter-tier=1&filter-tier=2&filter-tier=3&filter-resultStatus=success&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable

Maybe this is all related to the crash we see for update tests and bug 1375407? Somehow we might not see that crash.

Comment 8

[Robert Strong (they/them - no direct email)]

The crash in Core -> IPC, etc. is well outside app update code and it would be better to ask a developer that works on that code.

Comment 9

[Intermittent Failures Robot]

9 failures in 718 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 9

Platform breakdown:
* windows8-64: 3
* windows8-32: 2
* windows7-64: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324059&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Actually this is the same as bug 1293969 which has been filed for direct updates.