Closed Bug 1362051 Opened 3 years ago Closed 3 years ago

Intermittent Automation Error: mozprocess timed out after 300 seconds running ['python', 'firefox_ui_harness/cli_update.py' after restart of Firefox

Categories

(Testing :: Firefox UI Tests, defect, major)

Version 3
defect
Not set
major

Tracking

(firefox-esr52 unaffected, firefox53 unaffected, firefox54 unaffected, firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- unaffected
firefox55 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell fixed])

Attachments

(1 file)

Bug 1355888 landed yesterday, so I assume this is the reason for the total firefox-ui update bustage:

05:48:30     INFO - Automation Error: mozprocess timed out after 300 seconds running ['/Users/mozauto/jenkins/workspace/mozilla-central_update/build/venv/bin/python', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/venv/lib/python2.7/site-packages/firefox_ui_harness/cli_update.py', '--binary', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/application/FirefoxNightly.app/Contents/MacOS/firefox', '--address', 'localhost:2828', '--server-root', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/tests/firefox-ui/resources', '--workspace', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build', '--gecko-log=-', '--log-raw=-', '--log-html', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/blobber_upload_dir/report.html', '--log-xunit', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/blobber_upload_dir/report.xml', '-vv', '--update-channel', 'nightly', '--update-target-buildid', '20170504030320', '--disable-e10s', '--symbols-path', 'https://queue.taskcluster.net/v1/task/a3xjx-0rSQO3SRK0QXEI0Q/artifacts/public%2Fbuild%2Ffirefox-55.0a1.en-US.mac.crashreporter-symbols.zip', '/Users/mozauto/jenkins/workspace/mozilla-central_update/build/tests/firefox-ui/tests/update/manifest.ini']
05:48:30    ERROR - timed out after 300 seconds of no output
05:48:30    ERROR - Return code: -9

This is most likely because of the missing environment variable on our Jenkins slaves, because when you start with an older build, it will not set it, and after a restart we loose the connection to Marionette. I will have to prove that.
Blocks: 1355888
Keywords: regression
Severity: normal → major
Ok, I can prove that adding the `MOZ_MARIONETTE` environment variable to the global settings in Jenkins fixes the problem.

But we shouldn't do this for Jenkins only. As best it should be part of the firefox-ui-harness, specifically for the update tests runner.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Attachment #8864629 - Flags: review?(ato)
Comment on attachment 8864629 [details]
Bug 1362051 - Update tests have to set MOZ_MARIONETTE environment variable.

https://reviewboard.mozilla.org/r/136306/#review139622

This is a bit surprising, as I would have thought the builds from before the bug 1355888 changeset would rely on the marionette.enabled/marionette.defaultPrefs.enabled preference to ensure the server is activated after the restart, and that those after it would have MOZ_MARIONETTE set internally.

I guess the wrinkle to that story is when you call Services.startup.quit(eRestart) to upgrade from a build that uses marionette.enabled to a build that uses MOZ_MARIONETTE?
Attachment #8864629 - Flags: review?(ato) → review+
Comment on attachment 8864629 [details]
Bug 1362051 - Update tests have to set MOZ_MARIONETTE environment variable.

https://reviewboard.mozilla.org/r/136306/#review139632
Attachment #8864629 - Flags: review?(mjzffr) → review+
Comment on attachment 8864629 [details]
Bug 1362051 - Update tests have to set MOZ_MARIONETTE environment variable.

https://reviewboard.mozilla.org/r/136306/#review139622

Hm, strange. I just had another look and the env variable is named `MOZ_MARIONETTE_PREF_STATE_ACROSS_RESTARTS` in marionette.js. So why does it work when I use `MOZ_MARIONETTE`? Do I miss something?

> marionette.enabled/marionette.defaultPrefs.enabled preference to ensure the server is activated after the restart, and that those after it would have MOZ_MARIONETTE set internally.

Builds from before do not know about the env variable, and as such do not set it. So after the upgrade there is none, and Marionette doesn't obey the prefs any longer.
Comment on attachment 8864629 [details]
Bug 1362051 - Update tests have to set MOZ_MARIONETTE environment variable.

https://reviewboard.mozilla.org/r/136306/#review139622

> Builds from before do not know about the env variable, and as such do
> not set it. So after the upgrade there is none, and Marionette doesn't
> obey the prefs any longer.

Right, that’s what I was guessing.  Preserving the preferences, e.g.
marionette.enabled, in MOZ_MARIONETTE_PREF_STATE_ACROSS_RESTARTS won’t
help in this case because the new build that we restart to doesn’t
respect the marionette.enabled preference.

I think it makes sense to me.
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/41badc1634a8
Update tests have to set MOZ_MARIONETTE environment variable. r=ato,maja_zf
https://hg.mozilla.org/mozilla-central/rev/41badc1634a8
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Whiteboard: [stockwell fixed]
You need to log in before you can comment on or make changes to this bug.