1469534 - Only run Marionette unit tests on Android for integration branches and try

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P1)

Description

[Eliza Balazs [:ebalazs_]]

INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py', '--emulator', '--app=fennec', '--package=org.mozilla.firefox_beta', '--address=localhost:2828', '/builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini', '--disable-e10s', '--gecko-log=-', '-vv', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/marionette_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log', '--symbols-path=https://queue.taskcluster.net/v1/task/EkK3zH_XTRSsi0FO5jPCJg/artifacts/public/build/target.crashreporter-symbols.zip', '--startup-timeout=300', '--this-chunk', '2', '--total-chunks', '10'] in /builds/worker/workspace/build/tests/marionette
[task 2018-06-19T04:01:31.698Z] 04:01:31     INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.firefox_beta --address=localhost:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --disable-e10s --gecko-log=- -vv --log-raw=/builds/worker/workspace/build/blobber_upload_dir/marionette_raw.log --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log --symbols-path=https://queue.taskcluster.net/v1/task/EkK3zH_XTRSsi0FO5jPCJg/artifacts/public/build/target.crashreporter-symbols.zip --startup-timeout=300 --this-chunk 2 --total-chunks 10
[task 2018-06-19T04:01:31.698Z] 04:01:31     INFO - Using env: (same as previous command)
[task 2018-06-19T04:01:31.908Z] 04:01:31     INFO -  Using workspace for temporary data: "/builds/worker/workspace/build/tests/marionette"
[task 2018-06-19T04:01:32.123Z] 04:01:32     INFO -  adb adbd running as root
[task 2018-06-19T04:01:32.226Z] 04:01:32     INFO -  adb su -c supported
[task 2018-06-19T04:01:32.329Z] 04:01:32     INFO -  adb su 0 supported
[task 2018-06-19T04:01:32.534Z] 04:01:32     INFO -  adb /system/bin/ls -a supported
[task 2018-06-19T04:01:32.637Z] 04:01:32     INFO -  adb Native cp support: True
[task 2018-06-19T04:01:32.740Z] 04:01:32     INFO -  adb Native chmod -R support: True
[task 2018-06-19T04:01:32.843Z] 04:01:32     INFO -  adb Setting SELinux Permissive Mode
[task 2018-06-19T04:01:32.946Z] 04:01:32     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.
[task 2018-06-19T04:01:40.101Z] 04:01:40     INFO -  timed out waiting for profiles.ini
[task 2018-06-19T04:01:42.570Z] 04:01:42     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpzG_guRpidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-19T04:01:44.225Z] 04:01:44     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-19T04:01:46.880Z] 04:01:46     INFO -  - waiting for device -
[task 2018-06-19T04:06:51.125Z] 04:06:51     INFO -  {"thread": "MainThread", "level": "ERROR", "pid": 1027, "source": "Marionette test runner", "time": 1529381211123, "action": "log", "message": "Failure during harness execution", "stack": "Traceback (most recent call last):\n\n  File \"/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 94, in cli\n    failed = harness_instance.run()\n\n  File \"/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 74, in run\n    runner.run_tests(tests)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/runner/base.py\", line 857, in run_tests\n    self.marionette = self.driverclass(**self._build_kwargs())\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 615, in __init__\n    self.start_binary(self.startup_timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 639, in start_binary\n    reraise(IOError, msg.format(timeout), tb)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 630, in start_binary\n    self.raise_for_port(timeout=timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 710, in raise_for_port\n    self.host, self.port))\n\nIOError: Process killed after 300s because no connection to Marionette server could be established. Check gecko.log for errors\n"}
[task 2018-06-19T04:06:51.744Z] 04:06:51     INFO -  cleanup aborted: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell rm /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini; echo rc=$?, exitcode: 255, stdout: rm failed for /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini, No such file or directory
[task 2018-06-19T04:06:52.052Z] 04:06:52     INFO -  Traceback (most recent call last):
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py", line 497, in close
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -      "tcp:{}".format(self.marionette_port))
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 977, in remove_forwards
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -      self.command_output(cmd, timeout=timeout)
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 888, in command_output
[task 2018-06-19T04:06:52.054Z] 04:06:52     INFO -      timeout=timeout)
[task 2018-06-19T04:06:52.054Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 293, in command_output
[task 2018-06-19T04:06:52.054Z] 04:06:52 CRITICAL -      raise ADBProcessError(adb_process)
[task 2018-06-19T04:06:52.054Z] 04:06:52 CRITICAL -  ADBProcessError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device forward --remove tcp:2828, exitcode: 1, stdout: error: listener 'tcp:2828' not found
[task 2018-06-19T04:06:52.065Z] 04:06:52    ERROR - Return code: 1
[task 2018-06-19T04:06:52.066Z] 04:06:52    ERROR - No tests run or test summary not found
[task 2018-06-19T04:06:52.066Z] 04:06:52     INFO - TinderboxPrint: marionette<br/><em class="testfail">T-FAIL</em>
[task 2018-06-19T04:06:52.066Z] 04:06:52     INFO - ##### marionette log ends
[task 2018-06-19T04:06:52.066Z] 04:06:52 CRITICAL - # TBPL RETRY #
[task 2018-06-19T04:06:52.067Z] 04:06:52  WARNING - setting return code to 4
[task 2018-06-19T04:06:52.067Z] 04:06:52 CRITICAL - The marionette suite: marionette ran with return status: RETRY

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Firefox doesn't seem to have been started and Marionette run into a timeout error.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

This is permafailing on Beta since the uplift of 62 there. I don't think P3 is the right priority there unless we no longer care about Android Mn jobs.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #2)
> This is permafailing on Beta since the uplift of 62 there. I don't think P3
> is the right priority there unless we no longer care about Android Mn jobs.

So why does no-one mention that? When I check mozilla-beta for Android Mn jobs I see all the failing jobs marked as expected fail??? And this without using this bug number! This is absolutely not what should have happened here. Sebastian, can you please work with sheriffs so that those things are getting escalated way earlier?

Did any previously run nightly-to-beta test merge showed those failures? If not I wonder if that is caused by some late push to mozilla-central right before the official merge.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here the link to the Mn results on mozilla-beta:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=android%20mn

There is no uploaded log available. As such I triggered a one click loaner and was able to see the hang here:

> 08:08:18     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity
> .MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_C
> RASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,Medi
> aPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 N
> O_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpPiWu7fpidlog --es env9 R_LOG_LEVEL=6
> 08:08:20     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
> 08:08:21     INFO -  timed out waiting for 'org.mozilla.firefox_beta' process to start
> 08:08:22     INFO -  - waiting for device -

Firefox starts but we seem to no longer detect that, or Marionette doesn't start-up. Then after 300s the following traceback is visible:

>        08:13:27     INFO -  {"thread": "MainThread", "level": "ERROR", "pid": 1135, "source": "Marionette test runner", "time": 1529914407010, 
> "action": "log", "message": "Failure during harness execution", "stack": "Traceback (most recent call last):\n\n  File \"/builds/worker/workspac
> e/build/tests/marionette/harness/marionette_harness/runtests.py\", line 94, in cli\n    failed = harness_instance.run()\n\n  File \"/builds/work
> er/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 74, in run\n    runner.run_tests(tests)\n\n  File \"/builds/w
> orker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/runner/base.py\", line 857, in run_tests\n    self.marionette = self.d
> riverclass(**self._build_kwargs())\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\"
> , line 615, in __init__\n    self.start_binary(self.startup_timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages
> /marionette_driver/marionette.py\", line 639, in start_binary\n    reraise(IOError, msg.format(timeout), tb)\n\n  File \"/builds/worker/workspac
> e/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 630, in start_binary\n    self.raise_for_port(timeout=timeout)\
> n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 710, in raise_for_port\n    
> self.host, self.port))\n\nIOError: Process killed after 300s because no connection to Marionette server could be established. Check gecko.log fo
> r errors\n"}
> 08:13:27     INFO -  cleanup aborted: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device
>  shell rm /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini; echo rc=$?, exitcode: 255, stdout: rm failed for /data/data/org.mozill
> a.firefox_beta/files/mozilla/profiles.ini, No such file or directory
> 08:13:27     INFO -  Traceback (most recent call last):
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py", line 497, in c
> lose
> 08:13:27     INFO -      "tcp:{}".format(self.marionette_port))
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 984, in remove_forwards
> 08:13:27     INFO -      self.command_output(cmd, timeout=timeout)
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 895, in command_output
> 08:13:27     INFO -      timeout=timeout)
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 293, in command_output
> 08:13:27 CRITICAL -      raise ADBProcessError(adb_process)
> 08:13:27 CRITICAL -  ADBProcessError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device
>  forward --remove tcp:2828, exitcode: 1, stdout: error: listener 'tcp:2828' not found

With bug 1440714 we got some major changes for Android but those were already for Firefox 61. So it shouldn't have affected us here for the mozilla-central to mozilla-beta merge of Firefox 62.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

> 08:08:22 INFO - - waiting for device -

This would mean that we may have lost the connection to the device? Geoff, or Bob do you have an idea?

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> There is no uploaded log available. As such I triggered a one click loaner
> and was able to see the hang here:

The treeherder log buttons are inactive, but actually, all the logs are available as artifacts on the Job Details pane.

Comment 7

[Geoff Brown [:gbrown]]

The logcat artifact shows Firefox starting and living until being killed after the 5 minute timeout:

https://taskcluster-artifacts.net/c2_GisYeTQymnVCg33nhZw/5/public/test_info//logcat-emulator-5554.log

06-23 20:36:18.414   277   289 I ActivityManager: Start proc org.mozilla.firefox_beta for activity org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp: pid=760 uid=10043 gids={50043, 3003, 1028, 1015, 1006}
06-23 20:36:19.064   760   760 I GeckoApplication: zerdatime 141494 - application start
...
06-23 20:41:21.184    38    38 D Zygote  : Process 760 terminated by signal (15)
06-23 20:41:21.194   277   288 I ActivityManager: Process org.mozilla.firefox_beta (pid 760) has died.


" - waiting for device - " seems a bit odd, but might be nothing to worry about...needs more investigation.

But given that Firefox is starting, "IOError: Process killed after 300s because no connection to Marionette server could be established." seems to be the real trouble.

Comment 8

[Bob Clary [:bc] (inactive)]

The message 'waiting for device' appears to be created here: https://searchfox.org/mozilla-central/source/media/webrtc/trunk/build/android/adb_device_functions.sh#119 and does seem to say the connection was lost.

[task 2018-06-24T03:33:24.087Z] 03:33:24     INFO -  adb adbd running as root
[task 2018-06-24T03:33:24.190Z] 03:33:24     INFO -  adb su -c supported
[task 2018-06-24T03:33:24.293Z] 03:33:24     INFO -  adb su 0 supported
[task 2018-06-24T03:33:24.499Z] 03:33:24     INFO -  adb /system/bin/ls -a supported
[task 2018-06-24T03:33:24.602Z] 03:33:24     INFO -  adb Native cp support: True
[task 2018-06-24T03:33:24.706Z] 03:33:24     INFO -  adb Native chmod -R support: True
[task 2018-06-24T03:33:24.809Z] 03:33:24     INFO -  adb Unable to turn off logcat chatty
[task 2018-06-24T03:33:24.912Z] 03:33:24     INFO -  adb Setting SELinux Permissive Mode
[task 2018-06-24T03:33:25.015Z] 03:33:25     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.

up to this point, we had a connection via adb.py.

[task 2018-06-24T03:33:31.777Z] 03:33:31     INFO -  timed out waiting for profiles.ini

This is from https://searchfox.org/mozilla-central/source/testing/mozbase/mozrunner/mozrunner/devices/base.py#99 which seems to imply the connection was not lost (no ADBTimeoutError) but that the profile was not pushed to the emulator/device. This seems it should be fatal.


[task 2018-06-24T03:33:34.246Z] 03:33:34     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpE15XVupidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-24T03:33:35.903Z] 03:33:35     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-24T03:33:36.889Z] 03:33:36     INFO -  timed out waiting for 'org.mozilla.firefox_beta' process to start
[task 2018-06-24T03:33:37.954Z] 03:33:37     INFO -  - waiting for device -

Looking at the logcat logs doesn't help but I'm not familiar with how it looked when it ran ok. I didn't see anything about marionette or adb forward though. Fennec was started then killed via SIGTERM however. Is marionette starting dependent on the profile being uploaded?

Comment 9

[Geoff Brown [:gbrown]]

On mozilla-central, successful runs look like:

https://treeherder.mozilla.org/logviewer.html#?job_id=184662914&repo=mozilla-central&lineNumber=1323-1337

[task 2018-06-25T10:36:35.099Z] 10:36:35     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.
[task 2018-06-25T10:36:42.165Z] 10:36:42     INFO -  timed out waiting for profiles.ini
[task 2018-06-25T10:36:44.636Z] 10:36:44     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpMRWpUnpidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-25T10:36:46.368Z] 10:36:46     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-25T10:36:49.244Z] 10:36:49     INFO -  - waiting for device -
[task 2018-06-25T10:38:19.308Z] 10:38:19     INFO -  Profile path is /tmp/tmp9mhLNK.mozrunner
[task 2018-06-25T10:38:19.309Z] 10:38:19     INFO -  Starting fixture servers
[task 2018-06-25T10:38:20.514Z] 10:38:20     INFO -  Fixture server listening on http://172.17.0.5:37797/
[task 2018-06-25T10:38:20.515Z] 10:38:20     INFO -  Fixture server listening on https://172.17.0.5:40732/
[task 2018-06-25T10:38:20.522Z] 10:38:20     INFO -  mozinfo updated from: /builds/worker/workspace/build/tests/mozinfo.json
[task 2018-06-25T10:38:20.524Z] 10:38:20     INFO -  mozinfo is: {u'bin_suffix': u'', u'official': True, u'allow_legacy_extensions': True, u'pgo': False, u'sync': False, u'buildapp': u'mobile/android', u'crashreporter': True, u'devedition': False, u'require_signing': False, u'platform_guess': u'android-arm', u'appname': u'fennec', u'stylo': True, u'mozconfig': u'/builds/worker/workspace/build/src/.mozconfig', u'topsrcdir': u'/builds/worker/workspace/build/src', u'ubsan': False, 'os_version': StringVersion ('16.04'), 'version': 'Ubuntu 16.04', 'webrender': False, u'buildtype_guess': u'opt', 'bits': 32, 'has_sandbox': True, u'datareporting': True, u'artifact': False, u'toolkit': u'android', u'healthreport': True, u'updater': True, u'cc_type': u'clang', u'asan': False, u'release_or_beta': False, u'ccov': False, u'android_min_sdk': u'16', u'tests_enabled': True, 'linux_distro': 'Ubuntu', u'telemetry': False, u'tsan': False, u'nightly_build': True, u'debug': False, 'os': u'android', 'processor': u'arm'}
[task 2018-06-25T10:38:22.132Z] 10:38:22     INFO -  e10s is disabled
[task 2018-06-25T10:38:22.179Z] 10:38:22     INFO -  Failed to retrieve version information for None
[task 2018-06-25T10:38:22.180Z] 10:38:22     INFO -  SUITE-START | Running 50 tests
[task 2018-06-25T10:38:22.181Z] 10:38:22     INFO -  TEST-START | testing/marionette/harness/marionette_harness/tests/unit/test_cli_arguments.py

Comment 10

[Geoff Brown [:gbrown]]

On mozilla-central, logcat from successful runs includes:

06-25 03:45:23.056   756   791 I Gecko   : 1529923522768	addons.xpi	WARN	List of valid built-in add-ons could not be parsed.: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]"  nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: startup :: line 2093"  data: no] Stack trace: startup()@resource://gre/modules/addons/XPIProvider.jsm:2093
06-25 03:45:23.056   756   791 I Gecko   : callProvider()@resource://gre/modules/AddonManager.jsm:206
06-25 03:45:23.056   756   791 I Gecko   : _startProvider()@resource://gre/modules/AddonManager.jsm:654
06-25 03:45:23.056   756   791 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:813
06-25 03:45:23.056   756   791 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:2808
06-25 03:45:23.056   756   791 I Gecko   : observe()@jar:jar:file:///data/app/org.mozilla.fennec_aurora-1.apk!/assets/omni.ja!/components/addonManager.js:66
06-25 03:45:24.636   756   791 I Gecko   : 1529923524637	addons.xpi	WARN	No list of valid system add-ons found.
06-25 03:45:26.246   756   791 D GeckoThread: State changed to PROFILE_READY
06-25 03:45:26.736   756   791 I Gecko   : 1529923526735	Marionette	DEBUG	Received observer notification profile-after-change


On beta, there are no Marionette messages:

06-23 20:37:38.334   760   799 I Gecko   : 1529811458134	addons.xpi	WARN	List of valid built-in add-ons could not be parsed.: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]"  nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: startup :: line 2093"  data: no] Stack trace: startup()@resource://gre/modules/addons/XPIProvider.jsm:2093
06-23 20:37:38.334   760   799 I Gecko   : callProvider()@resource://gre/modules/AddonManager.jsm:206
06-23 20:37:38.334   760   799 I Gecko   : _startProvider()@resource://gre/modules/AddonManager.jsm:654
06-23 20:37:38.334   760   799 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:813
06-23 20:37:38.334   760   799 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:2808
06-23 20:37:38.334   760   799 I Gecko   : observe()@jar:jar:file:///data/app/org.mozilla.firefox_beta-1.apk!/assets/omni.ja!/components/addonManager.js:66
06-23 20:37:40.004   760   799 I Gecko   : 1529811460006	addons.xpi	WARN	No list of valid system add-ons found.
06-23 20:37:41.613   760   799 D GeckoThread: State changed to PROFILE_READY

Comment 11

[Ryan VanderMeulen [:RyanVM]]

I wonder if there's a changed addon that needs signing or something. That's the only thing I can think of that would explain why the simulations on Try are OK and "real" Beta isn't.

Comment 12

[Geoff Brown [:gbrown]]

I haven't fully parsed this, but it certainly seems related:

https://hg.mozilla.org/releases/mozilla-beta/rev/2ad47b8c9555 / bug 1460315

Maybe we shouldn't expect marionette to be enabled on beta?

Comment 13

[Geoff Brown [:gbrown]]

Discussed on #ateam: Android marionette is intentionally disabled on beta -- we should stop trying to run Android Mn tests on beta.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Geoff Brown [:gbrown] from comment #13)
> Discussed on #ateam: Android marionette is intentionally disabled on beta --
> we should stop trying to run Android Mn tests on beta.

This is not true for debug builds which also fail permanently. There is something else which caused this regression.

Comment 15

[Geoff Brown [:gbrown]]

We originally enabled marionette on all Android/debug builds, but bug 1460315 changed that to enable on opt+debug on select channels -- not including beta. I'm not sure of the back-story, but I believe we do not want marionette enabled in Firefox releases, presumably including beta. That seems to be the intention of the change in bug 1460315.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Geoff Brown [:gbrown] from comment #15)
> We originally enabled marionette on all Android/debug builds, but bug
> 1460315 changed that to enable on opt+debug on select channels -- not
> including beta. I'm not sure of the back-story, but I believe we do not want

That is not exactly what we originally discussed because we never decided to limit the testing off debug builds. But
seeing that this is the case now we would have to stop testing any kind of Android builds on those affected channels.

Problem here is that we cannot safely uplift changes for Marionette to mozilla-beta, or an esr-branch anymore because no testing is effectively in place for both of them, and we cannot guarantee that no regression is getting introduced during the beta cycle. 

Given that I'm not the one who can decide that I will set ni? for David, and James.

Comment 17

[Andreas Tolfsen ❲:ato❳]

(In reply to Henrik Skupin (:whimboo) from comment #16)

> Problem here is that we cannot safely uplift changes for
> Marionette to mozilla-beta, or an esr-branch anymore because no
> testing is effectively in place for both of them, and we cannot
> guarantee that no regression is getting introduced during the beta
> cycle.

Because Marionette is _not enabled_ in beta or esr on Fennec,
uplifting should not be a problem.

Marionette support for Fennec is best-effort and until we decide
it is good enough to ship as part of regular Fennec releases, the
current practice of ensuring patches to central don’t break Fennec
Nightly seems like the best we can do.  Enabling Marionette n debug
and not opt on Fennec beta makes little sense to me.

Comment 18

[David Burns :automatedtester]

I agree with Andreas here. Until we have done a RRA on marionette on Mobile we shouldnt really worry about things other than nightly unless there is a need for it from the geckoview team

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Alright then. I will upload a patch shortly which should disable Mn jobs on Android for release branches.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1469534 - Do not run Marionette tests for Android on release branches.

Review commit: https://reviewboard.mozilla.org/r/253998/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/253998/

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dustin, as it looks like the patch as attached doesn't work and cause an error in the decision task. How can I define the default for a list? Thanks.

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

`default: default` gives a default value of `"default"`, but that's not allowed.  I think what you want there is `default: built-projects`.

Comment 23

[Intermittent Failures Robot]

31 failures in 111 pushes (0.279 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-beta: 31

Platform breakdown:
* android-em-4-3-armv7-api16: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1469534&startday=2018-07-03&endday=2018-07-03&tree=trunk

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #22)
> `default: default` gives a default value of `"default"`, but that's not
> allowed.  I think what you want there is `default: built-projects`.

Thanks. Lets give it a try.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/253998/diff/1-2/

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261574

The try build looks fine now. But I can only test this on try. No idea how to make sure the tests won't run on release branches.

Dustin, I assume this is something tested somewhere else, and you may the best reviewer for the patch. Thanks.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I will be on PTO soon. So whenever this gets a r+ can someone please push to autoland? If it gets a r- Andreas or David will pick it up.

Comment 28

[Tom Prince [:tomprince]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261882

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

I think this should be `['trunk', 'try']`. The list of projects that this includes is defined [here](https://searchfox.org/mozilla-central/rev/6ef785903fee6c0b16a1eab79d722373d940fd78/taskcluster/taskgraph/util/attributes.py#15)

Comment 29

[Tom Prince [:tomprince]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261574

You can see what will get run on a beta by doing something like `mach taskgraph target-graph -p project=mozilla-beta`. You can also filter it by adding `--tasks-regex '.*marionette.*'.

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261882

> I think this should be `['trunk', 'try']`. The list of projects that this includes is defined [here](https://searchfox.org/mozilla-central/rev/6ef785903fee6c0b16a1eab79d722373d940fd78/taskcluster/taskgraph/util/attributes.py#15)

I was just following what eg. Talos does: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/talos.yml#319

`trunk` would include `comm-central` where Marionette is not supported.

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261900

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

I think that makes sense (to exclude comm-central)

Comment 32

[Andreas Tolfsen ❲:ato❳]

The full taskgraph:

> % ./mach taskgraph target-graph -p 'project=mozilla-beta' --tasks-regex '.*marionette.*'
> …
> source-test-python-marionette-harness-py2
> test-android-em-4.3-arm7-api-16/debug-marionette-1
> test-android-em-4.3-arm7-api-16/debug-marionette-10
> test-android-em-4.3-arm7-api-16/debug-marionette-2
> test-android-em-4.3-arm7-api-16/debug-marionette-3
> test-android-em-4.3-arm7-api-16/debug-marionette-4
> test-android-em-4.3-arm7-api-16/debug-marionette-5
> test-android-em-4.3-arm7-api-16/debug-marionette-6
> test-android-em-4.3-arm7-api-16/debug-marionette-7
> test-android-em-4.3-arm7-api-16/debug-marionette-8
> test-android-em-4.3-arm7-api-16/debug-marionette-9
> test-android-em-4.3-arm7-api-16/opt-marionette-1
> test-android-em-4.3-arm7-api-16/opt-marionette-10
> test-android-em-4.3-arm7-api-16/opt-marionette-2
> test-android-em-4.3-arm7-api-16/opt-marionette-3
> test-android-em-4.3-arm7-api-16/opt-marionette-4
> test-android-em-4.3-arm7-api-16/opt-marionette-5
> test-android-em-4.3-arm7-api-16/opt-marionette-6
> test-android-em-4.3-arm7-api-16/opt-marionette-7
> test-android-em-4.3-arm7-api-16/opt-marionette-8
> test-android-em-4.3-arm7-api-16/opt-marionette-9
> test-linux32-devedition/opt-marionette-e10s
> test-linux32-nightly/opt-marionette-e10s
> test-linux32/debug-marionette-e10s
> test-linux64-asan/opt-marionette-e10s
> test-linux64-asan/opt-marionette-headless-e10s
> test-linux64-devedition/opt-marionette-e10s
> test-linux64-devedition/opt-marionette-headless-e10s
> test-linux64-nightly/opt-marionette-e10s
> test-linux64-nightly/opt-marionette-headless-e10s
> test-linux64/debug-marionette-e10s
> test-linux64/debug-marionette-headless-e10s
> test-macosx64-devedition/opt-marionette-e10s
> test-macosx64-devedition/opt-marionette-headless-e10s
> test-macosx64-nightly/opt-marionette-e10s
> test-macosx64-nightly/opt-marionette-headless-e10s
> test-macosx64/debug-marionette-e10s
> test-macosx64/debug-marionette-headless-e10s
> test-windows10-64-devedition/opt-marionette-e10s
> test-windows10-64-devedition/opt-marionette-headless-e10s
> test-windows10-64-nightly/opt-marionette-e10s
> test-windows10-64-nightly/opt-marionette-headless-e10s
> test-windows10-64/debug-marionette-e10s
> test-windows10-64/debug-marionette-headless-e10s
> test-windows7-32-devedition/opt-marionette-e10s
> test-windows7-32-devedition/opt-marionette-headless-e10s
> test-windows7-32-nightly/opt-marionette-e10s
> test-windows7-32-nightly/opt-marionette-headless-e10s
> test-windows7-32/debug-marionette-e10s

Are we missing anything with the way this patch is written?

Comment 33

[Tom Prince [:tomprince]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261940

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

`comm-central` has its tests defined elsewhere, so it won't get run there for other reasons. I think `trunk` captures the intent here better.

I think talos should probably also change (at some point).

Comment 34

[Tom Prince [:tomprince]]

Andreas, are you testing with the patch applied? I don't get the android tasks when I test locally.

Comment 35

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andreas Tolfsen 「:ato」 from comment #32)
> > test-android-em-4.3-arm7-api-16/debug-marionette-1
> > test-android-em-4.3-arm7-api-16/debug-marionette-10
> > test-android-em-4.3-arm7-api-16/debug-marionette-2
> > test-android-em-4.3-arm7-api-16/debug-marionette-3
> > test-android-em-4.3-arm7-api-16/debug-marionette-4
> > test-android-em-4.3-arm7-api-16/debug-marionette-5
> > test-android-em-4.3-arm7-api-16/debug-marionette-6
> > test-android-em-4.3-arm7-api-16/debug-marionette-7
> > test-android-em-4.3-arm7-api-16/debug-marionette-8
> > test-android-em-4.3-arm7-api-16/debug-marionette-9
> > test-android-em-4.3-arm7-api-16/opt-marionette-1
> > test-android-em-4.3-arm7-api-16/opt-marionette-10
> > test-android-em-4.3-arm7-api-16/opt-marionette-2
> > test-android-em-4.3-arm7-api-16/opt-marionette-3
> > test-android-em-4.3-arm7-api-16/opt-marionette-4
> > test-android-em-4.3-arm7-api-16/opt-marionette-5
> > test-android-em-4.3-arm7-api-16/opt-marionette-6
> > test-android-em-4.3-arm7-api-16/opt-marionette-7
> > test-android-em-4.3-arm7-api-16/opt-marionette-8
> > test-android-em-4.3-arm7-api-16/opt-marionette-9
> 
> Are we missing anything with the way this patch is written?

Not sure what you were running here but I assume my patch was not applied to your local tree. When I run the proposed command I see the following output, which is fine:

> source-test-python-marionette-harness-py2
> test-linux32-devedition/opt-marionette-e10s
> test-linux32-nightly/opt-marionette-e10s
> test-linux32/debug-marionette-e10s
> test-linux64-asan/opt-marionette-e10s
> test-linux64-asan/opt-marionette-headless-e10s
> test-linux64-devedition/opt-marionette-e10s
> test-linux64-devedition/opt-marionette-headless-e10s
> test-linux64-nightly/opt-marionette-e10s
> test-linux64-nightly/opt-marionette-headless-e10s
> test-linux64/debug-marionette-e10s
> test-linux64/debug-marionette-headless-e10s
> test-macosx64-devedition/opt-marionette-e10s
> test-macosx64-devedition/opt-marionette-headless-e10s
> test-macosx64-nightly/opt-marionette-e10s
> test-macosx64-nightly/opt-marionette-headless-e10s
> test-macosx64/debug-marionette-e10s
> test-macosx64/debug-marionette-headless-e10s
> test-windows10-64-devedition/opt-marionette-e10s
> test-windows10-64-devedition/opt-marionette-headless-e10s
> test-windows10-64-nightly/opt-marionette-e10s
> test-windows10-64-nightly/opt-marionette-headless-e10s
> test-windows10-64/debug-marionette-e10s
> test-windows10-64/debug-marionette-headless-e10s
> test-windows7-32-devedition/opt-marionette-e10s
> test-windows7-32-devedition/opt-marionette-headless-e10s
> test-windows7-32-nightly/opt-marionette-e10s
> test-windows7-32-nightly/opt-marionette-headless-e10s
> test-windows7-32/debug-marionette-e10s
> test-windows7-32/debug-marionette-headless-e10s

I will try with `trunk` now.

Comment 36

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Except on try where I only get a single output for `source-test-python-marionette-harness-py2` it all looks fine. I will as best push another try build to verify in CI.

Comment 37

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/253998/diff/2-3/

Comment 38

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Tom Prince [:tomprince] from comment #33)
> > +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']
> 
> `comm-central` has its tests defined elsewhere, so it won't get run there
> for other reasons. I think `trunk` captures the intent here better.
> 
> I think talos should probably also change (at some point).

Joel, please note this FYI.

Comment 39

[Joel Maher ( :jmaher ) (UTC -8)]

:tomprince, what should talos change to- we can get a bug on file and work on it; talos run only on trunk (autoland/inbound/try/central) and mozilla-beta; it is all in tree except for 2 tooltool artifacts that live out of tree.  Are you suggesting we need to adjust the build-projects we use and annotate out all the branches to run on?

Comment 40

[Tom Prince [:tomprince]]

:jmaher I'm just suggesting that generally, if we want something to run on all of m-i, m-c and autoland, that it probably makes more sense to use the alias for that, rather than spelling it out everytime. So, talos would be `['mozilla-beta', 'trunk', 'try']` or `['trunk', 'try']` (looking at https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/talos.yml#319 I see some tasks running in each of those configurations). 

Doing it this way means that when m-i goes away, for example, there are fewer places to change.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3cb7eb89668a
Do not run Marionette tests for Android on release branches. r=dustin,tomprince

Comment 42

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/3cb7eb89668a

Comment 43

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please uplift this patch to beta so it will stop the perma failures for MN jobs on Android. Thanks.

Comment 44

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/0a939a1cd7f9

Comment 45

[Intermittent Failures Robot]

38 failures in 622 pushes (0.061 failures/push) were associated with this bug in the last 7 days. 

This is the #28 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-beta: 38

Platform breakdown:
* android-em-4-3-armv7-api16: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1469534&startday=2018-07-02&endday=2018-07-08&tree=trunk

Comment 46

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.