Only run Marionette unit tests on Android for integration branches and try

RESOLVED FIXED in Firefox 62

Status

defect
P1
normal
RESOLVED FIXED
Last year
11 months ago

People

(Reporter: ebalazs_, Assigned: whimboo)

Tracking

({intermittent-failure, regression})

62 Branch
mozilla63
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox-esr60 unaffected, firefox61 unaffected, firefox62 fixed, firefox63 fixed)

Details

Attachments

(1 attachment)

INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py', '--emulator', '--app=fennec', '--package=org.mozilla.firefox_beta', '--address=localhost:2828', '/builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini', '--disable-e10s', '--gecko-log=-', '-vv', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/marionette_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log', '--symbols-path=https://queue.taskcluster.net/v1/task/EkK3zH_XTRSsi0FO5jPCJg/artifacts/public/build/target.crashreporter-symbols.zip', '--startup-timeout=300', '--this-chunk', '2', '--total-chunks', '10'] in /builds/worker/workspace/build/tests/marionette
[task 2018-06-19T04:01:31.698Z] 04:01:31     INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py --emulator --app=fennec --package=org.mozilla.firefox_beta --address=localhost:2828 /builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit-tests.ini --disable-e10s --gecko-log=- -vv --log-raw=/builds/worker/workspace/build/blobber_upload_dir/marionette_raw.log --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log --symbols-path=https://queue.taskcluster.net/v1/task/EkK3zH_XTRSsi0FO5jPCJg/artifacts/public/build/target.crashreporter-symbols.zip --startup-timeout=300 --this-chunk 2 --total-chunks 10
[task 2018-06-19T04:01:31.698Z] 04:01:31     INFO - Using env: (same as previous command)
[task 2018-06-19T04:01:31.908Z] 04:01:31     INFO -  Using workspace for temporary data: "/builds/worker/workspace/build/tests/marionette"
[task 2018-06-19T04:01:32.123Z] 04:01:32     INFO -  adb adbd running as root
[task 2018-06-19T04:01:32.226Z] 04:01:32     INFO -  adb su -c supported
[task 2018-06-19T04:01:32.329Z] 04:01:32     INFO -  adb su 0 supported
[task 2018-06-19T04:01:32.534Z] 04:01:32     INFO -  adb /system/bin/ls -a supported
[task 2018-06-19T04:01:32.637Z] 04:01:32     INFO -  adb Native cp support: True
[task 2018-06-19T04:01:32.740Z] 04:01:32     INFO -  adb Native chmod -R support: True
[task 2018-06-19T04:01:32.843Z] 04:01:32     INFO -  adb Setting SELinux Permissive Mode
[task 2018-06-19T04:01:32.946Z] 04:01:32     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.
[task 2018-06-19T04:01:40.101Z] 04:01:40     INFO -  timed out waiting for profiles.ini
[task 2018-06-19T04:01:42.570Z] 04:01:42     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpzG_guRpidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-19T04:01:44.225Z] 04:01:44     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-19T04:01:46.880Z] 04:01:46     INFO -  - waiting for device -
[task 2018-06-19T04:06:51.125Z] 04:06:51     INFO -  {"thread": "MainThread", "level": "ERROR", "pid": 1027, "source": "Marionette test runner", "time": 1529381211123, "action": "log", "message": "Failure during harness execution", "stack": "Traceback (most recent call last):\n\n  File \"/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 94, in cli\n    failed = harness_instance.run()\n\n  File \"/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 74, in run\n    runner.run_tests(tests)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/runner/base.py\", line 857, in run_tests\n    self.marionette = self.driverclass(**self._build_kwargs())\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 615, in __init__\n    self.start_binary(self.startup_timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 639, in start_binary\n    reraise(IOError, msg.format(timeout), tb)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 630, in start_binary\n    self.raise_for_port(timeout=timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 710, in raise_for_port\n    self.host, self.port))\n\nIOError: Process killed after 300s because no connection to Marionette server could be established. Check gecko.log for errors\n"}
[task 2018-06-19T04:06:51.744Z] 04:06:51     INFO -  cleanup aborted: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell rm /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini; echo rc=$?, exitcode: 255, stdout: rm failed for /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini, No such file or directory
[task 2018-06-19T04:06:52.052Z] 04:06:52     INFO -  Traceback (most recent call last):
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py", line 497, in close
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -      "tcp:{}".format(self.marionette_port))
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 977, in remove_forwards
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -      self.command_output(cmd, timeout=timeout)
[task 2018-06-19T04:06:52.053Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 888, in command_output
[task 2018-06-19T04:06:52.054Z] 04:06:52     INFO -      timeout=timeout)
[task 2018-06-19T04:06:52.054Z] 04:06:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 293, in command_output
[task 2018-06-19T04:06:52.054Z] 04:06:52 CRITICAL -      raise ADBProcessError(adb_process)
[task 2018-06-19T04:06:52.054Z] 04:06:52 CRITICAL -  ADBProcessError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device forward --remove tcp:2828, exitcode: 1, stdout: error: listener 'tcp:2828' not found
[task 2018-06-19T04:06:52.065Z] 04:06:52    ERROR - Return code: 1
[task 2018-06-19T04:06:52.066Z] 04:06:52    ERROR - No tests run or test summary not found
[task 2018-06-19T04:06:52.066Z] 04:06:52     INFO - TinderboxPrint: marionette<br/><em class="testfail">T-FAIL</em>
[task 2018-06-19T04:06:52.066Z] 04:06:52     INFO - ##### marionette log ends
[task 2018-06-19T04:06:52.066Z] 04:06:52 CRITICAL - # TBPL RETRY #
[task 2018-06-19T04:06:52.067Z] 04:06:52  WARNING - setting return code to 4
[task 2018-06-19T04:06:52.067Z] 04:06:52 CRITICAL - The marionette suite: marionette ran with return status: RETRY
Firefox doesn't seem to have been started and Marionette run into a timeout error.
OS: Unspecified → Android
Hardware: Unspecified → ARM
Priority: -- → P3
This is permafailing on Beta since the uplift of 62 there. I don't think P3 is the right priority there unless we no longer care about Android Mn jobs.
Flags: needinfo?(hskupin)
Flags: needinfo?(ato)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #2)
> This is permafailing on Beta since the uplift of 62 there. I don't think P3
> is the right priority there unless we no longer care about Android Mn jobs.

So why does no-one mention that? When I check mozilla-beta for Android Mn jobs I see all the failing jobs marked as expected fail??? And this without using this bug number! This is absolutely not what should have happened here. Sebastian, can you please work with sheriffs so that those things are getting escalated way earlier?

Did any previously run nightly-to-beta test merge showed those failures? If not I wonder if that is caused by some late push to mozilla-central right before the official merge.
Flags: needinfo?(ryanvm)
Flags: needinfo?(hskupin)
Flags: needinfo?(aryx.bugmail)
Priority: P3 → P1
Here the link to the Mn results on mozilla-beta:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=android%20mn

There is no uploaded log available. As such I triggered a one click loaner and was able to see the hang here:

> 08:08:18     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity
> .MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_C
> RASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,Medi
> aPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 N
> O_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpPiWu7fpidlog --es env9 R_LOG_LEVEL=6
> 08:08:20     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
> 08:08:21     INFO -  timed out waiting for 'org.mozilla.firefox_beta' process to start
> 08:08:22     INFO -  - waiting for device -

Firefox starts but we seem to no longer detect that, or Marionette doesn't start-up. Then after 300s the following traceback is visible:

>        08:13:27     INFO -  {"thread": "MainThread", "level": "ERROR", "pid": 1135, "source": "Marionette test runner", "time": 1529914407010, 
> "action": "log", "message": "Failure during harness execution", "stack": "Traceback (most recent call last):\n\n  File \"/builds/worker/workspac
> e/build/tests/marionette/harness/marionette_harness/runtests.py\", line 94, in cli\n    failed = harness_instance.run()\n\n  File \"/builds/work
> er/workspace/build/tests/marionette/harness/marionette_harness/runtests.py\", line 74, in run\n    runner.run_tests(tests)\n\n  File \"/builds/w
> orker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/runner/base.py\", line 857, in run_tests\n    self.marionette = self.d
> riverclass(**self._build_kwargs())\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\"
> , line 615, in __init__\n    self.start_binary(self.startup_timeout)\n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages
> /marionette_driver/marionette.py\", line 639, in start_binary\n    reraise(IOError, msg.format(timeout), tb)\n\n  File \"/builds/worker/workspac
> e/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 630, in start_binary\n    self.raise_for_port(timeout=timeout)\
> n\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 710, in raise_for_port\n    
> self.host, self.port))\n\nIOError: Process killed after 300s because no connection to Marionette server could be established. Check gecko.log fo
> r errors\n"}
> 08:13:27     INFO -  cleanup aborted: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device
>  shell rm /data/data/org.mozilla.firefox_beta/files/mozilla/profiles.ini; echo rc=$?, exitcode: 255, stdout: rm failed for /data/data/org.mozill
> a.firefox_beta/files/mozilla/profiles.ini, No such file or directory
> 08:13:27     INFO -  Traceback (most recent call last):
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py", line 497, in c
> lose
> 08:13:27     INFO -      "tcp:{}".format(self.marionette_port))
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 984, in remove_forwards
> 08:13:27     INFO -      self.command_output(cmd, timeout=timeout)
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 895, in command_output
> 08:13:27     INFO -      timeout=timeout)
> 08:13:27     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 293, in command_output
> 08:13:27 CRITICAL -      raise ADBProcessError(adb_process)
> 08:13:27 CRITICAL -  ADBProcessError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device
>  forward --remove tcp:2828, exitcode: 1, stdout: error: listener 'tcp:2828' not found

With bug 1440714 we got some major changes for Android but those were already for Firefox 61. So it shouldn't have affected us here for the mozilla-central to mozilla-beta merge of Firefox 62.
> 08:08:22 INFO - - waiting for device -

This would mean that we may have lost the connection to the device? Geoff, or Bob do you have an idea?
Flags: needinfo?(gbrown)
Flags: needinfo?(bob)
Flags: needinfo?(ryanvm)
(In reply to Henrik Skupin (:whimboo) from comment #4)
> There is no uploaded log available. As such I triggered a one click loaner
> and was able to see the hang here:

The treeherder log buttons are inactive, but actually, all the logs are available as artifacts on the Job Details pane.
The logcat artifact shows Firefox starting and living until being killed after the 5 minute timeout:

https://taskcluster-artifacts.net/c2_GisYeTQymnVCg33nhZw/5/public/test_info//logcat-emulator-5554.log

06-23 20:36:18.414   277   289 I ActivityManager: Start proc org.mozilla.firefox_beta for activity org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp: pid=760 uid=10043 gids={50043, 3003, 1028, 1015, 1006}
06-23 20:36:19.064   760   760 I GeckoApplication: zerdatime 141494 - application start
...
06-23 20:41:21.184    38    38 D Zygote  : Process 760 terminated by signal (15)
06-23 20:41:21.194   277   288 I ActivityManager: Process org.mozilla.firefox_beta (pid 760) has died.


" - waiting for device - " seems a bit odd, but might be nothing to worry about...needs more investigation.

But given that Firefox is starting, "IOError: Process killed after 300s because no connection to Marionette server could be established." seems to be the real trouble.
The message 'waiting for device' appears to be created here: https://searchfox.org/mozilla-central/source/media/webrtc/trunk/build/android/adb_device_functions.sh#119 and does seem to say the connection was lost.

[task 2018-06-24T03:33:24.087Z] 03:33:24     INFO -  adb adbd running as root
[task 2018-06-24T03:33:24.190Z] 03:33:24     INFO -  adb su -c supported
[task 2018-06-24T03:33:24.293Z] 03:33:24     INFO -  adb su 0 supported
[task 2018-06-24T03:33:24.499Z] 03:33:24     INFO -  adb /system/bin/ls -a supported
[task 2018-06-24T03:33:24.602Z] 03:33:24     INFO -  adb Native cp support: True
[task 2018-06-24T03:33:24.706Z] 03:33:24     INFO -  adb Native chmod -R support: True
[task 2018-06-24T03:33:24.809Z] 03:33:24     INFO -  adb Unable to turn off logcat chatty
[task 2018-06-24T03:33:24.912Z] 03:33:24     INFO -  adb Setting SELinux Permissive Mode
[task 2018-06-24T03:33:25.015Z] 03:33:25     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.

up to this point, we had a connection via adb.py.

[task 2018-06-24T03:33:31.777Z] 03:33:31     INFO -  timed out waiting for profiles.ini

This is from https://searchfox.org/mozilla-central/source/testing/mozbase/mozrunner/mozrunner/devices/base.py#99 which seems to imply the connection was not lost (no ADBTimeoutError) but that the profile was not pushed to the emulator/device. This seems it should be fatal.


[task 2018-06-24T03:33:34.246Z] 03:33:34     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpE15XVupidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-24T03:33:35.903Z] 03:33:35     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.firefox_beta/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-24T03:33:36.889Z] 03:33:36     INFO -  timed out waiting for 'org.mozilla.firefox_beta' process to start
[task 2018-06-24T03:33:37.954Z] 03:33:37     INFO -  - waiting for device -

Looking at the logcat logs doesn't help but I'm not familiar with how it looked when it ran ok. I didn't see anything about marionette or adb forward though. Fennec was started then killed via SIGTERM however. Is marionette starting dependent on the profile being uploaded?
On mozilla-central, successful runs look like:

https://treeherder.mozilla.org/logviewer.html#?job_id=184662914&repo=mozilla-central&lineNumber=1323-1337

[task 2018-06-25T10:36:35.099Z] 10:36:35     INFO -  adb Unable to set SELinux Permissive due to args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device shell setenforce Permissive; echo rc=$?, exitcode: 1, stdout: setenforce: SELinux is disabled.
[task 2018-06-25T10:36:42.165Z] 10:36:42     INFO -  timed out waiting for profiles.ini
[task 2018-06-25T10:36:44.636Z] 10:36:44     INFO -  Application command: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb shell am start -a android.activity.MAIN -n org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp --es args '-no-remote -profile /sdcard/tests/profile -marionette' --es env0 MOZ_CRASHREPORTER=1 --es env1 R_LOG_VERBOSE=1 --es env2 MOZ_HIDE_RESULTS_TABLE=1 --es env3 MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4,MediaPipelineFactory:4 --es env4 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env5 R_LOG_DESTINATION=stderr --es env6 MOZ_CRASHREPORTER_NO_REPORT=1 --es env7 NO_EM_RESTART=1 --es env8 MOZ_PROCESS_LOG=/tmp/tmpMRWpUnpidlog --es env9 R_LOG_LEVEL=6
[task 2018-06-25T10:36:46.368Z] 10:36:46     INFO -  Starting: Intent { act=android.activity.MAIN cmp=org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp (has extras) }
[task 2018-06-25T10:36:49.244Z] 10:36:49     INFO -  - waiting for device -
[task 2018-06-25T10:38:19.308Z] 10:38:19     INFO -  Profile path is /tmp/tmp9mhLNK.mozrunner
[task 2018-06-25T10:38:19.309Z] 10:38:19     INFO -  Starting fixture servers
[task 2018-06-25T10:38:20.514Z] 10:38:20     INFO -  Fixture server listening on http://172.17.0.5:37797/
[task 2018-06-25T10:38:20.515Z] 10:38:20     INFO -  Fixture server listening on https://172.17.0.5:40732/
[task 2018-06-25T10:38:20.522Z] 10:38:20     INFO -  mozinfo updated from: /builds/worker/workspace/build/tests/mozinfo.json
[task 2018-06-25T10:38:20.524Z] 10:38:20     INFO -  mozinfo is: {u'bin_suffix': u'', u'official': True, u'allow_legacy_extensions': True, u'pgo': False, u'sync': False, u'buildapp': u'mobile/android', u'crashreporter': True, u'devedition': False, u'require_signing': False, u'platform_guess': u'android-arm', u'appname': u'fennec', u'stylo': True, u'mozconfig': u'/builds/worker/workspace/build/src/.mozconfig', u'topsrcdir': u'/builds/worker/workspace/build/src', u'ubsan': False, 'os_version': StringVersion ('16.04'), 'version': 'Ubuntu 16.04', 'webrender': False, u'buildtype_guess': u'opt', 'bits': 32, 'has_sandbox': True, u'datareporting': True, u'artifact': False, u'toolkit': u'android', u'healthreport': True, u'updater': True, u'cc_type': u'clang', u'asan': False, u'release_or_beta': False, u'ccov': False, u'android_min_sdk': u'16', u'tests_enabled': True, 'linux_distro': 'Ubuntu', u'telemetry': False, u'tsan': False, u'nightly_build': True, u'debug': False, 'os': u'android', 'processor': u'arm'}
[task 2018-06-25T10:38:22.132Z] 10:38:22     INFO -  e10s is disabled
[task 2018-06-25T10:38:22.179Z] 10:38:22     INFO -  Failed to retrieve version information for None
[task 2018-06-25T10:38:22.180Z] 10:38:22     INFO -  SUITE-START | Running 50 tests
[task 2018-06-25T10:38:22.181Z] 10:38:22     INFO -  TEST-START | testing/marionette/harness/marionette_harness/tests/unit/test_cli_arguments.py
On mozilla-central, logcat from successful runs includes:

06-25 03:45:23.056   756   791 I Gecko   : 1529923522768	addons.xpi	WARN	List of valid built-in add-ons could not be parsed.: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]"  nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: startup :: line 2093"  data: no] Stack trace: startup()@resource://gre/modules/addons/XPIProvider.jsm:2093
06-25 03:45:23.056   756   791 I Gecko   : callProvider()@resource://gre/modules/AddonManager.jsm:206
06-25 03:45:23.056   756   791 I Gecko   : _startProvider()@resource://gre/modules/AddonManager.jsm:654
06-25 03:45:23.056   756   791 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:813
06-25 03:45:23.056   756   791 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:2808
06-25 03:45:23.056   756   791 I Gecko   : observe()@jar:jar:file:///data/app/org.mozilla.fennec_aurora-1.apk!/assets/omni.ja!/components/addonManager.js:66
06-25 03:45:24.636   756   791 I Gecko   : 1529923524637	addons.xpi	WARN	No list of valid system add-ons found.
06-25 03:45:26.246   756   791 D GeckoThread: State changed to PROFILE_READY
06-25 03:45:26.736   756   791 I Gecko   : 1529923526735	Marionette	DEBUG	Received observer notification profile-after-change


On beta, there are no Marionette messages:

06-23 20:37:38.334   760   799 I Gecko   : 1529811458134	addons.xpi	WARN	List of valid built-in add-ons could not be parsed.: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]"  nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: startup :: line 2093"  data: no] Stack trace: startup()@resource://gre/modules/addons/XPIProvider.jsm:2093
06-23 20:37:38.334   760   799 I Gecko   : callProvider()@resource://gre/modules/AddonManager.jsm:206
06-23 20:37:38.334   760   799 I Gecko   : _startProvider()@resource://gre/modules/AddonManager.jsm:654
06-23 20:37:38.334   760   799 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:813
06-23 20:37:38.334   760   799 I Gecko   : startup()@resource://gre/modules/AddonManager.jsm:2808
06-23 20:37:38.334   760   799 I Gecko   : observe()@jar:jar:file:///data/app/org.mozilla.firefox_beta-1.apk!/assets/omni.ja!/components/addonManager.js:66
06-23 20:37:40.004   760   799 I Gecko   : 1529811460006	addons.xpi	WARN	No list of valid system add-ons found.
06-23 20:37:41.613   760   799 D GeckoThread: State changed to PROFILE_READY
I wonder if there's a changed addon that needs signing or something. That's the only thing I can think of that would explain why the simulations on Try are OK and "real" Beta isn't.
I haven't fully parsed this, but it certainly seems related:

https://hg.mozilla.org/releases/mozilla-beta/rev/2ad47b8c9555 / bug 1460315

Maybe we shouldn't expect marionette to be enabled on beta?
Flags: needinfo?(bob)
Discussed on #ateam: Android marionette is intentionally disabled on beta -- we should stop trying to run Android Mn tests on beta.
Flags: needinfo?(gbrown)
(In reply to Geoff Brown [:gbrown] from comment #13)
> Discussed on #ateam: Android marionette is intentionally disabled on beta --
> we should stop trying to run Android Mn tests on beta.

This is not true for debug builds which also fail permanently. There is something else which caused this regression.
We originally enabled marionette on all Android/debug builds, but bug 1460315 changed that to enable on opt+debug on select channels -- not including beta. I'm not sure of the back-story, but I believe we do not want marionette enabled in Firefox releases, presumably including beta. That seems to be the intention of the change in bug 1460315.
(In reply to Geoff Brown [:gbrown] from comment #15)
> We originally enabled marionette on all Android/debug builds, but bug
> 1460315 changed that to enable on opt+debug on select channels -- not
> including beta. I'm not sure of the back-story, but I believe we do not want

That is not exactly what we originally discussed because we never decided to limit the testing off debug builds. But
seeing that this is the case now we would have to stop testing any kind of Android builds on those affected channels.

Problem here is that we cannot safely uplift changes for Marionette to mozilla-beta, or an esr-branch anymore because no testing is effectively in place for both of them, and we cannot guarantee that no regression is getting introduced during the beta cycle. 

Given that I'm not the one who can decide that I will set ni? for David, and James.
Blocks: 1460315
Flags: needinfo?(snorp)
Flags: needinfo?(dburns)
(In reply to Henrik Skupin (:whimboo) from comment #16)

> Problem here is that we cannot safely uplift changes for
> Marionette to mozilla-beta, or an esr-branch anymore because no
> testing is effectively in place for both of them, and we cannot
> guarantee that no regression is getting introduced during the beta
> cycle.

Because Marionette is _not enabled_ in beta or esr on Fennec,
uplifting should not be a problem.

Marionette support for Fennec is best-effort and until we decide
it is good enough to ship as part of regular Fennec releases, the
current practice of ensuring patches to central don’t break Fennec
Nightly seems like the best we can do.  Enabling Marionette n debug
and not opt on Fennec beta makes little sense to me.
I agree with Andreas here. Until we have done a RRA on marionette on Mobile we shouldnt really worry about things other than nightly unless there is a need for it from the geckoview team
Flags: needinfo?(dburns)
Alright then. I will upload a patch shortly which should disable Mn jobs on Android for release branches.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Summary: ADBProcessError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb -s emulator-5554 wait-for-device forward --remove tcp:2828, exitcode: 1, stdout: error: listener 'tcp:2828' not found → Only run Marionette unit tests on Android for integration branches and try
Dustin, as it looks like the patch as attached doesn't work and cause an error in the decision task. How can I define the default for a list? Thanks.
Flags: needinfo?(dustin)
`default: default` gives a default value of `"default"`, but that's not allowed.  I think what you want there is `default: built-projects`.
Flags: needinfo?(dustin)
Keywords: regression
Version: Version 3 → 62 Branch
(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #22)
> `default: default` gives a default value of `"default"`, but that's not
> allowed.  I think what you want there is `default: built-projects`.

Thanks. Lets give it a try.
Flags: needinfo?(snorp)
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261574

The try build looks fine now. But I can only test this on try. No idea how to make sure the tests won't run on release branches.

Dustin, I assume this is something tested somewhere else, and you may the best reviewer for the patch. Thanks.
I will be on PTO soon. So whenever this gets a r+ can someone please push to autoland? If it gets a r- Andreas or David will pick it up.
Flags: needinfo?(ato)
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261882

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

I think this should be `['trunk', 'try']`. The list of projects that this includes is defined [here](https://searchfox.org/mozilla-central/rev/6ef785903fee6c0b16a1eab79d722373d940fd78/taskcluster/taskgraph/util/attributes.py#15)
Attachment #8988840 - Flags: review+
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261574

You can see what will get run on a beta by doing something like `mach taskgraph target-graph -p project=mozilla-beta`. You can also filter it by adding `--tasks-regex '.*marionette.*'.
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261882

> I think this should be `['trunk', 'try']`. The list of projects that this includes is defined [here](https://searchfox.org/mozilla-central/rev/6ef785903fee6c0b16a1eab79d722373d940fd78/taskcluster/taskgraph/util/attributes.py#15)

I was just following what eg. Talos does: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/talos.yml#319

`trunk` would include `comm-central` where Marionette is not supported.
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261900

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

I think that makes sense (to exclude comm-central)
Attachment #8988840 - Flags: review?(dustin) → review+
The full taskgraph:

> % ./mach taskgraph target-graph -p 'project=mozilla-beta' --tasks-regex '.*marionette.*'
> …
> source-test-python-marionette-harness-py2
> test-android-em-4.3-arm7-api-16/debug-marionette-1
> test-android-em-4.3-arm7-api-16/debug-marionette-10
> test-android-em-4.3-arm7-api-16/debug-marionette-2
> test-android-em-4.3-arm7-api-16/debug-marionette-3
> test-android-em-4.3-arm7-api-16/debug-marionette-4
> test-android-em-4.3-arm7-api-16/debug-marionette-5
> test-android-em-4.3-arm7-api-16/debug-marionette-6
> test-android-em-4.3-arm7-api-16/debug-marionette-7
> test-android-em-4.3-arm7-api-16/debug-marionette-8
> test-android-em-4.3-arm7-api-16/debug-marionette-9
> test-android-em-4.3-arm7-api-16/opt-marionette-1
> test-android-em-4.3-arm7-api-16/opt-marionette-10
> test-android-em-4.3-arm7-api-16/opt-marionette-2
> test-android-em-4.3-arm7-api-16/opt-marionette-3
> test-android-em-4.3-arm7-api-16/opt-marionette-4
> test-android-em-4.3-arm7-api-16/opt-marionette-5
> test-android-em-4.3-arm7-api-16/opt-marionette-6
> test-android-em-4.3-arm7-api-16/opt-marionette-7
> test-android-em-4.3-arm7-api-16/opt-marionette-8
> test-android-em-4.3-arm7-api-16/opt-marionette-9
> test-linux32-devedition/opt-marionette-e10s
> test-linux32-nightly/opt-marionette-e10s
> test-linux32/debug-marionette-e10s
> test-linux64-asan/opt-marionette-e10s
> test-linux64-asan/opt-marionette-headless-e10s
> test-linux64-devedition/opt-marionette-e10s
> test-linux64-devedition/opt-marionette-headless-e10s
> test-linux64-nightly/opt-marionette-e10s
> test-linux64-nightly/opt-marionette-headless-e10s
> test-linux64/debug-marionette-e10s
> test-linux64/debug-marionette-headless-e10s
> test-macosx64-devedition/opt-marionette-e10s
> test-macosx64-devedition/opt-marionette-headless-e10s
> test-macosx64-nightly/opt-marionette-e10s
> test-macosx64-nightly/opt-marionette-headless-e10s
> test-macosx64/debug-marionette-e10s
> test-macosx64/debug-marionette-headless-e10s
> test-windows10-64-devedition/opt-marionette-e10s
> test-windows10-64-devedition/opt-marionette-headless-e10s
> test-windows10-64-nightly/opt-marionette-e10s
> test-windows10-64-nightly/opt-marionette-headless-e10s
> test-windows10-64/debug-marionette-e10s
> test-windows10-64/debug-marionette-headless-e10s
> test-windows7-32-devedition/opt-marionette-e10s
> test-windows7-32-devedition/opt-marionette-headless-e10s
> test-windows7-32-nightly/opt-marionette-e10s
> test-windows7-32-nightly/opt-marionette-headless-e10s
> test-windows7-32/debug-marionette-e10s

Are we missing anything with the way this patch is written?
Flags: needinfo?(mozilla)
Flags: needinfo?(hskupin)
Flags: needinfo?(ato)
Comment on attachment 8988840 [details]
Bug 1469534 - Do not run Marionette tests for Android on release branches.

https://reviewboard.mozilla.org/r/253998/#review261940

::: taskcluster/ci/test/marionette.yml:27
(Diff revision 2)
>  marionette:
>      description: "Marionette unittest run"
>      treeherder-symbol: Mn
> +    run-on-projects:
> +        by-test-platform:
> +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']

`comm-central` has its tests defined elsewhere, so it won't get run there for other reasons. I think `trunk` captures the intent here better.

I think talos should probably also change (at some point).
Andreas, are you testing with the patch applied? I don't get the android tasks when I test locally.
Flags: needinfo?(mozilla)
(In reply to Andreas Tolfsen 「:ato」 from comment #32)
> > test-android-em-4.3-arm7-api-16/debug-marionette-1
> > test-android-em-4.3-arm7-api-16/debug-marionette-10
> > test-android-em-4.3-arm7-api-16/debug-marionette-2
> > test-android-em-4.3-arm7-api-16/debug-marionette-3
> > test-android-em-4.3-arm7-api-16/debug-marionette-4
> > test-android-em-4.3-arm7-api-16/debug-marionette-5
> > test-android-em-4.3-arm7-api-16/debug-marionette-6
> > test-android-em-4.3-arm7-api-16/debug-marionette-7
> > test-android-em-4.3-arm7-api-16/debug-marionette-8
> > test-android-em-4.3-arm7-api-16/debug-marionette-9
> > test-android-em-4.3-arm7-api-16/opt-marionette-1
> > test-android-em-4.3-arm7-api-16/opt-marionette-10
> > test-android-em-4.3-arm7-api-16/opt-marionette-2
> > test-android-em-4.3-arm7-api-16/opt-marionette-3
> > test-android-em-4.3-arm7-api-16/opt-marionette-4
> > test-android-em-4.3-arm7-api-16/opt-marionette-5
> > test-android-em-4.3-arm7-api-16/opt-marionette-6
> > test-android-em-4.3-arm7-api-16/opt-marionette-7
> > test-android-em-4.3-arm7-api-16/opt-marionette-8
> > test-android-em-4.3-arm7-api-16/opt-marionette-9
> 
> Are we missing anything with the way this patch is written?

Not sure what you were running here but I assume my patch was not applied to your local tree. When I run the proposed command I see the following output, which is fine:

> source-test-python-marionette-harness-py2
> test-linux32-devedition/opt-marionette-e10s
> test-linux32-nightly/opt-marionette-e10s
> test-linux32/debug-marionette-e10s
> test-linux64-asan/opt-marionette-e10s
> test-linux64-asan/opt-marionette-headless-e10s
> test-linux64-devedition/opt-marionette-e10s
> test-linux64-devedition/opt-marionette-headless-e10s
> test-linux64-nightly/opt-marionette-e10s
> test-linux64-nightly/opt-marionette-headless-e10s
> test-linux64/debug-marionette-e10s
> test-linux64/debug-marionette-headless-e10s
> test-macosx64-devedition/opt-marionette-e10s
> test-macosx64-devedition/opt-marionette-headless-e10s
> test-macosx64-nightly/opt-marionette-e10s
> test-macosx64-nightly/opt-marionette-headless-e10s
> test-macosx64/debug-marionette-e10s
> test-macosx64/debug-marionette-headless-e10s
> test-windows10-64-devedition/opt-marionette-e10s
> test-windows10-64-devedition/opt-marionette-headless-e10s
> test-windows10-64-nightly/opt-marionette-e10s
> test-windows10-64-nightly/opt-marionette-headless-e10s
> test-windows10-64/debug-marionette-e10s
> test-windows10-64/debug-marionette-headless-e10s
> test-windows7-32-devedition/opt-marionette-e10s
> test-windows7-32-devedition/opt-marionette-headless-e10s
> test-windows7-32-nightly/opt-marionette-e10s
> test-windows7-32-nightly/opt-marionette-headless-e10s
> test-windows7-32/debug-marionette-e10s
> test-windows7-32/debug-marionette-headless-e10s

I will try with `trunk` now.
Flags: needinfo?(hskupin)
Except on try where I only get a single output for `source-test-python-marionette-harness-py2` it all looks fine. I will as best push another try build to verify in CI.
(In reply to Tom Prince [:tomprince] from comment #33)
> > +            android-em.*: ['mozilla-central', 'mozilla-inbound', 'autoland', 'try']
> 
> `comm-central` has its tests defined elsewhere, so it won't get run there
> for other reasons. I think `trunk` captures the intent here better.
> 
> I think talos should probably also change (at some point).

Joel, please note this FYI.
Flags: needinfo?(jmaher)
:tomprince, what should talos change to- we can get a bug on file and work on it; talos run only on trunk (autoland/inbound/try/central) and mozilla-beta; it is all in tree except for 2 tooltool artifacts that live out of tree.  Are you suggesting we need to adjust the build-projects we use and annotate out all the branches to run on?
Flags: needinfo?(jmaher) → needinfo?(mozilla)
:jmaher I'm just suggesting that generally, if we want something to run on all of m-i, m-c and autoland, that it probably makes more sense to use the alias for that, rather than spelling it out everytime. So, talos would be `['mozilla-beta', 'trunk', 'try']` or `['trunk', 'try']` (looking at https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/talos.yml#319 I see some tasks running in each of those configurations). 

Doing it this way means that when m-i goes away, for example, there are fewer places to change.
Flags: needinfo?(mozilla)
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3cb7eb89668a
Do not run Marionette tests for Android on release branches. r=dustin,tomprince
https://hg.mozilla.org/mozilla-central/rev/3cb7eb89668a
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Please uplift this patch to beta so it will stop the perma failures for MN jobs on Android. Thanks.
Whiteboard: [checkin-needed-beta]
Whiteboard: [checkin-needed-beta]
Blocks: 1484220
You need to log in before you can comment on or make changes to this bug.