1981210 - [mozdevice] Force close a formerly running application to avoid port usage conflicts

Status: RESOLVED FIXED

(Testing :: Mozbase Rust, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

as seen on this try push, wdspec is basically orange across the board with no pattern of specific failing tests.

I see this specific error:
[task 2025-07-28T17:42:45.511+00:00] 17:42:45 INFO - TEST-UNEXPECTED-ERROR | /webdriver/tests/classic/close_window/close.py | test_element_usage_after_closing_browsing_context - setup error: webdriver.error.UnknownErrorException: unknown error (500): Could not launch Android org.mozilla.geckoview.test_runner/.App: Resource temporarily unavailable (os error 11)

I am not sure why this fails, as it obviously works, it just intermittently fails.

This is not an issue on the existing android 7.0 emulator. I assume this has something to do with either how wdspec is run or interacting with the new emulator.

:whimboo, do you have any thoughts on how we can move forward with this?

Comment 1

[Julian Descottes [:jdescottes]]

(Henrik is away until next monday)

From what I can see in the logs, when the tests fail we are simultaneously trying to stop AND start geckoview: https://treeherder.mozilla.org/logviewer?job_id=519854084&repo=try&lineNumber=50904-50940

[task 2025-07-28T17:45:45.303+00:00] 17:45:45 INFO - PID 3953 | 1753724745300 geckodriver::android DEBUG Pushing GeckoView configuration file to /data/local/tmp/org.mozilla.geckoview.test_runner-geckoview-config.yaml
[task 2025-07-28T17:45:45.303+00:00] 17:45:45 INFO - PID 3953 | 1753724745300 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:45.304+00:00] 17:45:45 INFO - PID 3953 | 1753724745300 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:45.304+00:00] 17:45:45 INFO - PID 3953 | 1753724745300 mozdevice TRACE execute_host_command: >> "shell:ls /data/local/tmp"
[task 2025-07-28T17:45:45.318+00:00] 17:45:45 INFO - PID 3953 | 1753724745317 mozdevice TRACE execute_host_command: << "test_root\n"
[task 2025-07-28T17:45:45.362+00:00] 17:45:45 INFO - PID 3953 | 1753724745361 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:45.362+00:00] 17:45:45 INFO - PID 3953 | 1753724745361 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:45.363+00:00] 17:45:45 INFO - PID 3953 | 1753724745361 mozdevice TRACE execute_host_command: >> "shell:am set-debug-app --persistent org.mozilla.geckoview.test_runner"
[task 2025-07-28T17:45:45.389+00:00] 17:45:45 INFO - PID 3953 | 1753724745388 mozdevice TRACE execute_host_command: << ""
[task 2025-07-28T17:45:45.389+00:00] 17:45:45 INFO - PID 3953 | 1753724745388 geckodriver::android DEBUG Launching org.mozilla.geckoview.test_runner/.App
[task 2025-07-28T17:45:45.393+00:00] 17:45:45 INFO - PID 3953 | 1753724745392 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:45.393+00:00] 17:45:45 INFO - PID 3953 | 1753724745393 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:45.394+00:00] 17:45:45 INFO - PID 3953 | 1753724745393 mozdevice TRACE execute_host_command: >> "shell:am start -W -n org.mozilla.geckoview.test_runner/.App -a android.intent.action.VIEW -d about: ... dcard/Android/data/org.mozilla.geckoview.test_runner/files/test_root/org.mozilla.geckoview.test_runner-geckodriver-profile""
[task 2025-07-28T17:45:50.686+00:00] 17:45:50 INFO - PID 3953 | 1753724750685 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:50.686+00:00] 17:45:50 INFO - PID 3953 | 1753724750685 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:50.687+00:00] 17:45:50 INFO - PID 3953 | 1753724750685 mozdevice TRACE execute_host_command: >> "shell:am clear-debug-app org.mozilla.geckoview.test_runner"
[task 2025-07-28T17:45:50.716+00:00] 17:45:50 INFO - PID 3953 | 1753724750715 mozdevice TRACE execute_host_command: << ""
[task 2025-07-28T17:45:50.717+00:00] 17:45:50 INFO - PID 3953 | 1753724750715 geckodriver::android DEBUG Disabled reading from configuration file
[task 2025-07-28T17:45:50.717+00:00] 17:45:50 INFO - PID 3953 | 1753724750715 mozdevice DEBUG Deleting /data/local/tmp/org.mozilla.geckoview.test_runner-geckoview-config.yaml
[task 2025-07-28T17:45:50.717+00:00] 17:45:50 INFO - PID 3953 | 1753724750716 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:50.718+00:00] 17:45:50 INFO - PID 3953 | 1753724750716 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:50.718+00:00] 17:45:50 INFO - PID 3953 | 1753724750716 mozdevice TRACE execute_host_command: >> "shell:rm -rf /data/local/tmp/org.mozilla.geckoview.test_runner-geckoview-config.yaml"
[task 2025-07-28T17:45:50.737+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 mozdevice TRACE execute_host_command: << ""
[task 2025-07-28T17:45:50.737+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 geckodriver::android DEBUG Deleted GeckoView configuration file
[task 2025-07-28T17:45:50.737+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 mozdevice DEBUG Deleting /sdcard/Android/data/org.mozilla.geckoview.test_runner/files/test_root
[task 2025-07-28T17:45:50.738+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:50.738+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:50.739+00:00] 17:45:50 INFO - PID 3953 | 1753724750736 mozdevice TRACE execute_host_command: >> "shell:rm -rf /sdcard/Android/data/org.mozilla.geckoview.test_runner/files/test_root"
[task 2025-07-28T17:45:50.756+00:00] 17:45:50 INFO - PID 3953 | 1753724750755 mozdevice TRACE execute_host_command: << ""
[task 2025-07-28T17:45:50.757+00:00] 17:45:50 INFO - PID 3953 | 1753724750755 geckodriver::android DEBUG Deleted test root folder: /sdcard/Android/data/org.mozilla.geckoview.test_runner/files/test_root
[task 2025-07-28T17:45:50.757+00:00] 17:45:50 INFO - PID 3953 | 1753724750755 geckodriver::android DEBUG Stop forwarding Marionette port (40601 -> 2829)
[task 2025-07-28T17:45:50.757+00:00] 17:45:50 INFO - PID 3953 | 1753724750756 mozdevice TRACE execute_host_command: >> "host:transport:emulator-5554"
[task 2025-07-28T17:45:50.758+00:00] 17:45:50 INFO - PID 3953 | 1753724750756 mozdevice TRACE execute_host_command: << []
[task 2025-07-28T17:45:50.758+00:00] 17:45:50 INFO - PID 3953 | 1753724750756 mozdevice TRACE execute_host_command: >> "host-serial:emulator-5554:killforward:tcp:40601"
[task 2025-07-28T17:45:50.759+00:00] 17:45:50 INFO - PID 3953 | 1753724750756 mozdevice TRACE execute_host_command: << ""
[task 2025-07-28T17:45:50.759+00:00] 17:45:50 INFO - PID 3953 | 1753724750756 webdriver::server DEBUG <- 500 Internal Server Error {"value":{"error":"unknown error","message":"Could not launch Android org.mozilla.geckoview.test_runner/.App: Resource temporarily unavailable (os error 11)","stacktrace":""}}
[task 2025-07-28T17:45:50.843+00:00] 17:45:50 INFO - STDOUT: ERROR

See in the log above, I highlighted a "Launching" instruction coming from https://searchfox.org/mozilla-central/rev/f6a806c38c459e0e0d797d264ca0e8ad46005105/testing/geckodriver/src/android.rs#438, as well as a clear instruction coming from https://searchfox.org/mozilla-central/rev/f6a806c38c459e0e0d797d264ca0e8ad46005105/testing/geckodriver/src/android.rs#94

It's possible that we are trying to delete / modify folders that are required by the new Geckoview instance to work properly?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Joel, which steps do I have to take to reproduce it locally?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

to reproduce on try, you need this patch (rebased today):
https://hg-edge.mozilla.org/try/rev/8700822619f903bdd201f72d67a43c5578382a2a

to reproduce locally, you will need a few changes:

diff --git a/python/mozboot/mozboot/android.py b/python/mozboot/mozboot/android.py
index 4a2ab2b6d89e..15d78a348e1b 100644
--- a/python/mozboot/mozboot/android.py
+++ b/python/mozboot/mozboot/android.py
@@ -34,7 +34,7 @@ BUNDLETOOL_URL = f"https://github.com/google/bundletool/releases/download/{BUNDL
 X86_64_ANDROID_AVD = "linux64-android-avd-x86_64-repack"
 ARM64_ANDROID_AVD = "linux64-android-avd-arm64-repack"

-AVD_MANIFEST_X86_64 = Path(__file__).resolve().parent / "android-avds/x86_64.json"
+AVD_MANIFEST_X86_64 = Path(__file__).resolve().parent / "android-avds/android34-x86_64.json"
 AVD_MANIFEST_ARM64 = Path(__file__).resolve().parent / "android-avds/arm64.json"

 MOZBUILD_PATH = Path(get_state_dir())
diff --git a/testing/mozbase/mozrunner/mozrunner/devices/android_device.py b/testing/mozbase/mozrunner/mozrunner/devices/android_device.py
index b748a3130f18..de5acea3de87 100644
--- a/testing/mozbase/mozrunner/mozrunner/devices/android_device.py
+++ b/testing/mozbase/mozrunner/mozrunner/devices/android_device.py
@@ -159,7 +159,7 @@ AVD_DICT = {
     ),
     "x86_64": AvdInfo(
         "Android x86_64",
-        "mozemulator-x86_64",
+        "mozemulator-android34-x86_64",
         [
             "-skip-adb-auth",
             "-verbose",
@@ -171,12 +171,11 @@ AVD_DICT = {
             "3072",
             "-cores",
             "4",
-            "-skin",
-            "800x1280",
             "-prop",
             "ro.test_harness=true",
             "-no-snapstorage",
             "-no-snapshot",
+            "-skin", "1080x1920",
         ],
         True,
     ),

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Joel, sorry for the delay. I had too much bugs to keep up with last week. Now I've taken a look and there is a discrepancy that's not the clear to me. When I apply the above patches I do not get a Android 14 release but Android 11:

✗ adb shell getprop ro.build.version.release
11

Do I miss something?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

odd, i just tried ./mach android-emulator and did adb shell getprop ro.build.version.release, my result is 14.

I wonder if you have a cached version of the emulator? maybe it didn't start the emulator and adb is connecting to something else? A couple weeks ago was the first time I have ever ran the android emulator, so this is all new to me.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I removed all the files and run mach bootstrap again. What's suspicious is the android30 part, which actually corresponds to Android 11.

We are now installing the following Android packages:
system-images;android-30;default;arm64-v8a
platforms;android-36
build-tools;36.0.0
platform-tools
emulator

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, I run the reset sub command and now I see that python/mozboot/mozboot/android-avds/arm64.json is used. You don't seem to have updated it. Changing it's entry for android-30 to android-34 now downloads Android 14!

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

oh, is it possible your machine is a mac/aarch64? that might be what is going on.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, that is what I have. So you should probably update those other json files as well. Now when I'm trying to run wdspec tests it hangs in trying to fetch the host utilities. I'll check tomorrow what's wrong now.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Ted- we have been focused on updating the x86_64 emulator for CI, but :whimboo ran into an issue locally. Should we ensure all emulators are on android 14, for example: https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/android-avds/arm64.json ?

we currently have a mix

Comment 11

[Ted Campbell [:tcampbell]]

Yep, we should move that to whatever our default for x86_64 is. I believe it is Android 11 because Android 7 emulator/image combo didn't work on Apple Silicon. In the immediate term, we could add a android34-arm64 config until we are ready to remove the old android 7 builds so we don't disrupt local flows.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #9)

Yes, that is what I have. So you should probably update those other json files as well. Now when I'm trying to run wdspec tests it hangs in trying to fetch the host utilities. I'll check tomorrow what's wrong now.

I got the download of the host utils working and now all the tests run just fine. Could this be a problem with the host system the emulator runs on and some specific settings? Joel, can you please try to run locally as well given that you seem to have a x86_64 system? It would be good to know if it can be reproduced outside of CI or not.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

after spending a couple of hours trying to setup geckoview artifact or full builds, I am unable to do this, which means I cannot run the tests locally on an android emulator.

Possibly someone who works more frequently on android could run the tests locally on a x86_64 machine?

Comment 14

[Ted Campbell [:tcampbell]]

I reproduce locally though intermittently and only if I run a large enough number of tests. What appears to be happening is the previous force-stop may be unreliable so we fail to launch again with EAGAIN returned. From some reading, it sounds like force-stop doesn't do any and doesn't block if the app is already doing a shutdown. I can do some more testing, but I'd like to try adding a simple sleep and retry in https://searchfox.org/firefox-main/rev/60308bc3792ef201b82377682de068a5a1c72575/testing/mozbase/rust/mozdevice/src/lib.rs#604-605 and see if that helps.

We possibly might get away with just adding -S to here: https://searchfox.org/firefox-main/rev/60308bc3792ef201b82377682de068a5a1c72575/testing/mozbase/rust/mozdevice/src/lib.rs#593

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

sleeping 2 seconds, I get success:
https://treeherder.mozilla.org/jobs?repo=try&revision=6d1d220f744e0bb6ec4d705a92ba83cefce3f557

sleeping 1 second, I get only a few issues:
https://treeherder.mozilla.org/jobs?repo=try&revision=d396597f32ff25f19a912efad53a38ba6b57d6e0

adding the -S didn't fix enough:
https://treeherder.mozilla.org/jobs?repo=try&revision=429519514d824650fbb9f5ea1002a1d5dfe8184b

possibly there is something smarter to do, the theory is correct that we need more time- I wonder if it is possible to do this "conditionally" where if it fails, we wait 2 seconds and then try again?

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, I didn't know about the -S option and that you can force kill + wait until the application has quit. As it looks like with this option it works quite well. There is no more a resource issue visible.

Note that all the failures on the given try build are unrelated and shouldn't block us to get this fix landed to stabilize the handling of launching and connecting to the Android app. Joel, I'm happy to review a patch if you can put it up to Phabricator.

Maybe this is just bug 1791216 now for Android.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1981210 - [rust-mozdevice] Use "-S" argument for "am start" to force stop an already running application

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b37deaf74830
https://hg.mozilla.org/integration/autoland/rev/e3baea541672
[rust-mozdevice] Use "-S" argument for "am start" to force stop an already running application r=whimboo,webdriver-reviewers

Comment 19

[Ted Campbell [:tcampbell]]

Thank you both for do the actual verification and patch!

Comment 20

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/e3baea541672

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

for some reason this isn't being used in CI:
https://treeherder.mozilla.org/jobs?repo=try&author=jmaher%40mozilla.com&selectedTaskRun=GOVwYamKTKqAWs0yQCSP2A.0

I wonder if we need to adjust version numbers or release something?

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Joel, are you sure that we rebased your branch before pushing to try? When I check the latest mozilla-central job I can certainly see that -S is used when starting the application.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

thanks, all is good. there exists more intermittents:
https://treeherder.mozilla.org/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=4e2d188f202716455f9da4c1a4ce121bc817c757

like:

TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/integration/parallel_execution/browsing_context_print.py | test_when_browsingcontext_recreated - Failed: DID NOT RAISE <class 'webdriver.bidi.error.UnknownErrorException'>
TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/network/add_intercept/add_intercept.py | test_two_intercepts - webdriver.bidi.modules.script.ScriptEvaluateResultException: AbortError: The operation was aborted.

a few more random failures than normal; should we just turn these on and let intermittent tracking sort out what we run/skip/investigate with some time?

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #23)

thanks, all is good. there exists more intermittents:

That's good to hear! So lets close this bug.

a few more random failures than normal; should we just turn these on and let intermittent tracking sort out what we run/skip/investigate with some time?

Yes, leave them as is and we will handle the upcoming intermittent failure bugs. As it looks like none of them is perma failing.