Closed Bug 1776799 Opened 3 years ago Closed 3 years ago

Intermittent wrench INFRA-ERROR: Unable to start emulator after 5 attempts

Categories

(Testing :: General, defect, P5)

Product:
Testing
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=382689179&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/J2Uc2tygQLSwGlcTGTucdw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/J2Uc2tygQLSwGlcTGTucdw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2022-06-27T22:35:16.800Z] 22:35:16     INFO - >> Verify Android boot completed: Attempt #5 of 30
[task 2022-06-27T22:35:16.806Z] adb Using adb 1.0.41
[task 2022-06-27T22:36:16.922Z] 22:36:16     INFO - Maximum retry run-time of 330 seconds exceeded; remaining attempts abandoned
[task 2022-06-27T22:36:16.922Z] 22:36:16  WARNING - Unable to verify Android boot completion
[task 2022-06-27T22:36:16.922Z] 22:36:16     INFO - Not saving screenshot: no XRE configured
[task 2022-06-27T22:36:16.923Z] 22:36:16     INFO - Killing every process called qemu-system-x86_64
[task 2022-06-27T22:36:16.939Z] UID        PID  PPID  C STIME TTY          TIME CMD
[task 2022-06-27T22:36:16.939Z] worker       1     0  0 22:03 pts/0    00:00:00 /usr/bin/python3 -u /builds/worker/bin/run-task --gecko-checkout=/builds/worker/checkouts/gecko --gecko-sparse-profile=build/sparse-profiles/webrender --fetch-hgfingerprint -- bash -cx export WRENCH_APK=$MOZ_FETCHES_DIR/wrench-release.apk && export WRENCH_REFTESTS=$GECKO_PATH/gfx/wr/wrench/reftests && export MOZHARNESS_PATH=$GECKO_PATH/testing/mozharness && $GECKO_PATH/taskcluster/scripts/tester/test-linux.sh
[task 2022-06-27T22:36:16.939Z] message+    24     1  0 22:03 ?        00:00:00 /usr/bin/dbus-daemon --system
[task 2022-06-27T22:36:16.939Z] worker      73     1  0 22:06 pts/0    00:00:00 /bin/bash -xe /builds/worker/checkouts/gecko/taskcluster/scripts/tester/test-linux.sh
[task 2022-06-27T22:36:16.939Z] worker      82     1  0 22:06 ?        00:00:00 SCREEN -dmS xvfb Xvfb :0 -nolisten tcp -screen 0 1600x1200x24
<...>
[task 2022-06-27T22:36:16.943Z] worker    1362     1  0 22:30 pts/0    00:00:00 /builds/worker/fetches/android-sdk-linux/emulator/emulator64-crash-service -pipe 5 -ppid 1359 -data-dir /tmp/android-worker/04d1683b-3467-4c72-bbc7-fb01d0106df8
[task 2022-06-27T22:36:16.943Z] worker    1417   245  0 22:36 pts/0    00:00:00 ps -ef
[task 2022-06-27T22:36:16.944Z] 22:36:16     INFO - rmtree: /tmp/android-worker
[task 2022-06-27T22:36:16.945Z] 22:36:16     INFO - retry: Calling rmtree with args: ('/tmp/android-worker',), kwargs: {}, attempt #1
[task 2022-06-27T22:36:21.948Z] 22:36:21     INFO - wrote dummy /builds/worker/fetches/android-device/ddms.cfg
[task 2022-06-27T22:36:21.948Z] 22:36:21     INFO - Found avds at /builds/worker/fetches/android-device/avd
[task 2022-06-27T22:36:21.948Z] 22:36:21     INFO - Found sdk at /builds/worker/fetches/android-sdk-linux
[task 2022-06-27T22:36:21.949Z] 22:36:21     INFO - Old Config: ['avd.ini.encoding=UTF-8\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', 'path=/builds/worker/fetches/android-device/avd/mozemulator-x86_64.avd\n', 'target=android-24\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n', '\n']
[task 2022-06-27T22:36:21.949Z] 22:36:21     INFO - Updating path from: path=/builds/worker/fetches/android-device/avd/mozemulator-x86_64.avd
[task 2022-06-27T22:36:21.949Z] 22:36:21     INFO - Running command: ['ls', '-l', '/dev/kvm']
[task 2022-06-27T22:36:21.949Z] 22:36:21     INFO - Copy/paste: ls -l /dev/kvm
[task 2022-06-27T22:36:21.953Z] 22:36:21     INFO -  crw-rw-rw- 1 root root 10, 232 Jun 27 22:03 /dev/kvm
[task 2022-06-27T22:36:21.954Z] 22:36:21     INFO - Return code: 0
[task 2022-06-27T22:36:21.954Z] 22:36:21     INFO - Running command: ['kvm-ok']
[task 2022-06-27T22:36:21.954Z] 22:36:21     INFO - Copy/paste: kvm-ok
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO -  INFO: /dev/kvm exists
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO -  KVM acceleration can be used
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO - Return code: 0
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO - Running command: ['emulator', '-accel-check']
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO - Copy/paste: emulator -accel-check
[task 2022-06-27T22:36:21.981Z] 22:36:21     INFO - Using env: (same as previous command)
[task 2022-06-27T22:36:21.998Z] 22:36:21     INFO -  accel:
[task 2022-06-27T22:36:21.998Z] 22:36:21     INFO -  0
[task 2022-06-27T22:36:21.999Z] 22:36:21     INFO -  KVM (version 12) is installed and usable.
[task 2022-06-27T22:36:21.999Z] 22:36:21     INFO -  accel
[task 2022-06-27T22:36:21.999Z] 22:36:21     INFO - Return code: 0
[task 2022-06-27T22:36:22.000Z] 22:36:22     INFO - emulator env: {'MOZ_FETCHES': '[{"artifact": "project/gecko/android-sdk/android-sdk-linux.tar.zst", "extract": true, "task": "QS_GMI_CTUGQnpmLbEuySw"}, {"artifact": "project/gecko/android-system-images/android-system-images-linux.tar.zst", "extract": true, "task": "NSf53a1RSOuuIjVeFn8nHw"}, {"artifact": "project/gecko/jdk/jdk-linux.tar.zst", "extract": true, "task": "KMgbILWsTRi0M3_CRHTSgA"}, {"artifact": "public/build/android-avd-linux.tar.zst", "extract": true, "task": "GFCPAxO6QCyNrabKGPJhPw"}, {"artifact": "public/build/wrench-release.apk", "extract": true, "task": "AOkSpj5fSOe0J4WoF5BH7Q"}]', 'LC_ALL': 'en_US.UTF-8', 'TOOLTOOL_CACHE': '/builds/worker/tooltool-cache', 'MOZ_AUTOMATION': '1', 'LANG': 'en_US.UTF-8', 'GECKO_PATH': '/builds/worker/checkouts/gecko', 'DISPLAY': ':0.0', 'HOSTNAME': 'taskcluster-worker', 'TASKCLUSTER_NAME': '/tender_diffie/taskcluster', 'OLDPWD': '/builds/worker/workspace', 'RUST_BACKTRACE': 'full', 'TASKCLUSTER_PUBLIC_IP': '44.204.76.173', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/mozilla-central', 'JAVA_HOME': '/builds/worker/fetches/jdk/jdk-17.0.1+12', 'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.10', 'TASKCLUSTER_WORKER_GROUP': 'us-east-1', 'RUN_ID': '0', 'USER': 'worker', 'TASKCLUSTER_PROXY_URL': 'http://taskcluster', 'NEED_WINDOW_MANAGER': 'true', 'WRENCH_REFTESTS': '/builds/worker/checkouts/gecko/gfx/wr/wrench/reftests', 'TASKCLUSTER_INSTANCE_TYPE': 'r5.metal', 'PWD': '/builds/worker/workspace', 'HOME': '/builds/worker', 'TASKCLUSTER_WORKER_TYPE': 't-linux-metal', 'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified', 'MOZ_SOURCE_CHANGESET': '5563ec0537500a2ac4a38aa9271452b851a0c3ab', 'TASKCLUSTER_WORKER_LOCATION': '{"availabilityZone":"us-east-1c","cloud":"aws","region":"us-east-1"}', 'DEBIAN_FRONTEND': 'noninteractive', 'GECKO_HEAD_REV': '5563ec0537500a2ac4a38aa9271452b851a0c3ab', 'MOZHARNESS_PATH': '/builds/worker/checkouts/gecko/testing/mozharness', 'MOZHARNESS_CONFIG': 'android/wrench.py', 'MOZ_SCM_LEVEL': '3', 'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp', 'TASKCLUSTER_PORT': 'tcp://172.17.0.10:80', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com', 'TASK_ID': 'J2Uc2tygQLSwGlcTGTucdw', 'MOZ_FETCHES_DIR': '/builds/worker/fetches', 'WRENCH_APK': '/builds/worker/fetches/wrench-release.apk', 'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/mozilla-central', 'SHLVL': '2', 'LOGNAME': 'worker', 'DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-NrqDT4b95Q,guid=2a49f8293430a39e64282ae162ba29f6', 'TASKCLUSTER_PORT_80_TCP_PORT': '80', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builds/worker/bin:/builds/worker/fetches/android-sdk-linux/emulator:/builds/worker/fetches/android-sdk-linux/tools:/builds/worker/fetches/android-sdk-linux/platform-tools', 'HG_STORE_PATH': '/builds/worker/checkouts/hg-store', 'SCCACHE_DISABLE': '1', 'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.10:80', 'RUSTFLAGS': '--deny warnings', 'MOZHARNESS_SCRIPT': 'android_wrench.py', 'MACH_MAIN_PID': '210', 'PYTHONDONTWRITEBYTECODE': '1', 'ANDROID_EMULATOR_HOME': '/builds/worker/fetches/android-device', 'ANDROID_AVD_HOME': '/builds/worker/fetches/android-device/avd', 'ANDROID_SDK_HOME': '/builds/worker/fetches/android-sdk-linux', 'ANDROID_SDK_ROOT': '/builds/worker/fetches/android-sdk-linux'}
[task 2022-06-27T22:36:22.000Z] 22:36:22     INFO - Launching the emulator with: emulator -avd mozemulator-x86_64 -gpu on -skip-adb-auth -verbose -show-kernel -ranchu -selinux permissive -memory 3072 -cores 4 -skin 800x1280 -no-snapstorage -no-snapshot -prop ro.test_harness=true
[task 2022-06-27T22:36:22.000Z] 22:36:22     INFO - Writing log to /builds/worker/workspace/build/logs/emulator-xgivrcsm.log
[task 2022-06-27T22:36:22.003Z] 22:36:22    FATAL - INFRA-ERROR: Unable to start emulator after 5 attempts
[task 2022-06-27T22:36:22.003Z] 22:36:22    FATAL - Running post_fatal callback...
[task 2022-06-27T22:36:22.003Z] 22:36:22    FATAL - Exiting 4
Product: Firefox Build System → Testing

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.