1868680 - Perma toolchain avd-arm-linux [taskcluster:error] Task timeout after 1800 seconds after Force killing container. adb: no devices/emulators found

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Treeherder Bug Filer]

Filed by: nerli [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=439114392&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/M8oE5_JWQpatQcKr4XFAAQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-12-06T22:00:18.524Z] WARNING | *** Basic token auth should only be used by android-studio ***
[task 2023-12-06T22:00:18.525Z] INFO    | The active JSON Web Key Sets can be found here: /builds/worker/.android/avd/running/354/jwks/f5d7a383-c93e-45e5-8f76-8cc1626bc1c6/active.jwk
[task 2023-12-06T22:00:18.525Z] INFO    | Scanning /builds/worker/.android/avd/running/354/jwks/f5d7a383-c93e-45e5-8f76-8cc1626bc1c6 for jwk keys.
[task 2023-12-06T22:00:18.527Z] INFO    | Started GRPC server at 127.0.0.1:8554, security: Local, auth: +token
[task 2023-12-06T22:00:18.530Z] WARNING | Using fallback path for the emulator registration directory.
[task 2023-12-06T22:00:18.530Z] INFO    | Advertising in: /builds/worker/.android/avd/running/pid_354.ini
[task 2023-12-06T22:00:18.539Z] [364:364:20231206,220018.539223:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
[task 2023-12-06T22:00:18.539Z] [364:364:20231206,220018.539276:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)
[task 2023-12-06T22:00:46.199Z] adb: no devices/emulators found
[task 2023-12-06T22:01:16.202Z] adb: no devices/emulators found
[task 2023-12-06T22:01:46.205Z] adb: no devices/emulators found
[task 2023-12-06T22:02:16.208Z] adb: no devices/emulators found
[task 2023-12-06T22:02:46.211Z] adb: no devices/emulators found
[task 2023-12-06T22:03:16.214Z] adb: no devices/emulators found
[task 2023-12-06T22:03:46.217Z] adb: no devices/emulators found
[task 2023-12-06T22:04:16.220Z] adb: no devices/emulators found
[task 2023-12-06T22:04:46.223Z] adb: no devices/emulators found
[task 2023-12-06T22:05:16.226Z] adb: no devices/emulators found
[task 2023-12-06T22:05:46.229Z] adb: no devices/emulators found
[task 2023-12-06T22:06:16.232Z] adb: no devices/emulators found
[task 2023-12-06T22:06:46.235Z] adb: no devices/emulators found
[task 2023-12-06T22:07:16.238Z] adb: no devices/emulators found
[task 2023-12-06T22:07:46.240Z] adb: no devices/emulators found
[task 2023-12-06T22:08:16.243Z] adb: no devices/emulators found
[task 2023-12-06T22:08:46.246Z] adb: no devices/emulators found
[task 2023-12-06T22:09:16.249Z] adb: no devices/emulators found
[task 2023-12-06T22:09:46.252Z] adb: no devices/emulators found
[task 2023-12-06T22:10:16.255Z] adb: no devices/emulators found
[task 2023-12-06T22:10:46.258Z] adb: no devices/emulators found
[task 2023-12-06T22:11:16.260Z] adb: no devices/emulators found
[task 2023-12-06T22:11:46.263Z] adb: no devices/emulators found
[task 2023-12-06T22:12:16.266Z] adb: no devices/emulators found
[task 2023-12-06T22:12:46.269Z] adb: no devices/emulators found
[task 2023-12-06T22:13:16.272Z] adb: no devices/emulators found
[task 2023-12-06T22:13:46.275Z] adb: no devices/emulators found
[task 2023-12-06T22:14:16.278Z] adb: no devices/emulators found
[task 2023-12-06T22:14:46.281Z] adb: no devices/emulators found
[task 2023-12-06T22:15:16.284Z] adb: no devices/emulators found
[task 2023-12-06T22:15:46.287Z] adb: no devices/emulators found
[task 2023-12-06T22:16:16.290Z] adb: no devices/emulators found
[task 2023-12-06T22:16:46.292Z] adb: no devices/emulators found
[task 2023-12-06T22:17:16.295Z] adb: no devices/emulators found
[task 2023-12-06T22:17:46.299Z] adb: no devices/emulators found
[task 2023-12-06T22:18:16.301Z] adb: no devices/emulators found
[task 2023-12-06T22:18:46.304Z] adb: no devices/emulators found
[task 2023-12-06T22:19:16.307Z] adb: no devices/emulators found
[task 2023-12-06T22:19:46.309Z] adb: no devices/emulators found
[task 2023-12-06T22:20:16.312Z] adb: no devices/emulators found
[task 2023-12-06T22:20:46.315Z] adb: no devices/emulators found
[task 2023-12-06T22:21:16.318Z] adb: no devices/emulators found
[task 2023-12-06T22:21:46.321Z] adb: no devices/emulators found
[task 2023-12-06T22:22:16.324Z] adb: no devices/emulators found
[task 2023-12-06T22:22:46.327Z] adb: no devices/emulators found
[task 2023-12-06T22:23:16.330Z] adb: no devices/emulators found
[task 2023-12-06T22:23:46.333Z] adb: no devices/emulators found
[task 2023-12-06T22:24:16.335Z] adb: no devices/emulators found
[task 2023-12-06T22:24:46.338Z] adb: no devices/emulators found
[task 2023-12-06T22:25:16.341Z] adb: no devices/emulators found
[task 2023-12-06T22:25:46.344Z] adb: no devices/emulators found
[task 2023-12-06T22:26:16.347Z] adb: no devices/emulators found
[task 2023-12-06T22:26:46.349Z] adb: no devices/emulators found
[task 2023-12-06T22:27:16.352Z] adb: no devices/emulators found
[task 2023-12-06T22:27:46.356Z] adb: no devices/emulators found
[task 2023-12-06T22:28:16.358Z] adb: no devices/emulators found
[task 2023-12-06T22:28:46.361Z] adb: no devices/emulators found

[taskcluster:error] Task timeout after 1800 seconds. Force killing container.
[taskcluster 2023-12-06 22:28:56.114Z] === Task Finished ===
[taskcluster 2023-12-06 22:28:56.114Z] Unsuccessful task run with exit code: -1 completed in 1815.895 seconds

Comment 1

[Mike Hommey [:glandium]]

Last time the task worked, there was this in the log, which doesn't appear anymore:

[task 2023-11-24T10:33:45.074Z] INFO    | Your emulator is out of date, please update by launching Android Studio:
[task 2023-11-24T10:33:45.074Z]  - Start Android Studio
[task 2023-11-24T10:33:45.074Z]  - Select menu "Tools > Android > SDK Manager"
[task 2023-11-24T10:33:45.074Z]  - Click "SDK Tools" tab
[task 2023-11-24T10:33:45.074Z]  - Check "Android Emulator" checkbox
[task 2023-11-24T10:33:45.074Z]  - Click "OK"

Comment 2

[Mike Hommey [:glandium]]

(obviously, this implies the emulator was updated)

Comment 3

[Mike Hommey [:glandium]]

Interestingly, the other AVD task that uses the emulator and doesn't fail still has the out-of-date message.

Comment 4

[Mike Hommey [:glandium]]

For the record:
broken version: Android qemu version 33.1.23.0 (build_id 11150993) (CL:N/A)
working version: Android qemu version 32.1.15.0 (build_id 10696886) (CL:N/A)

Comment 5

[Johan Lorenzo [:jlorenzo] - On PTO - Back on July 28th]

While the qemu version is not explicitly set in tree, I see the same emulator version is used both in esr115[1] and central[2]. Marking all current trains as affected.

[1] https://searchfox.org/mozilla-esr115/rev/aae82b51e34318dc7353389708654dfbd68ca41d/python/mozboot/mozboot/android-avds/arm.json#2
[2] https://searchfox.org/mozilla-central/rev/1a63fee6c2a8ea6846f6f16b85edeb6d19fc9f23/python/mozboot/mozboot/android-avds/arm.json#2

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1868680 - Stop prewarming the ARM AVD.

prewarming the AVDs was done for CI, where AVDs are used only once in
each task using them, and the cost of first use is high. Prewarming
reduces that overhead.

However, the ARM AVD is not used on CI (anymore?), so we don't need
them prewarmed. This unblocks the situation wrt the failure to run
the emulator on CI, while not addressing the underlying issues, so that
CoT keys can be rotated without waiting for the full situation to be
resolved.

Comment 7

[Intermittent Failures Robot]

8 failures in 3671 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 4
• mozilla-central: 2
• mozilla-release: 1
• mozilla-beta: 1

Platform and build breakdown:

• toolchains: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868680&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3ae67b2bcfa9
Stop prewarming the ARM AVD. r=geckoview-reviewers,owlish

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/3ae67b2bcfa9

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:glandium, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox121 to wontfix.

For more information, please visit BugBot documentation.

Comment 11

[Mike Hommey [:glandium]]

Johan, I suppose we want this on all branches? Or, as nothing uses that task there, maybe we don't care?

Comment 12

[Ryan VanderMeulen [:RyanVM]]

We don't need to uplift this. We already landed a workaround on the impacted branches.

Comment 13

[Intermittent Failures Robot]

1 failures in 4224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• toolchains: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868680&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 14

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/47845fd30982

Comment 15

[Intermittent Failures Robot]

1 failures in 3349 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• toolchains: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868680&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 16

[Intermittent Failures Robot]

2 failures in 1162 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2

Platform and build breakdown:

• toolchains: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868680&startday=2023-12-25&endday=2023-12-31&tree=all

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/f7e37ab080b0

Comment 18

[Intermittent Failures Robot]

3 failures in 2274 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 3

Platform and build breakdown:

• toolchains: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868680&startday=2024-01-01&endday=2024-01-07&tree=all