Closed Bug 1868944 Opened 3 months ago Closed 3 months ago

Perma all Android 7.0 test tasks fail with exceptions with "INTERMITTENT_TASK" after multiple retries

Categories

(Firefox Build System :: Task Configuration, defect)

defect

Tracking

(firefox120 fixed, firefox121 fixed, firefox122 fixed)

RESOLVED FIXED
122 Branch
Tracking Status
firefox120 --- fixed
firefox121 --- fixed
firefox122 --- fixed

People

(Reporter: NarcisB, Assigned: bhearsum)

References

Details

(Whiteboard: [stockwell disable-recommended])

Attachments

(1 file)

Summary: Perma Android exceptions with "INTERMITTENT_TASK" after multiple retries → Perma all Android 7.0 test tasks fail with exceptions with "INTERMITTENT_TASK" after multiple retries

The emulator is failing to start.

Screenshot artifacts for the failed tasks show an emulator pop-up: title="Emulator Running in Nested Virtualization", body="The emulator is running using nested virtualization. This is not recommended. It may not work at all. And typically the performance is not quite good.", check-box="Never show this again." (unchecked).

It's unclear if this is actually related to bug 1868680, but the recent failures seem coincidental.

See Also: → 1868680

Queue referenced: gecko-t/t-linux-kvm-gcp

No changes have been made to the image used by this queue recently.

Flags: needinfo?(aerickson)

Recent (yesterday's) successful android-em test tasks have emulator logs that show:

Android emulator version 32.1.15.0 (build_id 10696886) (CL:N/A)

The failing tasks' emulator logs have:

Android emulator version 33.1.23.0 (build_id 11150993) (CL:N/A)

This is not really the best fix - ultimately the pinned version should go in a task definition somewhere, I think, and we should even consider pinning in https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/android.py and pinning for even non-Taskcluster users (as we do for the NDK, cmdline tools, etc.)

But this is the safest, quickest fix for the immediate term.

Assignee: nobody → bhearsum
Status: NEW → ASSIGNED

The warning in comment 2 has been in the emulator for a long time...I don't know why it is popping up for us now in this version.

The emulator even has an override for that warning: --no-nested-warnings. Trying to determine if that helps...

Pushed by nfay@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f5601bcf5ef
pin android emulator version in toolchain tasks r=gbrown! CLOSED TREE
See Also: → 1869038
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Blocks: 1869069

The issue is still lingering around on Android 7 TV-nofis as seen here

Flags: needinfo?(bhearsum)
Component: General → Task Configuration
Product: Taskcluster → Firefox Build System
Target Milestone: --- → 122 Branch

(In reply to Norisz Fay [:noriszfay] from comment #12)

The issue is still lingering around on Android 7 TV-nofis as seen here

Most android-em tests use the android-emulator-linux artifact - https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/taskcluster/ci/test/mochitest.yml#188 - but test-verify uses android-sdk-linux - https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/taskcluster/ci/test/misc.yml#124. I doubt there's a good reason for that: test-verify should use android-emulator-linux as well. I'll see if that works...

Flags: needinfo?(bhearsum) → needinfo?(gbrown)
Blocks: 1869220

(In reply to Norisz Fay [:noriszfay] from comment #12)

The issue is still lingering around on Android 7 TV-nofis as seen

This should be fixed by bug 1869220.

Flags: needinfo?(gbrown)
Blocks: 1869407
You need to log in before you can comment on or make changes to this bug.