1868944 - Perma all Android 7.0 test tasks fail with exceptions with "INTERMITTENT_TASK" after multiple retries

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Narcis Beleuzu [:NarcisB]]

Live_backing.log: https://firefoxci.taskcluster-artifacts.net/V40nJfjUQ_WYzYEJIlBQsg/5/public/logs/live_backing.log
TH: https://treeherder.mozilla.org/jobs?repo=mozilla-beta&selectedTaskRun=V40nJfjUQ_WYzYEJIlBQsg.5&fromchange=889c2b945041a3cdfba4f89a49812c637bb8cbe9&searchStr=android%2C7.0%2Cx86-64%2Cwebrender%2Cdebug-isolated-process%2Ctest-android-em-7.0-x86_64-qr%2Fdebug-isolated-process-geckoview-gtest-1proc%2Cgtest-1proc

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

autoland is closed for this.

Comment 2

[Geoff Brown [:gbrown]]

The emulator is failing to start.

Screenshot artifacts for the failed tasks show an emulator pop-up: title="Emulator Running in Nested Virtualization", body="The emulator is running using nested virtualization. This is not recommended. It may not work at all. And typically the performance is not quite good.", check-box="Never show this again." (unchecked).

Comment 3

[Geoff Brown [:gbrown]]

It's unclear if this is actually related to bug 1868680, but the recent failures seem coincidental.

Comment 4

[Andrew Erickson [:aerickson]]

Queue referenced: gecko-t/t-linux-kvm-gcp

No changes have been made to the image used by this queue recently.

Comment 5

[Geoff Brown [:gbrown]]

Recent (yesterday's) successful android-em test tasks have emulator logs that show:

Android emulator version 32.1.15.0 (build_id 10696886) (CL:N/A)

The failing tasks' emulator logs have:

Android emulator version 33.1.23.0 (build_id 11150993) (CL:N/A)

Comment 6

[[e-mail] (:bhearsum)]

Attachment: Bug 1868944: pin android emulator version in toolchain tasks r?gbrown!

This is not really the best fix - ultimately the pinned version should go in a task definition somewhere, I think, and we should even consider pinning in https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/android.py and pinning for even non-Taskcluster users (as we do for the NDK, cmdline tools, etc.)

But this is the safest, quickest fix for the immediate term.

Comment 7

[Geoff Brown [:gbrown]]

The warning in comment 2 has been in the emulator for a long time...I don't know why it is popping up for us now in this version.

The emulator even has an override for that warning: --no-nested-warnings. Trying to determine if that helps...

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f5601bcf5ef
pin android emulator version in toolchain tasks r=gbrown! CLOSED TREE

Comment 9

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/44539b631fb1

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/129e77822016

Comment 11

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/6f5601bcf5ef

Comment 12

[Norisz Fay [:noriszfay]]

The issue is still lingering around on Android 7 TV-nofis as seen here

Comment 13

[Intermittent Failures Robot]

275 failures in 639 pushes (0.43 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 84
• mozilla-central: 187
• autoland: 4

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 184
  • debug-isolated-process: 21
  • debug: 162
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 67
  • opt: 67
• android-em-7-0-x86_64-shippable-qr: 23
  • opt: 23
• android-em-7-0-x86_64-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868944&startday=2023-12-08&endday=2023-12-08&tree=all

Comment 14

[Intermittent Failures Robot]

20 failures in 161 pushes (0.124 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 15
  • opt: 5
  • debug-isolated-process: 5
  • debug: 5
• android-em-7-0-x86_64-lite-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868944&startday=2023-12-09&endday=2023-12-09&tree=all

Comment 15

[Geoff Brown [:gbrown]]

(In reply to Norisz Fay [:noriszfay] from comment #12)

The issue is still lingering around on Android 7 TV-nofis as seen here

Most android-em tests use the android-emulator-linux artifact - https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/taskcluster/ci/test/mochitest.yml#188 - but test-verify uses android-sdk-linux - https://searchfox.org/mozilla-central/rev/c003be8b9727672e7d30972983b375f4c200233f/taskcluster/ci/test/misc.yml#124. I doubt there's a good reason for that: test-verify should use android-emulator-linux as well. I'll see if that works...

Comment 16

[Intermittent Failures Robot]

303 failures in 3671 pushes (0.083 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 84
• mozilla-central: 187
• autoland: 32

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 205
  • debug-isolated-process: 28
  • debug: 169
  • opt: 8
• android-em-7-0-x86_64-shippable-lite-qr: 67
  • opt: 67
• android-em-7-0-x86_64-shippable-qr: 23
  • opt: 23
• android-em-7-0-x86_64-lite-qr: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868944&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 17

[Geoff Brown [:gbrown]]

(In reply to Norisz Fay [:noriszfay] from comment #12)

The issue is still lingering around on Android 7 TV-nofis as seen

This should be fixed by bug 1869220.

Comment 18

[Intermittent Failures Robot]

33 failures in 813 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 30
• mozilla-release: 3

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 7
  • opt: 7
• android-em-7-0-x86_64-qr: 23
  • debug: 7
  • debug-isolated-process: 7
  • opt: 9
• android-em-7-0-x86: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868944&startday=2023-12-11&endday=2023-12-11&tree=all

Comment 19

[Intermittent Failures Robot]

33 failures in 4224 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #32 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 30
• mozilla-release: 3

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 7
  • opt: 7
• android-em-7-0-x86_64-qr: 23
  • debug: 7
  • debug-isolated-process: 7
  • opt: 9
• android-em-7-0-x86: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868944&startday=2023-12-11&endday=2023-12-17&tree=all