1866612 - Intermittent linux asan jsreftest jobs fail as exceptions with "claim_expired" after multiple retries

Status: NEW

(Taskcluster :: Workers, defect)

Description

[Cosmin Sabou [:CosminS]]

First noticed this here: https://treeherder.mozilla.org/jobs?repo=autoland&revision=7dc6122ebd68b46a9d4320c40c86f20c2b78984a&selectedTaskRun=ZkyCwM6BR-eR9O9vxqOEfg.5&searchStr=Linux%2C18.04%2Cx64%2CWebRender%2Casan%2Copt%2CReftests%2Ctest-linux1804-64-asan-qr%2Fopt-jsreftest%2CJ1 and probably coming from changes in Bug 1847258.
The jobs keep on retrying until they end up as an exception with claim_expired.
There was a similar bug for browser-chrome tests in 1859204 that was fixed by increasing RAM for the machines in https://hg.mozilla.org/integration/autoland/rev/fb7b6fc608af4116fcc99a1003dfafc5bb78818a.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1847258

Comment 2

[Cosmin Sabou [:CosminS]]

Could be a combination of Bug 1865910 and then bug 1847258. https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&searchStr=Linux%2C18.04%2Cx64%2CWebRender%2Casan%2Copt%2CReftests%2Ctest-linux1804-64-asan-qr%2Fopt-jsreftest%2CJ&tochange=7dc6122ebd68b46a9d4320c40c86f20c2b78984a&fromchange=ee683c3699a76e2e7159a5297800dc28b78ac22a&selectedTaskRun=Wrbq080wTLyKFnTwDsm13w.5

Comment 3

[Markus Stange [:mstange]]

Bug 1865910 only affects code inside #ifdef JS_ION_PERF which is off in all CI builds.

Comment 4

[Cosmin Sabou [:CosminS]]

Added some retriggers in this range in case it could be from changes in Bug 1852098.

Comment 5

[Intermittent Failures Robot]

16 failures in 3623 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 7

Platform and build breakdown:

• linux1804-64-asan-qr: 15
  • opt: 15
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1866612&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 6

[Pete Moore [:pmoore][:pete]]

On initial inspection, this appears to be either:

1. a Docker Worker issue, or
2. related to https://github.com/taskcluster/taskcluster/issues/6682

If it is 1) this should be resolved when Docker Worker workers have been replaced with Generic Worker workers in fxci (Docker Worker is no longer supported)
@aerickson - do you have a bug/issue tracking that work?

If this is 2), the SRE team are investigating excessive HTTP 502 errors which seem to be the root cause of many of the claim expired issues we have been seeing.
@wezhou - do you have a bug/issue tracking that work?

Comment 7

[Andrew Erickson [:aerickson]]

(In reply to Pete Moore [:pmoore][:pete] from comment #6)

If it is 1) this should be resolved when Docker Worker workers have been replaced with Generic Worker workers in fxci (Docker Worker is no longer supported)
@aerickson - do you have a bug/issue tracking that work?

We're tracking that in https://mozilla-hub.atlassian.net/browse/RELOPS-528.

Comment 8

[:wezhou]

(In reply to Pete Moore [:pmoore][:pete] from comment #6)

If this is 2), the SRE team are investigating excessive HTTP 502 errors which seem to be the root cause of many of the claim expired issues we have been seeing.
@wezhou - do you have a bug/issue tracking that work?

Here is the ticket, https://mozilla-hub.atlassian.net/browse/SVCSE-1609

Comment 9

[Intermittent Failures Robot]

15 failures in 700 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 7
  • opt: 7
• windows11-64-2009-asan-qr: 7
  • opt: 7
• linux2204-64-wayland: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1866612&startday=2023-11-27&endday=2023-11-27&tree=all

Comment 10

[Intermittent Failures Robot]

37 failures in 4108 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 28
• mozilla-central: 6
• mozilla-release: 2
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 29
  • opt: 29
• linux2204-64-wayland: 1
  • opt: 1
• windows11-64-2009-asan-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1866612&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 11

[Intermittent Failures Robot]

13 failures in 3671 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 2
• mozilla-esr115: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 13
  • opt: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1866612&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 12

[Cosmin Sabou [:CosminS]]

Stopped failing around Dec 6th: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-11-22&endday=2023-12-22&tree=all&failurehash=all&bug=1866612