1621323 - Linux x64 tsan mochitest 20 M(20) frequently retried and failing,

Status: RESOLVED WORKSFORME

(Core :: Sanitizers, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Since yesterday UTC evening, the Linux x64 tsan M(20) frequently gets retried, up to the limit of 5 retries (and then fails with an exception):

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Cpending%2Crunning%2Cretry%2Cusercancel%2Ctestfailed%2Cbusted%2Cexception&fromchange=dc30c7d0df8392adf5c90262ce563ff9f6bbe180&searchStr=07846be0250651c0251e1b10f61192ba4dbbd754&tochange=28fa31759b1000b73f9cf8b054e6c53a7a927615&selectedJob=292427013

Many of the tasks which complete fail with bug 1619072.

The log for the treeherder link above mentions

SUMMARY: ThreadSanitizer: out-of-memory (/builds/worker/workspace/build/tests/bin/xpcshell+0x55864)

Backfill is inconclusive, failures start with a change which should only affect mochitest browser-chrome and not mochitest-plain: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=292471838&resultStatus=success%2Cpending%2Crunning%2Cretry%2Cusercancel%2Ctestfailed%2Cbusted%2Cexception&fromchange=00a5c2ac8f8466c4b5c3a5e5eeb95f815768cca7&searchStr=linux%2Ctsan%2Cmochitest&tochange=1692426a9549efd97f3241b8fc314eac0786a750

Christian, can you take a look or redirect if necessary, please?

Comment 1

[Christian Holler (:decoder)]

Attachment: Bug 1621323 - Disable an OOMing test for TSan and fix TSan OOM options. r?froydnj

Comment 2

[Christian Holler (:decoder)]

This should disable the test that OOMs (in some runs, it also simply times out), but it also fixes TSan's options about OOMing to behave like ASan does (return NULL when an allocation fails due to OOM, rather than aborting).

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a7a6063bd5a4
Disable an OOMing test for TSan and fix TSan OOM options. r=froydnj

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Tasks still getting frequently retried but if they complete then mostly without errors now.

Comment 5

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/a7a6063bd5a4

Comment 6

[Intermittent Failures Robot]

57 failures in 1083 pushes (0.053 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 53

Platform breakdown:

• linux64-tsan: 57

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1621323&startday=2020-03-10&endday=2020-03-10&tree=all

Comment 7

[Andreea Pavel [:apavel]]

Exceptions are still occurring: https://firefox-ci-tc.services.mozilla.com/tasks/NnkD44SbQ46bo53WF3YDPg

log: https://firefox-ci-tc.services.mozilla.com/tasks/NnkD44SbQ46bo53WF3YDPg/runs/5/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FNnkD44SbQ46bo53WF3YDPg%2Fruns%2F5%2Fartifacts%2Fpublic%2Flogs%2Flive.log

Comment 8

[Alexandru Michis [:malexandru]]

Also happening on (1) jobs:

• https://firefox-ci-tc.services.mozilla.com/tasks/Qky6F7xcQiuhXuN4Ra3MGA
• https://firefox-ci-tc.services.mozilla.com/tasks/XrxmAoUpT96kGK94-YYoVg

Comment 9

[Intermittent Failures Robot]

77 failures in 5336 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 6
• autoland: 71

Platform breakdown:

• linux64-tsan: 77

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1621323&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:decoder, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 11

[Christian Holler (:decoder)]

:aryx, this is gone, right?

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yes. M(19) and M(20) sometimes hit 1-2 retries but in general it's gone.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:decoder, maybe it's time to close this bug?