1712866 - Perma Linux 18.04 x64 tsan opt crashtest exceptions claim_expired

Status: RESOLVED WORKSFORME

(Core :: Sanitizers, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=cskOja4STlqXcOfNbHj8_w.5&searchStr=linux%2C18.04%2Cx64%2Ctsan%2Copt%2Creftests%2Ctest-linux1804-64-tsan%2Fopt-crashtest-e10s%2Cc1&revision=ac170743db6e3739747938e47c51f57227662c72

Task: https://firefox-ci-tc.services.mozilla.com/tasks/cskOja4STlqXcOfNbHj8_w/runs/0/

Range: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&fromchange=0faf5e8f142b929a2d775eefdfe39250843a5dd5&searchStr=linux%2C18.04%2Cx64%2Ctsan%2Copt%2Creftests%2Ctest-linux1804-64-tsan%2Fopt-crashtest-e10s%2Cc1&tochange=299b3a1c144d17f8cd3f2bf27b801eca577df28a

Could this be related to Bug 1692068 in any way?

Comment 1

[Kris Wright :KrisWright]

Crashtests were having issues with some suites where it triggered some sort of mysterious infinite-retry on running out of resources. I couldn't work out why it was happening, but I did find and turn off the series of tests that seemed to be causing it. After that I didn't run into any of these, but looks like it's still happening in the wild. I'll examine more of the tests tonight to see if any of them are responsible.

If there are more tests left over in the suite doing this at high enough volume, we may need to rollback until we can find the incriminating tests.

Comment 2

[Kris Wright :KrisWright]

I notice there's no logs for any of these. Is there any way to get logs? Or are they discarded for retry?

Comment 3

[Natalia Csoregi [:nataliaCs]]

None of the crashtest exceptions have any logs, or at least they are not visible on our side.
Aryx, do you know what we can do to access them?
Thank you.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The machines terminates before the logs can be uploaded. Requesting an interactive worker, logging in an running the tests might be the only way to catch this.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1712198 which has the first failure added a crashtest. Emilio, can this be fixed soon?

Comment 6

[Emilio Cobos Álvarez (:emilio)]

Yeah I'll try to repro. Worst case we can disable the crastest on tsan or something.

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1712866 - Skip 1712198.html under TSAN. r=aryx,#intermittent-reviewers

Not ideal, but not worse than having the jobs not terminate. I can try
to spend some time digging, but I have other stuff on my plate so this
should at least go back to the previous state.

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Let's disable for now on tsan, I don't have a lot of cycles this month.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/52ddb53e7363
Skip 1712198.html under TSAN. r=aryx

Comment 10

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/52ddb53e7363

Comment 11

[Intermittent Failures Robot]

19 failures in 2908 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-tsan: 19
  • opt: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 12

[Intermittent Failures Robot]

18 failures in 2888 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• mozilla-central: 3
• try: 3

Platform and build breakdown:

• linux1804-64-tsan: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-06-14&endday=2021-06-20&tree=all

Comment 13

[Intermittent Failures Robot]

3 failures in 3115 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64-tsan: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-06-21&endday=2021-06-27&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 2387 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 15

[Intermittent Failures Robot]

4 failures in 3025 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 3
  • opt: 3
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-08-09&endday=2021-08-15&tree=all

Comment 16

[Intermittent Failures Robot]

4 failures in 2899 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• linux1804-64-asan-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-08-16&endday=2021-08-22&tree=all

Comment 17

[Intermittent Failures Robot]

2 failures in 3266 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 18

[Cristian Tuns]

This started to fail as an exception on Linux WebRender asan opt.
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cpending%2Crunning%2Cretry&fromchange=55fa1d670b119502b16f8795990f9456e1bfcd8c&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Casan%2Copt%2Cmochitests%2Cwith%2Csoftware%2Cwebrender%2Cand%2Cfission%2Cenabled%2Ctest-linux1804-64-asan-qr%2Fopt-mochitest-browser-chrome-swr-fis-e10s%2Cbc13&tochange=728ead24c5e80f438419b406678c567ed1215c50&selectedTaskRun=MlF6ilf-Q_itqcygvcLqQA.5

Comment 19

[Intermittent Failures Robot]

20 failures in 3375 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 19
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-09-13&endday=2021-09-19&tree=all

Comment 20

[Kris Wright :KrisWright]

(In reply to Cristian Tuns from comment #18)

This started to fail as an exception on Linux WebRender asan opt.
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cpending%2Crunning%2Cretry&fromchange=55fa1d670b119502b16f8795990f9456e1bfcd8c&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Casan%2Copt%2Cmochitests%2Cwith%2Csoftware%2Cwebrender%2Cand%2Cfission%2Cenabled%2Ctest-linux1804-64-asan-qr%2Fopt-mochitest-browser-chrome-swr-fis-e10s%2Cbc13&tochange=728ead24c5e80f438419b406678c567ed1215c50&selectedTaskRun=MlF6ilf-Q_itqcygvcLqQA.5

In the past when I've seen this it was often related to resource exhaustion. If there is something that has recently caused asan to go out of memory then we may run into this issue. From my understanding the machine crashes before it can generate any usable artifacts.

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Recent cases are being handled in bug 1731580 (and regressor seems to have been identified).

Comment 22

[Intermittent Failures Robot]

3 failures in 2873 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2021-09-20&endday=2021-09-26&tree=all

Comment 23

[Andreea Pavel [:apavel]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #21)

Recent cases are being handled in bug 1731580 (and regressor seems to have been identified).

Can this be closed?

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Usually we keep the bug open until the issue has been fixed and don't close it when the test gets skipped. Developers can still choose to close it and work on the fix in a different bug.

Comment 25

[Intermittent Failures Robot]

5 failures in 3159 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2022-04-18&endday=2022-04-24&tree=all

Comment 26

[Intermittent Failures Robot]

2 failures in 2449 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• linux64: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2022-10-03&endday=2022-10-09&tree=all

Comment 27

[Intermittent Failures Robot]

2 failures in 3643 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-02-20&endday=2023-02-26&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 3276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-07-31&endday=2023-08-06&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 3341 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-08-07&endday=2023-08-13&tree=all

Comment 30

[Intermittent Failures Robot]

1 failures in 3019 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 31

[Intermittent Failures Robot]

2 failures in 3267 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 32

[Intermittent Failures Robot]

1 failures in 3337 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 33

[Intermittent Failures Robot]

3 failures in 3850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 3726 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 35

[Intermittent Failures Robot]

2 failures in 3764 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 36

[Intermittent Failures Robot]

1 failures in 3484 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1712866&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 37

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:emilio, maybe it's time to close this bug?
For more information, please visit BugBot documentation.

Comment 38

[Emilio Cobos Álvarez (:emilio)]

Not working actively on it, but it seems this hasn't happened in quite a while, let's file a new bug if it ever happens again.