1623229 - Perma retries/timeouts on linux64-tsan opt Xpcshell opt tests that end as exceptions/timeouts

Status: RESOLVED DUPLICATE of bug 1624267

(Firefox :: Remote Settings Client, defect)

Description

[Cosmin Sabou [:CosminS]]

These xpcshell jobs started retrying a lot after Bug 1620185 landed here: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=linux64-tsan%2Copt%2Cxpcshell%2Ctests%2Ctest-linux64-tsan%2Fopt-xpcshell-e10s-1%2Cx%28x1%29&tochange=01641a691d4d4f67cd148c8c0b9981b0d8c2497b&fromchange=9ee1a15632f89853a5ab28ef60566f5508c62783&selectedJob=293602028

task: https://firefox-ci-tc.services.mozilla.com/tasks/Il-pKu4PREKoQ7_CeKnF7w

Matthew, please take a look over this.

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1623229 - run Linux tsan xcshell tests with t-linux-xlarge to prevent retries and out-of-memory crashes. r=jmaher DONTBUILD

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/291cd2aba045
run Linux tsan xcshell tests with t-linux-xlarge to prevent retries and out-of-memory crashes. r=jmaher DONTBUILD

Comment 3

[Mathieu Leplatre [:leplatrem]]

Where can I see which test is failing or causing issue?
Thanks!

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Assigning a more powerful machine didn't help, Linux tsan xpcshell still gets retried. Christian, any suggestions what could be done here?

Tom: Can you check what's different for the runs on Linux tsan X1 before and after bug 1620185 landed? It gets retried and fails with an exception after 5 retries and no publicly accessible log.

Comment 5

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4c82ec92ccea
Backed out changeset 291cd2aba045 as requested by Aryx.

Comment 6

[Alexandru Michis [:malexandru]]

We've also started seeing an increase in xpcshell timeouts in linux tsan opt since the changes in Bug 1620185 landed:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=linux64-tsan%2Copt%2Cxpcshell&tochange=613b3f3a2599f2603fe5f4d9b274b0e1517b7910&fromchange=c5a8b641d9050b73462b24cfea6dd89833e560b6&selectedJob=293684129

Comment 7

[Intermittent Failures Robot]

54 failures in 908 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 50
• mozilla-central: 4

Platform breakdown:

• linux64-tsan: 54

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-18&endday=2020-03-18&tree=all

Comment 8

[Christian Holler (:decoder)]

I've investigated this a bit, using the one log where we fail but the log was not empty. It shows this:

[task 2020-03-18T23:35:53.304Z] 23:35:53     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_contentscript_api_injection.js | took 300003ms
[task 2020-03-18T23:46:36.021Z] 23:46:36     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_proxy_startup.js | took 300002ms
[task 2020-03-18T23:49:21.089Z] 23:49:21     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_sandbox_var.js | took 300003ms
[task 2020-03-18T23:56:56.128Z] 23:56:27     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_storage_content.js | took 364577ms
[task 2020-03-18T23:57:31.119Z] 23:57:31     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_userScripts_telemetry.js | took 345839ms
[task 2020-03-18T23:57:31.123Z] 23:57:31     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_userScripts.js | took 509902ms
[task 2020-03-18T23:57:31.123Z] 23:57:31     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_userScripts_exports.js | took 443237ms
[task 2020-03-19T00:06:13.092Z] 00:06:12     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_filterResponseData.js | took 346304ms
[task 2020-03-19T00:06:13.097Z] 00:06:13     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_auth.js | took 403819ms
[task 2020-03-19T00:06:13.097Z] 00:06:13     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_cancelWithReason.js | took 457827ms
[task 2020-03-19T00:06:13.097Z] 00:06:13     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_cached.js | took 487932ms
[task 2020-03-19T00:14:39.964Z] 00:14:39     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_host.js | took 409210ms
[task 2020-03-19T00:14:39.969Z] 00:14:39     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_permission.js | took 378528ms
[task 2020-03-19T00:14:39.969Z] 00:14:39     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_requestSize.js | took 421203ms
[task 2020-03-19T00:14:39.969Z] 00:14:39     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_incognito.js | took 453425ms
[task 2020-03-19T00:23:48.226Z] 00:23:48     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_set_cookie.js | took 357756ms
[task 2020-03-19T00:23:48.229Z] 00:23:48     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_responseBody.js | took 406775ms
[task 2020-03-19T00:23:48.229Z] 00:23:48     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_startup.js | took 533459ms
[task 2020-03-19T00:23:48.229Z] 00:23:48     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_suspend.js | took 506387ms
[task 2020-03-19T00:33:30.160Z] 00:33:29     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_userContextId.js | took 435538ms
[task 2020-03-19T00:33:30.170Z] 00:33:30     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_webSocket.js | took 498627ms
[task 2020-03-19T00:33:30.170Z] 00:33:30     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_xhr_capabilities.js | took 525875ms
[task 2020-03-19T00:33:30.170Z] 00:33:30     INFO -  TEST-TIMEOUT | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_permissions.js | took 517064ms

I've grepped old logs for the first test and this test used to complete in around 50 seconds on TSan. I could now disable all of these tests, but I am reluctant to do so without understanding why these tests now suddenly seem to take more than 10 times as long or even produce a hang, while they worked before.

Mathieu, now that we know which tests are failing, can you provide some insights as to how your changes might be related?

Comment 9

[Mathieu Leplatre [:leplatrem]]

Those tests seem completely unrelated to me. The extensions component does not use Remote Settings 🤷

It might be somewhere else :/

Comment 10

[Intermittent Failures Robot]

128 failures in 935 pushes (0.137 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 113
• mozilla-central: 11
• try: 4

Platform breakdown:

• linux64-tsan: 128

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-19&endday=2020-03-19&tree=all

Comment 11

[Intermittent Failures Robot]

130 failures in 863 pushes (0.151 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 120
• mozilla-central: 6
• try: 4

Platform breakdown:

• linux64-tsan: 130

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-20&endday=2020-03-20&tree=all

Comment 12

[Intermittent Failures Robot]

39 failures in 238 pushes (0.164 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 35
• mozilla-central: 4

Platform breakdown:

• linux64-tsan: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-21&endday=2020-03-21&tree=all

Comment 13

[Intermittent Failures Robot]

431 failures in 4955 pushes (0.087 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 392
• mozilla-central: 29
• try: 10

Platform breakdown:

• linux64-tsan: 431

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 14

[Intermittent Failures Robot]

44 failures in 873 pushes (0.05 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 36
• mozilla-central: 4
• try: 4

Platform breakdown:

• linux64-tsan: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-23&endday=2020-03-23&tree=all

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Resolved by TSan fix in bug 1624267.

Comment 16

[Intermittent Failures Robot]

44 failures in 4773 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• mozilla-central: 4
• try: 4

Platform breakdown:

• linux64-tsan: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 5015 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1623229&startday=2020-03-30&endday=2020-04-05&tree=all