1904963 - Perma linux asan canvas [tier 2] [taskcluster:error] Task timeout after 1800 seconds. Force killing container. | single tracking bug

Status: RESOLVED FIXED

(Core :: DOM: Content Processes, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=464147572&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bMi-owgmS5a7mCdjaeQWUQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-06-26T19:03:18.467Z] 19:03:18     INFO - TEST-START | /html/canvas/offscreen/path-objects/2d.path.clip.basic.2.html
[task 2024-06-26T19:03:18.545Z] 19:03:18     INFO - Closing window 3ab6258d-39ca-4ed1-8f41-80cd56bf4685
[task 2024-06-26T19:03:18.619Z] 19:03:18     INFO - PID 1012 | -----------------------------------------------------
[task 2024-06-26T19:03:18.620Z] 19:03:18     INFO - PID 1012 | Suppressions used:
[task 2024-06-26T19:03:18.621Z] 19:03:18     INFO - PID 1012 |   count      bytes template
[task 2024-06-26T19:03:18.621Z] 19:03:18     INFO - PID 1012 |      31      16288 nsComponentManagerImpl
[task 2024-06-26T19:03:18.622Z] 19:03:18     INFO - PID 1012 |       2        288 libfontconfig.so
[task 2024-06-26T19:03:18.623Z] 19:03:18     INFO - PID 1012 |       1       9240 style::sharing::SHARING_CACHE_KEY
[task 2024-06-26T19:03:18.624Z] 19:03:18     INFO - PID 1012 |       1       4104 style::bloom::BLOOM_KEY
[task 2024-06-26T19:03:18.624Z] 19:03:18     INFO - PID 1012 | -----------------------------------------------------
[taskcluster:error] Task timeout after 1800 seconds. Force killing container.
[taskcluster 2024-06-26 19:03:21.518Z] === Task Finished ===
[taskcluster 2024-06-26 19:03:21.518Z] Unsuccessful task run with exit code: -1 completed in 1804.01 seconds

Comment 1

[Iulian Moraru]

Hi Andrew! Can you please take a look at this? Could this be something regressed by the recent changes from Bug 1901076?
Thank you!

Comment 2

[Cosmin Sabou [:CosminS]]

The regressor cannot be identified on autoland as the job will fail with No checks run.
It first started with this pushlog merged to central: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=653f0dc8442dd9cae70845896eb3c0a0252677b3

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

The test case that is failing doesn't even use any images, so it seems unlikely:
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/html/canvas/element/path-objects/2d.path.clip.basic.2.html

Comment 4

[Timothy Nikkel (:tnikkel)]

I bisected this on try to bug 1728331.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1728331

:nika, since you are the author of the regressor, bug 1728331, could you take a look?

For more information, please visit BugBot documentation.

Comment 6

[Nika Layzell [:nika] (ni? for response)]

It appears that this timeout is happening because the test is taking too long to run, rather than because of a hang or similar while running the test.

Given that this is a nofis test, my guess is that the change to remove E10S-only process recycling is negatively interacting with this particular test suite in some way, causing it to run much longer than previously. In the log linked from comment 0, there are Suppressions used: outputs relatively frequently, which I am guessing correspond to a content process shutting down. Looking at a non-failing run from the backlog, I see much less frequent Suppressions used: logs (e.g. https://treeherder.mozilla.org/logviewer?job_id=464147617&repo=mozilla-central&lineNumber=6899).

We probably don't want to try to restore the E10S recycling logic, as we don't support e10s on desktop anymore, and it's a significant amount of complexity which would only be used for test code. It might be possible to mitigate the process recycling by setting dom.ipc.keepProcesses.web to a non-zero number in this test suite when running with Fission disabled, which may reduce process shutdown/startup cycles.

Comment 7

[Intermittent Failures Robot]

14 failures in 4116 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14

Platform and build breakdown:

• linux1804-64-asan-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 8

[Intermittent Failures Robot]

11 failures in 3199 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11

Platform and build breakdown:

• linux1804-64-asan-qr: 10
  • opt: 10
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 9

[Jens Stutte [:jstutte]]

It might be possible to mitigate the process recycling by setting dom.ipc.keepProcesses.web to a non-zero number in this test suite when running with Fission disabled, which may reduce process shutdown/startup cycles.

Artur, would you mind to try this ?

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1728331

Comment 11

[Artur Iunusov]

Just changed .ini file, and then questioned myself, if this pref is actually used anywhere:

https://searchfox.org/mozilla-central/search?q=keepProcessesAlive.web&path=&case=false&regexp=false

(still looking)

Comment 12

[Artur Iunusov]

Probably, used there: https://searchfox.org/mozilla-central/source/dom/ipc/ContentParent.cpp#2173

Comment 13

[Artur Iunusov]

Attachment: Bug 1904963 - canvas wpt: set dom.ipc.keepProcessesAlive.web = 1 when fission is disabled, r=jstutte

Comment 14

[Intermittent Failures Robot]

17 failures in 3780 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 17

Platform and build breakdown:

• linux1804-64-asan-qr: 16
  • opt: 16
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-07-08&endday=2024-07-14&tree=all

Comment 15

[Intermittent Failures Robot]

17 failures in 3799 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 16
  • opt: 16
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2bfa2e58766c
canvas wpt: set dom.ipc.keepProcessesAlive.web = 1 when fission is disabled, r=jstutte

Comment 17

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/2bfa2e58766c

Comment 18

[Iulian Moraru]

Hi Artur! Can you please take another look at this? the issue is still happening -> check here
Thank you!

Comment 19

[Artur Iunusov]

(looking)

Comment 20

[Artur Iunusov]

https://bugzilla.mozilla.org/show_bug.cgi?id=1891526 - see also

(going to try them locally in asan environment)

Comment 21

[Intermittent Failures Robot]

25 failures in 4124 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 21
• try: 4

Platform and build breakdown:

• linux1804-64-asan-qr: 23
  • opt: 23
• linux1804-64-qr: 1
  • debug: 1
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-07-22&endday=2024-07-28&tree=all

Comment 22

[Intermittent Failures Robot]

20 failures in 3839 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 19
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-07-29&endday=2024-08-04&tree=all

Comment 23

[Intermittent Failures Robot]

24 failures in 3459 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 23
• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 21
  • opt: 21
• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 24

[Intermittent Failures Robot]

21 failures in 2154 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 20
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 21
  • opt: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We noticed a bug in Marionette lately that I'm going to fix on bug 1761634. With it's landing each test will need some more milliseconds to complete because now we will correctly wait for the initial about:blank to be loaded. Based on that I'm going ahead and split the canvas jobs into 3 chunks and also increase the task timeout from 1800 to 2700, which brings us to the same settings as for other jobs. That means that this should also help here so that we no longer see these task timeouts.

Comment 26

[Intermittent Failures Robot]

16 failures in 3218 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13
• try: 3

Platform and build breakdown:

• linux1804-64-asan-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1904963&startday=2024-08-19&endday=2024-08-25&tree=all

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The failures seem to have stopped after my patch landed. I'll re-check later this week and will mark the bug as fixed if there will be still no failures reported.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I can verify that ASAN builds do no longer cause a timeout of the task since bug 1761634 landed. Marking this bug as fixed.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:whimboo, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox130 to wontfix.

For more information, please visit BugBot documentation.