1734543 - JavaScript error: resource://gre/modules/CrashManager.jsm, line <random>: AbortError: IOUtils: Shutting down and refusing additional I/O tasks

Status: NEW

(Toolkit :: Crash Reporting, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0 ID:20211003201113

Running the Puppeteer unit tests on MacOS locally result in a lot of shutdown hangs. For most of these cases I can see the following JavaScript error in the log:

JavaScript error: resource://gre/modules/CrashManager.jsm, line 828: AbortError: IOUtils: Shutting down and refusing additional I/O tasks

As result the crash reporter claims that no crash report could be written because the application data file hasn't been found. And as it looks like here we have a case that writing this file to disk is refused by IOUtils because of an ongoing shutdown.

The steps to take to reproduce this are:

1. Change this line to use describe.only
2. Run ./mach puppeteer-test --subset

To reproduce the shutdown hangs you have to use an artifact build (maybe also a full build will be ok), but a Nightly will not show them. I'm still investigating why these shutdown hangs only happen for these kind of builds.

Gabriele, could you please have a look why we are failing here?

Comment 1

[Gabriele Svelto [:gsvelto]]

Our error-handling in that particular area is incomplete, this is caused by bug 1694624.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:gsvelto, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Gabriele, most of the shutdown hangs as listed on bug 1358898 have that error in the lines. Do you think that it could be one of the culprits?

Comment 4

[Gabriele Svelto [:gsvelto]]

Yes, it could be. We are considering getting rid of the crash manager entirely in bug 1911301.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Gabriele. Does it mean bug 1694624 can be closed as wontfix or could this bug a workaround until bug 1911301 is fixed, which seems to be a complex refactoring that might take longer.

Comment 6

[Gabriele Svelto [:gsvelto]]

Yes, I forgot about bug 1694624, that should be an effective workaround and is relatively easy to fix. I'll prioritize it.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The line has changed and as I can see a good number of bugs seem to have been classified for bug 1358898. I'll try to correct some of them.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriele Svelto [:gsvelto] from comment #6)

Yes, I forgot about bug 1694624, that should be an effective workaround and is relatively easy to fix. I'll prioritize it.

The number of failures (21 in the last week) isn't that high but I wonder if you were able to make some progress on bug 1694624. Thanks Gabriele!

Comment 9

[Gabriele Svelto [:gsvelto]]

No I hadn't, but I've quickly cobbled a patch together, let's see if it survives testing.

Comment 10

[Intermittent Failures Robot]

23 failures in 3574 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• mozilla-beta: 11
• autoland: 3
• try: 4
• mozilla-esr128: 1

Platform and build breakdown:

• linux1804-64-qr: 7
  • debug: 7
• linux1804-64-devedition-qr: 6
  • opt: 6
• linux1804-64-asan-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 2
  • opt: 2
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• windows10-64-2009-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-09-30&endday=2024-10-06&tree=all

Comment 11

[Intermittent Failures Robot]

37 failures in 4679 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 21
• autoland: 7
• try: 2
• mozilla-release: 2
• mozilla-esr115: 1
• mozilla-esr128: 4

Platform and build breakdown:

• linux1804-64-devedition-qr: 9
  • opt: 9
• windows10-64-2009-shippable-qr: 2
  • opt: 2
• linux1804-64-qr: 10
  • debug: 10
• windows11-64-2009-devedition-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 9
  • opt: 9
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• linux1804-64-asan-qr: 2
  • opt: 2
• macosx1015-64-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-10-07&endday=2024-10-13&tree=all

Comment 12

[Intermittent Failures Robot]

29 failures in 4579 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 16
• mozilla-central: 4
• try: 5
• mozilla-esr115: 1
• autoland: 3

Platform and build breakdown:

• linux1804-64-qr: 9
  • debug: 9
• windows10-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 2
  • opt: 2
• linux1804-64-devedition-qr: 8
  • opt: 8
• linux1804-64-shippable-qr: 4
  • opt: 4
• macosx1015-64-devedition-qr: 1
  • opt: 1
• windows11-32-2009-shippable-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-10-14&endday=2024-10-20&tree=all

Comment 13

[Intermittent Failures Robot]

9 failures in 4456 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 1
• mozilla-central: 2
• autoland: 3

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 2
  • opt: 2
• linux1804-64-qr: 5
  • debug: 5
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 14

[Intermittent Failures Robot]

9 failures in 4456 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 1
• mozilla-central: 2
• autoland: 3

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 2
  • opt: 2
• linux1804-64-qr: 5
  • debug: 5
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 15

[Intermittent Failures Robot]

31 failures in 4361 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 6
• try: 6
• mozilla-beta: 13
• mozilla-release: 6

Platform and build breakdown:

• linux1804-64-qr: 7
  • debug: 7
• linux1804-64-devedition-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 6
  • opt: 6
• windows11-32-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-devedition-qr: 3
  • opt: 3
• linux1804-64-asan-qr: 1
  • opt: 1
• windows10-64-2009-shippable-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-10-28&endday=2024-11-03&tree=all

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Gabriele, there are still issues on autoland. Is that expected?

Comment 17

[Gabriele Svelto [:gsvelto]]

It wasn't expected but I see it's a different problem. I now realize that my patch only addressed issues around the crash manager store not being accessible, but we can still have failures with I/O operations on other files - like the ones in a crash report. I have to put guards around all I/O operations to make this code more robust. I'll file a separate bug.

Comment 18

[Intermittent Failures Robot]

23 failures in 4515 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 2
• mozilla-beta: 12
• mozilla-esr128: 2
• mozilla-release: 5

Platform and build breakdown:

• windows11-32-2009-shippable-qr: 3
  • opt: 3
• linux1804-64-qr: 2
  • debug: 2
• linux1804-64-devedition-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 10
  • opt: 10
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 3
  • opt: 3
• windows11-64-2009-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1734543&startday=2024-11-04&endday=2024-11-10&tree=all

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Intermittent Failures Robot from comment #18)

23 failures in 4515 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

All these failures seem to be unrelated to this bug. I filed bug 1930437 to cover this specific shutdown hang.