1863599 - Crash in [@ shutdownhang | js::frontend::ScopeStencil::appendScopeStencilAndData<T>]

Status: NEW

(Core :: JavaScript Engine, defect, P3)

Description

[Robert Hartmann]

Crash report: https://crash-stats.mozilla.org/report/index/7f0d8967-bd8c-48a3-ae2e-e78d30231107

MOZ_CRASH Reason: Shutdown hanging at step AppShutdownConfirmed. Something is blocking the main-thread.

Top 10 frames of crashing thread:

0  xul.dll  js::frontend::ScopeStencil::appendScopeStencilAndData<js::ScopeKind&, mozilla::Maybe<js::ScopeIndex>&, unsigned int&, mozilla::Maybe<unsigned int>&>  js/src/vm/Scope.cpp:1344
1  ?  @0x0000026f7e828f3f  
2  mozglue.dll  moz_arena_malloc  memory/build/malloc_decls.h:150
3  xul.dll  js_free  js/public/Utility.h:418
3  xul.dll  js::TempAllocPolicy::free_  js/public/AllocPolicy.h:207
3  xul.dll  mozilla::detail::HashTable<const js::frontend::TaggedParserAtomIndex, mozilla::HashSet<js::frontend::TaggedParserAtomIndex, js::frontend::TaggedParserAtomIndexHasher, js::TempAllocPolicy>::SetHashPolicy, js::TempAllocPolicy>::freeTable  mfbt/HashTable.h:1701
3  xul.dll  mozilla::detail::HashTable<const js::frontend::TaggedParserAtomIndex, mozilla::HashSet<js::frontend::TaggedParserAtomIndex, js::frontend::TaggedParserAtomIndexHasher, js::TempAllocPolicy>::SetHashPolicy, js::TempAllocPolicy>::changeTableSize  mfbt/HashTable.h:1887
4  ?  @0x000000e2a27fb557  
5  xul.dll  js::frontend::RewritingParseNodeVisitor<FoldVisitor>::visit  js/src/frontend/ParseNodeVisitor.h:120
6  ?  @0x0000026f7f7fffff  

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::JavaScript Engine' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a crash signature, thus the bug will be considered confirmed.

Comment 3

[Steven DeTar [:sdetar]]

arai, since this seems stencil related, would it be appropriate for you to look at this.

Comment 4

[Nicolas B. Pierron [:nbp]]

Given the corrupted stack trace, and the low reproducibility on crash-stat … unfortunately there is not much actionable in the current crash report.

Comment 5

[Nicolas B. Pierron [:nbp]]

Is this crash reproducible, or this was only this one time?

Comment 6

[Tooru Fujisawa [:arai]]

If the ParseNodeVisitor frames seen in some reports are really related to the crash, this might be related to bug 1801916.
(it's unclear, given all reports have corrupted stack trace and ParseNodeVisitor frame is beyond the corrupted frame)

if that's the case, there's ongoing work to refactor/clean up the allocation and the error handling inside the parser (bug 1849732 etc) in order to further investigate the issue around bug 1801916, and that might help here as well.

Comment 7

[Robert Hartmann]

(In reply to Nicolas B. Pierron [:nbp] from comment #5)

Is this crash reproducible, or this was only this one time?

well, that is a good question. But I do not know the answer; in the last time I get multiple (different) stack traces
containing FF shutdown hangs/crashes :

bp-61f4d1de-cb0a-4ae7-ac24-ad96e0231111 submitted at 11.11.2023, 17:39
bp-4423deda-81df-43d1-b5fa-7d7d80231107 submitted at 07.11.2023, 21:05
bp-b3488f84-4f04-4f4b-8cfb-5fca80231107 submitted at 07.11.2023, 21:05
bp-7f0d8967-bd8c-48a3-ae2e-e78d30231107 submitted at 07.11.2023, 21:04
bp-b15b540a-c2ea-4ace-9e0b-c6fb80231105 submitted at 05.11.2023, 20:43
bp-d49ca966-d72e-4c6c-b772-76b280231028 submitted at 28.10.2023, 10:47
bp-fb2048dc-8e28-4179-9728-814bd0231014 submitted at 14.10.2023, 17:12
bp-6da90ba8-98f6-4964-b678-b94b40231012 submitted at 12.10.2023, 14:05
bp-f25c0a12-c358-440e-8109-417fe0231012 submitted at 12.10.2023, 14:04
bp-2398daa2-583b-4dfe-84b0-508630231012 submitted at 12.10.2023, 13:51
bp-40a8ee51-e450-4cc4-a9b0-17bf10231012 submitted at 12.10.2023, 13:51
bp-62eb3e9b-6a8a-4e44-a553-1361d0231011 submitted at 11.10.2023, 17:21
bp-66da74c6-b456-4a24-8840-ecb680231010 submitted at 10.10.2023, 21:24
bp-8522a86b-c3d9-4a46-a22b-d0dfd0231010 submitted at 10.10.2023, 21:24
bp-5dc36ba2-80d8-46f0-aeb1-cda600231003 submitted at 03.10.2023, 16:38
bp-b8099ebe-bbbc-4004-8e46-a9be90230924 submitted at 24.09.2023, 19:30
bp-ebd3e370-e620-4f06-bce6-2819d0230903 submitted at 03.09.2023, 12:28
bp-ca098112-4df4-46ef-8e86-c7c930230903 submitted at 03.09.2023, 12:28

Comment 8

[Nicolas B. Pierron [:nbp]]

Thanks for listing all these crash signature, the problem is that shutdown hangs are difficult to investigate as they have many different signatures which seems unrelated with the hanging task …

I do not know how to handle these issues.

Some of the signatures are reported against the JavaScript Engine, Firefox, XPCOM or the Networking stack. And the reality is that probably none of these components are responsible for what you are experiencing. They just happen to be running because some script needs to be executed and something might miss-behave, or the time to execute this script is larger than expected.

These bugs are: Bug 1726168, Bug 1749178, Bug 1801819 are similar issues reported by others previous and Bug 1863228, Bug 1863600, Bug 1863602, Bug 1858237, Bug 1858238, Bug 1859120, Bug 1858662, Bug 1858665, Bug 1861866, Bug 1858677 were reported by you previously.

I would think that the most plausible explanation is that most of the bug reports you made have the same root cause.

The best I can do is raise awareness around this bug hopping that this would reach the right person …

Comment 9

[Jens Stutte [:jstutte]]

It seems to me that most of these crashes are coming from background tasks, namely either defaultagent or backgroundupdate (look out for the BackgroundTaskName annotation). I think we should somehow collect those separately.

This bug as such is probably not really helpful but more a reminder that we should find better ways of associating those crashes with more specific bugs. Things that could help to cluster them that are in our crash reports might be:

• BackgroundTaskName if present
• The shutdown phase from MOZ_CRASH_REASON
• The rest of the signature as is

From my previous peeking into background task shutdown I would assume/hope that there is no "good" reason for those to hang (we usually just execute a simple Javascript payload that might or might not be expensive and then just exit) and that putting some more attention on them could lead to a common root cause, like a race between the OS asking us to shutdown and the payload still executing or such.

Comment 10

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

From my previous peeking into background task shutdown I would assume/hope that there is no "good" reason for those to hang (we usually just execute a simple Javascript payload that might or might not be expensive and then just exit) and that putting some more attention on them could lead to a common root cause, like a race between the OS asking us to shutdown and the payload still executing or such.

This is my belief as well. At one time the background task mechanism had few delays so that startup was racing shutdown but we think that we have largely eliminated that with https://bugzilla.mozilla.org/show_bug.cgi?id=1832252.

Strong +1 for including BackgroundTaskName in a prominent place for crash bugs filed. Suhaib, perhaps you can help with this? BackgroundTaskName is a crash annotation added in https://bugzilla.mozilla.org/show_bug.cgi?id=1697875 and supported by Socorro after https://bugzilla.mozilla.org/show_bug.cgi?id=1796132.

Comment 11

[Gabriele Svelto [:gsvelto]]

(In reply to Nick Alexander :nalexander [he/him] from comment #10)

Strong +1 for including BackgroundTaskName in a prominent place for crash bugs filed. Suhaib, perhaps you can help with this? BackgroundTaskName is a crash annotation added in https://bugzilla.mozilla.org/show_bug.cgi?id=1697875 and supported by Socorro after https://bugzilla.mozilla.org/show_bug.cgi?id=1796132.

We'll need to change Socorro's signature generation for that, I'll file a bug.

Comment 12

[Gabriele Svelto [:gsvelto]]

FYI I quickly checked what this would yield and here's the breakdown:

• Roughly half of the shutdownhang signatures will remain the same, as they have no BackgroundTaskName annotation
• ~2,5% of the crashes will have the defaultagent task name in the signature
• ~1% of the crashes will have the removeDirectory task name in the signature
• a whopping 42% of the crashes of the crashes will have backgroundupdate in the signature, which sounds like it's currently the biggest reason for the hangs

Comment 13

[Suhaib Mujahid [:suhaib]]

(In reply to Nick Alexander :nalexander [he/him] from comment #10)

Strong +1 for including BackgroundTaskName in a prominent place for crash bugs filed. Suhaib, perhaps you can help with this?

We could make the bot indicate that in the bug description. We could do it similarly to how the bot indicates whether the crash is on a null address. However, if this will be part of the crash signature, I'm not sure if it is necessary to include it in the bug description as well.

Comment 14

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to Gabriele Svelto [:gsvelto] from comment #12)

FYI I quickly checked what this would yield and here's the breakdown:

• Roughly half of the shutdownhang signatures will remain the same, as they have no BackgroundTaskName annotation
• ~2,5% of the crashes will have the defaultagent task name in the signature
• ~1% of the crashes will have the removeDirectory task name in the signature
• a whopping 42% of the crashes of the crashes will have backgroundupdate in the signature, which sounds like it's currently the biggest reason for the hangs

There's two things happening here:

1. backgroundupdate is by far the most common background task -- essentially the only one in release at this time;
2. It's clearly exposing something in the JS engine that's not right :)

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release

:willyelm, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit BugBot documentation.