1592336 - Crash in [@ static void mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<T>::operator()::<T>::__invoke]

Status: RESOLVED DUPLICATE of bug 1588498

(Core :: Storage: Quota Manager, defect, P3)

Description

[Calixte Denizet (:calixte)]

This bug is for crash report bp-01101f98-95f5-48d9-80bc-7adfb0191024.

Top 10 frames of crashing thread:

0 xul.dll static void mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<unnamed-tag>::operator dom/localstorage/ActorsParent.cpp:9254
1 xul.dll nsTimerImpl::Fire xpcom/threads/nsTimerImpl.cpp:561
2 xul.dll nsTimerEvent::Run xpcom/threads/TimerThread.cpp:260
3 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1225
4 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:486
5 xul.dll void mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads dom/localstorage/ActorsParent.cpp:9272
6 xul.dll void mozilla::dom::quota::QuotaManager::Shutdown dom/quota/ActorsParent.cpp:3816
7 xul.dll static void mozilla::dom::quota::QuotaManager::ShutdownInstance dom/quota/ActorsParent.cpp:3403
8 xul.dll mozilla::dom::quota::RecvShutdownQuotaManager dom/quota/ActorsParent.cpp:2661
9 xul.dll mozilla::ipc::BackgroundParentImpl::RecvShutdownQuotaManager ipc/glue/BackgroundParentImpl.cpp:1037

There are 73 crashes (from 73 installations) in 70.0 with buildid 20191016161957. In analyzing the backtrace, the regression may have been introduced by patch [1] to fix bug 1587931.

[1] https://hg.mozilla.org/releases/mozilla-release/rev?node=8b0e6968831f

Comment 1

[Jan Varga [:janv]]

No, this is just another signature variant for LS shutdown hangs, bug 1541928.

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

tracking in the other bug.

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

Jan, actually it might make sense to keep this bug open for now, since bug 1587931 is fixed, and this is a really specific spike on release (that we'll likely see go away after 70.1) What do you think?

Comment 4

[Jan Varga [:janv]]

Ok.

Comment 5

[Gabriele Svelto [:gsvelto]]

Adding some macOS- and Linux-specific signatures. I noticed that there's other bugs tracking similar signatures such as bug 1593863 and bug 1588498. Should we close all the redundant ones and move the signatures to one of those?

Comment 6

[Jens Stutte [:jstutte]]

(In reply to Gabriele Svelto [:gsvelto] from comment #5)

Adding some macOS- and Linux-specific signatures. I noticed that there's other bugs tracking similar signatures such as bug 1593863 and bug 1588498. Should we close all the redundant ones and move the signatures to one of those?

Hi Gabriele, yes, please. Can you proceed with this?

Comment 8

[Gabriele Svelto [:gsvelto]]

I consolidated all the signatures here and closed bug 1593863. Also moved the NI? from there.

Comment 9

[Jens Stutte [:jstutte]]

Andrew, I moved also your assignment here. Let me know, if you can work on this, though. Thank you!

Comment 10

[Jens Stutte [:jstutte]]

Jan, probably this is more up your alley?

Comment 11

[Jan Varga [:janv]]

Some time ago I split the crashes into individual bugs for IDB and LSNG, but given low frequency at the time I closed those bugs and filed bug 1588498 for remaining issues.

It seems we need to create individual bugs again like bug 1541928 and bug 1541776 to see how given quota clients (modules) behave.

mozilla::dom::(anonymous namespace)::* are shutdown hangs in LSNG
mozilla::dom::indexedDB::(anonymous namespace)::* are shutdown hangs in IDB

Comment 12

[Jens Stutte [:jstutte]]

So I understand, that:

• the crashes here are the same as those collected in bug 1588498
• we actually want to have two bugs

Do we then want to keep bug 1588498 as meta bug collecting all signatures and create concrete todos as bugs that mitigate single aspects/crashes? This seems to be the current idea behind bug 1588498, AFAICS. So once we know, what we can do about a single crash variant, we file the bug for it. This would mean to just close this as duplicate, as it does not add any information, to my feeling.

Comment 13

[Jan Varga [:janv]]

Yeah, we need just two bugs, one for IDB and one for LSNG.
So bug 1588498 needs to be split, including the dependent bugs.
The signatures in this bug should be used in the new bugs. Once the new bugs are filed, this bug can be closed.

Comment 15

[Jens Stutte [:jstutte]]

It seems from the volume that the release of FF84 helped a lot here?

Comment 16

[Jan Varga [:janv]]

(In reply to Jens Stutte [:jstutte] from comment #15)

It seems from the volume that the release of FF84 helped a lot here?
Parallel quota client shutdown ?

Comment 17

[Simon Giesecke [:sg] [he/him]]

(In reply to Jan Varga [:janv] from comment #16)

(In reply to Jens Stutte [:jstutte] from comment #15)

It seems from the volume that the release of FF84 helped a lot here?
Parallel quota client shutdown ?

Possibly, but unfortunately there are so many signature variants (changing between platforms and builds), that it's not trivial to get a reliable overview.

Comment 18

[Jens Stutte [:jstutte]]

(In reply to Simon Giesecke [:sg] [he/him] from comment #17)

Possibly, but unfortunately there are so many signature variants (changing between platforms and builds), that it's not trivial to get a reliable overview.

The few uncovered signatures I found now did not add significant volume. But yes, the constantly varying signatures (including thread numbers, it seems?) are a bit annoying here.

Comment 19

[Simon Giesecke [:sg] [he/him]]

(In reply to Jens Stutte [:jstutte] from comment #18)

(In reply to Simon Giesecke [:sg] [he/him] from comment #17)

Possibly, but unfortunately there are so many signature variants (changing between platforms and builds), that it's not trivial to get a reliable overview.

The few uncovered signatures I found now did not add significant volume.

How did you check that?

But yes, the constantly varying signatures (including thread numbers, it seems?) are a bit annoying here.

I think the number you are referring to, e.g. in mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_91::__invoke, is the number of the lambda expression in some scope.

Comment 20

[Jens Stutte [:jstutte]]

(In reply to Simon Giesecke [:sg] [he/him] from comment #19)

The few uncovered signatures I found now did not add significant volume.

How did you check that?

Looking at the graph here on the bug after adding the signature. Or is there a time lag in reflecting changes to the signatures there?

But yes, the constantly varying signatures (including thread numbers, it seems?) are a bit annoying here.

I think the number you are referring to, e.g. in mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_91::__invoke, is the number of the lambda expression in some scope.

Ah ok, that makes more sense, indeed.

Comment 21

[Simon Giesecke [:sg] [he/him]]

(In reply to Jens Stutte [:jstutte] from comment #20)

(In reply to Simon Giesecke [:sg] [he/him] from comment #19)

The few uncovered signatures I found now did not add significant volume.

How did you check that?

Looking at the graph here on the bug after adding the signature. Or is there a time lag in reflecting changes to the signatures there?

My question was more about "uncovered signatures". I now added another one, which is the most frequent for 84 as it seems. The parallel shutdown is not in 84.

Comment 22

[Jens Stutte [:jstutte]]

(In reply to Simon Giesecke [:sg] [he/him] from comment #21)

(In reply to Jens Stutte [:jstutte] from comment #20)

Looking at the graph here on the bug after adding the signature. Or is there a time lag in reflecting changes to the signatures there?

My question was more about "uncovered signatures". I now added another one, which is the most frequent for 84 as it seems.

I just searched for "ShutdownWorkThreads" in crashstats. But I now realized, that I need to widen the date interval to get more of them due to the frequent signature changes.

The parallel shutdown is not in 84.

That is probably good news, because we can still hope it has some effect then here!

(Removing :asuth's old needinfo)

Comment 23

[Jens Stutte [:jstutte]]

Just to recap the state of this bug after this longish chat (sorry for the noise):

• In the last months nothing changed significantly (but we need to constantly update the signatures to catch all of them).
• There might be a significant, real drop with the release of FF 85 due to bug 1666214. Until then we will just continue to monitor numbers here, no concrete actions are planned until then.

Tracking the dependency to bug 1666214 also here, in order to remind us to check back.

Comment 24

[Andrew McCreight [:mccr8]]

I filed bug 1688249 about scrubbing at least some of the integers out of these signatures.

Comment 26

[Simon Giesecke [:sg] [he/him]]

Cleaning up signatures after Bug 1688249: The unstable number of the lambda expression is no longer part of the signatures, and the existing crash reports have been reprocessed (apart from 3 I still found, but these few can be ignored).

Comment 27

[Jan Varga [:janv]]

(In reply to Simon Giesecke [:sg] [he/him] from comment #26)

Cleaning up signatures after Bug 1688249: The unstable number of the lambda expression is no longer part of the signatures, and the existing crash reports have been reprocessed (apart from 3 I still found, but these few can be ignored).

Thanks, that is really useful, it was becoming really insane.

Comment 28

[Andrei Purice]

Hey Calixte,
Can you still reproduce this issue or should we close it?

Comment 29

[Calixte Denizet (:calixte)]

I filed the bug based on crash-stats data and mercurial history.
So I've no idea on how to reproduce it or not.

Comment 30

[Andrew McCreight [:mccr8]]

(In reply to Andrei Purice from comment #28)

Can you still reproduce this issue or should we close it?

If you look at crash stats, this is still happening in recent versions.

Comment 31

[Byron Campen [:bwc]]

I'm pretty sure I just saw this on try, if that is useful at all.

https://treeherder.mozilla.org/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=d742e72b4e1f3a6c9583801f45b180f5caf86670&selectedTaskRun=ONrAdW2YTWKgheCOy0l8kA.0

Comment 32

[Gabriele Svelto [:gsvelto]]

Support for inlined frames seems to have created a new signature.

Comment 33

[Jens Stutte [:jstutte]]

There were quite some moot signatures, too.

There are some recent reports on very old (unsupported) versions, I removed them from the list:

[@ mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::<T>::__invoke ]
[@ mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$::__invoke ]

Comment 34

[Gabriele Svelto [:gsvelto]]

On this topic I've asked Marco if they could tweak the bugzilla bots to automatically remove crash signatures that have no corresponding crashes anymore, that should help with this kind of long-running issues.

Comment 35

[Jens Stutte [:jstutte]]

(In reply to Gabriele Svelto [:gsvelto] from comment #34)

On this topic I've asked Marco if they could tweak the bugzilla bots to automatically remove crash signatures that have no corresponding crashes anymore, that should help with this kind of long-running issues.

Thanks, we might want to consider also to just not record crash reports for unsupported versions (like everything older than the major number of our oldest ESR version supported) ?

Comment 36

[Gabriele Svelto [:gsvelto]]

(In reply to Jens Stutte [:jstutte] from comment #35)

Thanks, we might want to consider also to just not record crash reports for unsupported versions (like everything older than the major number of our oldest ESR version supported) ?

We already do that but I don't know where the cut-off point is. I'll ask and report back.

Comment 37

[Gabriele Svelto [:gsvelto]]

FYI the cut-off point is 2 years, basically all crash reports coming from build IDs older than two years get dropped.

Comment 38

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to topcrash signatures, which match the following criteria:

• Top 5 desktop browser crashes on Mac on release
• Top 20 desktop browser crashes on release
• Top 20 desktop browser crashes on beta
• Top 5 desktop browser crashes on Windows on beta
• Top 5 desktop browser crashes on Windows on release

For more information, please visit auto_nag documentation.

Comment 40

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.

Comment 41

[Intermittent Failures Robot]

4 failures in 3766 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 1

Platform and build breakdown:

• windows10-64-2004-qr: 2
  • debug: 2
• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows10-32-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1592336&startday=2023-02-06&endday=2023-02-12&tree=all

Comment 42

[Yannis Juglaret [:yannis]]

I often get this crash as an intermittent failure on treeherder when running Windows gtest these last few days:
Hit MOZ_CRASH(Quota manager shutdown timed out) at /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:3361

Comment 43

[Jens Stutte [:jstutte]]

(In reply to Yannis Juglaret from comment #42)

I often get this crash as an intermittent failure on treeherder when running Windows gtest these last few days:
Hit MOZ_CRASH(Quota manager shutdown timed out) at /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:3361

You probably see bug 1815834. Would you mind to check if your tree contains already the patches from that bug and bug 1815999 ? Thanks!

Comment 44

[Yannis Juglaret [:yannis]]

Thanks [:jstutte]! My tree was pointing here, so it seems that I already had these patches:

$ hg log --keyword 1815834
changeset:   727571:1b64bebd3967
user:        Jari Jalkanen <[e-mail]>
date:        Thu Feb 09 10:57:39 2023 +0000
summary:     Bug 1815834 - Remove custom TestFileSystemQuotaClient test cleanup. r=dom-storage-reviewers,janv

$ hg log --keyword 1815999
changeset:   727690:4f32061b6a73
parent:      727681:83ff1b06372e
user:        Jan Varga <[e-mail]>
date:        Fri Feb 10 10:50:39 2023 +0000
summary:     Bug 1815999 - Fix FileSystemDataManager::GetOrCreateFileSystemManager to always resolve/reject returned MozPromise; r=dom-storage-reviewers,jari

I'll try with latest central nonetheless to confirm.

Comment 45

[Jens Stutte [:jstutte]]

(In reply to Yannis Juglaret from comment #44)

Thanks [:jstutte]! My tree was pointing here, so it seems that I already had these patches:

$ hg log --keyword 1815834
changeset:   727571:1b64bebd3967
user:        Jari Jalkanen <[e-mail]>
date:        Thu Feb 09 10:57:39 2023 +0000
summary:     Bug 1815834 - Remove custom TestFileSystemQuotaClient test cleanup. r=dom-storage-reviewers,janv

$ hg log --keyword 1815999
changeset:   727690:4f32061b6a73
parent:      727681:83ff1b06372e
user:        Jan Varga <[e-mail]>
date:        Fri Feb 10 10:50:39 2023 +0000
summary:     Bug 1815999 - Fix FileSystemDataManager::GetOrCreateFileSystemManager to always resolve/reject returned MozPromise; r=dom-storage-reviewers,jari

I'll try with latest central nonetheless to confirm.

Thanks, in case feel free to re-open bug 1815834 (or file a new one). This bug here is kind of a meta-bug.

Comment 46

[Yannis Juglaret [:yannis]]

It seems all green now with the latest mozilla central. I'll reopen that bug if I encounter the failures again. Thank you!

Comment 47

[Yannis Juglaret [:yannis]]

By the way, this seems to be the number 1 top crasher for Firefox release these days. Are we aware of it and working on this? Do we understand the underlying problem?

Comment 48

[Jan Varga [:janv]]

Yes, we are aware of it and we understand the problem.

Comment 49

[Intermittent Failures Robot]

1 failures in 3752 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1592336&startday=2023-02-27&endday=2023-03-05&tree=all

Comment 50

[Jens Stutte [:jstutte]]

Investigation should be kept in bug 1588498, moving the relevant signatures there.