1830145 - Hangs switching to unified folder inbox of 260k messages using folder mode settings, threads=All and `Search Online`. Also moving between folders in unified folder mode (example: from Inbox to Sent)

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect, P3)

Description

[[:Aureliano Buendía]]

STRs:

1. starts TB and switch to Unified Folders: there is a delay about 10"
2. set message threads on and set to All;
3. switch to All Folder view and toggle Unified Folders to off;
4. close TB
5. starts TB and switch to Unified Folders: there is a delay about 56"

Here profiling as permalink https://share.firefox.dev/40F0Y79

TB 113.0b4

Comment 1

[Thomas D. (:thomas8)]

Aurelio, is it better in Beta 113.0b5 (64-bit) where bug 1817367 has been fixed?

Comment 2

[[:Aureliano Buendía]]

Ciao Thomas, I don't know the answer, because until now I've tried bug #1817367 always with views-threads set to off. The observed result [with views-threads set to off], after the fix, is the one described in my comment https://bugzilla.mozilla.org/show_bug.cgi?id=1817367#c43
Trying (post fix) also with views-threads set to on, I encountered the performance problems described here.

Comment 3

[[:Aureliano Buendía]]

I can confirm this issue also in 114.0a1 (2023-05-04) (64-bit)

Comment 4

[[:Aureliano Buendía]]

The problem still persists in version 114.0beta3. Here the profiling result as permalink https://share.firefox.dev/42N6C9b
Related to nsThread::ProcessNextEvent ?

Comment 6

[Geoff Lankow (:darktrojan)]

How many messages are we talking about here? From the profile's screen shots it looks like your combined inbox has a 5-figure total, but it's hard to tell. I've been testing with a combined 70,000 messages and the busiest functions in your profile are not problematic for me.

Are you also hitting bug 1836393? It looks like you might be, and that could be related. I don't see that problem either.

I don't suppose you know if the same problem existed in 102, do you?

Comment 7

[[:Aureliano Buendía]]

(In reply to Geoff Lankow (:darktrojan) from comment #6)

How many messages are we talking about here?

Hi Geoff.
My 'Inbox on Unified Folders' has around 260k emails. This issue isn't a regression, because it behaves the same way in TB 102 as well.

Upon further investigation, I have verified that by selecting my 'Inbox on Unified Folders', there is high online traffic to my accounts' IMAP servers. Going to the 'Inbox on Unified Folders' virtual folder, I saw that the option 'Search Online (Gives up-to-date results for IMAP etc...)' is selected by default. By setting 'Search Online (Gives up-to-date results for IMAP etc...)' to false, I no longer have the performance problems mentioned here.

This ticket should be considered Invalid, as the performance of a search on an IMAP server depends on the IMAP server itself and not from TB?
For end users, it should be documented that in the 'Inbox on Unified Folders', the flag 'Search Online (Gives up-to-date results for IMAP etc...)'
is it true by default so there might be performance issues with large Inboxes?
Other ideas?

Thank you.

Comment 8

[Wayne Mery (:wsmwk)]

Aureliano, thanks for the update.

Dropping from supernova list based on comment 7

Comment 9

[Wayne Mery (:wsmwk)]

Aureliano, Does this happen with latest beta?

Comment 10

[[:Aureliano Buendía]]

(In reply to Wayne Mery (:wsmwk) from comment #9)

Aureliano, Does this happen with latest beta?

Unfortunately no, the problem is still here and makes the Unified Folders mode unusable in my case.
118.0b4 (64-bit) on Windows 11

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to [:Aureliano Buendía] from comment #10)

(In reply to Wayne Mery (:wsmwk) from comment #9)

Aureliano, Does this happen with latest beta?

Unfortunately no, the problem is still here and makes the Unified Folders mode unusable in my case.
118.0b4 (64-bit) on Windows 11

For every unified folder type?
Tried repairing any of the subfolders?

Comment 12

[Wayne Mery (:wsmwk)]

Using config editor try setting mail.db.max_open and mail.db.idle_limit to larger values. For example doubled.

Comment 13

[Wayne Mery (:wsmwk)]

Basically 100% of the activity is DOM related, specifically the stack is
nsMsgSearchDBView::AddHdrFromFolder
nsMsgSearchDBView::MoveThreadAt
nsMsgDBView::CollapseByIndex
nsMsgSearchDBView::RemoveRows
nsCOMArray_base::RemoveObjectAt
...

So changing mail.db.max_open or mail.db.idle_limit will not help here.

Comment 14

[Wayne Mery (:wsmwk)]

Aureliano, to what degree do you still see this problem?

Comment 15

[[:Aureliano Buendía]]

Hi Wayne, 'Unified Folders' continues to be unusable in production also fro TB 126.0b1 (64-bit) on Windows 11. Here the profiling result as permalink https://share.firefox.dev/3Uc3jFv

Comment 16

[Wayne Mery (:wsmwk)]

Is it still true that this only happen when you have Search Online enabled? (which is the default)

Two open bug reports cite Search online https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1098069%201757315&list_id=17004189

https://profiler.firefox.com/public/p2jpxd2k8wgwq7y6q135reg177eh9ff8zj77td0/calltree/?globalTrackOrder=2w501&hiddenLocalTracksByPid=40080-01~27476-02w7&range=24671m6003&thread=0&v=10 is interesting

Comment 17

[[:Aureliano Buendía]]

(In reply to Wayne Mery (:wsmwk) from comment #16)

Is it still true that this only happen when you have Search Online enabled? (which is the default)

Yes, it is.

Comment 18

[Wayne Mery (:wsmwk)]

Does it help to disable status bar at View > Toolbars ?

Comment 19

[[:Aureliano Buendía]]

No Wayne, I have the same problem even without status-bar: TB in that case is not usable.
TB 126.0b3 (64-bit) on Windows 11

Comment 20

[Wayne Mery (:wsmwk)]

Aureliano, does this still reproduce for you when using a current beta?

Comment 21

[[:Aureliano Buendía]]

Hi Wayne, still unusable. TB 133.0b5 (64-bit) on Windows 11
https://share.firefox.dev/3YX9cbO here the permalink for profiling the issue.

Comment 22

[Geoff Lankow (:darktrojan)]

I think I can virtually eliminate the big grey chunks in that profile. nsMsgDatabase::GetSearchResultsTable is called over and over again and I can't think of a reason why we'd expect a different result, so let's do it once and remember the result. I'm running all of our tests to find out if that breaks anything.

Comment 23

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1830145 - Cache search results tables and charsets. r=#thunderbird-back-end-reviewers

Comment 24

[Geoff Lankow (:darktrojan)]

Let's get this shipped. It fixes the larger part of the problem. I'm not sure what we can do about the other part which is the ordering of search results. Fortunately this problem will disappear when we get our new database.

Thanks for your patience Aureliano. Hopefully this will feature will be somewhat more bearable for you.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/b912eb902e6d
Cache search results tables and charsets. r=mkmelin

Comment 26

[Corey Bryant]

Geoff, ready to uplift this to beta?

Comment 27

[Geoff Lankow (:darktrojan)]

Comment on attachment 9439925 [details]
Bug 1830145 - Cache search results tables and charsets. r=#thunderbird-back-end-reviewers

[Approval Request Comment]
Regression caused by (bug #): always like this
User impact if declined: huge virtual folders could be very slow for no real reason
Testing completed (on c-c, etc.): it's been on central 6 days but there's not actually been a Daily build with it
Risk to taking this patch (and alternatives if risky): low

Comment 28

[Corey Bryant]

Comment on attachment 9439925 [details]
Bug 1830145 - Cache search results tables and charsets. r=#thunderbird-back-end-reviewers

[Triage Comment]
Approved for beta

Comment 29

[Rob Lemley [:rjl]]

Thunderbird 134.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/247c626de65b

Comment 30

[[:Aureliano Buendía]]

Hi Geoff, I did tests on 134.0b3 (64-bit) Windows 11 Pro: it's much better but it's still not usable in a production environment. Looking forward to releasing of new database structure. Thanks!

Comment 31

[Wayne Mery (:wsmwk)]

(In reply to Geoff Lankow (:darktrojan) from comment #24)

I'm not sure what we can do about the other part which is the ordering of search results. Fortunately this problem will disappear when we get our new database.

Thanks Geoff. What ordering are you referring to, something happening in mork? Or Gloda?

Good enough for now for 128?

Comment 32

[Wayne Mery (:wsmwk)]

Aureliano, thanks for that feedback.

Have a look at https://mzl.la/4gjtDa9 - are there any that at least partly describe what remains of problem?