1757315 - Window "not responding" for ~5 seconds clicking on 70k message Unified Folders inbox view when imap "search online" is used (advanced/saved search)

Status: UNCONFIRMED

(MailNews Core :: Networking: IMAP, defect)

Description

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0

Steps to reproduce:

I have four IMAP email accounts linked to my Thunderbird - two Dovecot, one Outlook.com, and one Gmail. Combined there are around 70.000 emails in my inboxes. I use the "unified folders" view to view all my email in my inboxes. This is sorted primarily by unread, and secondarily by date and time.

Actual results:

Whenever I am on a different page (folder) from the unified inbox view, and I click on the "Inbox", Thunderbird hangs for around ~5 seconds. During this time, CPU usage on one core maximises, and the Window becomes "not responding" (on both my Windows 10 x64 desktop and Windows 11 x64 laptop). I can see new email arriving during this time, but I cannot interact with the window. I can reliably reproduce this.

Expected results:

The window should remain responsive, and process whatever it is processing in the background instead of on the UI thread.

Comment 1

[Koen]

I have tried to figure out exactly what is causing this, but to no avail. Clicking on any of the individual inboxes is not a problem, and neither is clicking on the unified sent folder. Memory usage stays relatively stable during this time:

The error console shows no messages, and there are no add-ons installed. If there is something I can do to help debug this issue, please let me know.

Comment 2

[Wayne Mery (:wsmwk)]

I suspect you what you are seeing is related to Bug 1098069 - Constant Freezing for minutes at a time with “Searching Folder...” in status bar, busy spinner, and "(Not Responding)" in title bar for unified folder. Workaround unselect the checkbox "Search Online" in unified folder properties.

How long has this been happening?
Is it possible to reduce the total size of your inboxes?

Comment 3

[Koen]

If I turn off "search online", the window no longer freezes, but everything remains rather choppy (it feels like a refresh rate of 1 frame per second which impacts the entire GUI/window [1]) for around the same duration as it previously froze. CPU usage is also similar.

It has been happening since I started using Thunderbird, which is approximately 2 months ago. Reducing my inboxes is technically possible of course, but would require me to quite significantly change my workflow, so I would rather not.

[1] Here in GIF form. After around 6 seconds I switch to my unified inbox, after which it becomes choppy for 22 seconds: https://i.imgur.com/tC1zPlo.gifv

Comment 4

[Wayne Mery (:wsmwk)]

(In reply to Koen from comment #3)

If I turn off "search online", the window no longer freezes,

Excellent. So we know there is an imap issue, which is unfortuantely expected.

but everything remains rather choppy (it feels like a refresh rate of 1 frame per second which impacts the entire GUI/window [1]) for around the same duration as it previously froze. CPU usage is also similar.

Using unified folders will result in reduced performance due to the merging of folders.
And If many messages have the exact same subject(s), this could further magnify the problem.

Maybe check the total size of your Inbox.msf files - I would estimate it should not exceed 20-25MB total.
Unsubscribe from the gmail "All mail" folder.
Also make sure your AV software excludes the Thunderbird profile directory.
And hopefully both of your machines have at least 8gb of memory.

I have tried to figure out exactly what is causing this, but to no avail. Clicking on any of the individual inboxes is not a problem, and neither is clicking on the unified sent folder. Memory usage stays relatively stable during this time:

Thunderbird's memory usage is ...?

Comment 5

[Wayne Mery (:wsmwk)]

What is Thunderbird's memory usage ?

Comment 6

[Koen]

My apologies for the late reply, apparently my embedded image was filtered in the Markdown input.
Memory usage seemed/seems stable between 200 and 300 MB. The machines themselves have 16 GB and 32 GB of RAM.

Let me try to share the image again, this time in text form: https://i.imgur.com/cyjsiqs.png (same applies to the 64-bit version)

Comment 7

[Wayne Mery (:wsmwk)]

Does version 102 have any improvement to offer?

Comment 8

[Koen]

I updated my laptop to 102, my desktop has not received the update yet, but the behaviour remains the same as with 91. Thus no (noticeable) improvement.

Comment 9

[gene smith]

This is a response to the NI flag.
I did a fairly deep analysis of this issue back on bug 1611887 and concluded that the disable of online search was the only reasonable workaround and that fixing it fully was pretty hard.
But now with recent releases when I set up unified folders, the "Search Online" selection for Inbox is grayed out so not available as a setting. Don't know why.

One question for reporter from comment 0:

Whenever I am on a different page from the unified inbox view, and I click on the "Inbox", Thunderbird hangs for around ~5 seconds.

Not sure what you mean by a "different page". Do you mean a different application other than TB and come back to TB? Or do you mean a different folder in TB and then come back to Inbox and you see the 5 sec. hang?

Comment 10

[Koen]

Not sure what you mean by a "different page". Do you mean a different application other than TB and come back to TB? Or do you mean a different folder in TB and then come back to Inbox and you see the 5 sec. hang?

The latter. Whenever I switch folders within Thunderbird and then switch back to the unified inbox, it hangs for a couple seconds.

Comment 11

[gene smith]

Thanks for the clarification.
I was able to un-gray the "Search online" thing. I think it was because all my folders had no offline store and my Inbox search was just to "match all" so no reason to do a search on the server for anything. I had to un-select search on all the inboxes, restart tb and re-select them and, since I had at least one inbox with offline store, it enabled the checkbox. At least that's my theory.

I have 13 inboxes on imap and one on pop in my unified inbox test with a total of 52661 messages most of which are only stored on the server. I assume you are using offline store probably mbox format for imap inboxes and store message bodies locally.

When I switch from a normal folder to unified inbox, I also see a short "hang" while searching the inbox for gmail where I have at a lot of messages. I noticed this also in bug 1611887. In the status bar along the bottom when searching gmail I'm unable to scroll the messages for a a few seconds so I think this is the same "hang" you see. Searching smaller inboxes doesn't cause a hang so I can scroll the message list while searching. In the log I see a search for "undeleted" UIDs as I mentioned in bug 1611887. Having to process a very long list of about 40k message UIDs from gmail inbox seems to be where the bottleneck occurs.

If you just have the search setting for Inbox at "match all messages" and aren't really searching for anything specific, or if all your messages are all stored locally (the default), then I don't see a reason to do "Search Online". When I turn it off, I don't see the hang since the "search undeleted" never occurs so my large number of messages in gmail inbox doesn't cause a delay. (But this is probably just restating what you and Wayne have already discussed.)

Comment 12

[Wayne Mery (:wsmwk)]

Koen, do you still see this problem?

Comment 13

[Koen]

I can confirm that I still see this problem on 115.10.1 (64-bit) on Windows 10

Comment 14

[Wayne Mery (:wsmwk)]

Koen, does this reproduce with version 128.6.0 or newer?

Comment 15

[Koen]

Yes it still happens with 128.7.1esr

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Koen from comment #15)

Yes it still happens with 128.7.1esr

Thanks. Can you get a performance profile using version 140 esr? Instructions at https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance