1891416 - Supernova is superslow / unresponsive doing IMAP sync from row invalidate. Not status bar

Status: RESOLVED WORKSFORME

(Thunderbird :: Folder and Message Lists, defect, P2)

Description

[duparchy]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0

Steps to reproduce:

Have a large IMAP mailbox to sync (several thousands of emails). Click on it.

Actual results:

The sync is painfully slow and TB becomes unresponsive.
While the unresponsiveness is solved in TB 120+ Beta , the slow syncing is not.

Note that this seems to be related to screen rendering because i incidentally found a workaround : Click on another mailbox (small or empty) and then Syncing of the previous mailbox is done in a flash.

Comment 1

[Wayne Mery (:wsmwk)]

Please start Windows in safe mode with networking enabled - https://support.microsoft.com/en-us/help/12376/windows-10-start-your-pc-in-safe-mode

Still In Windows safe mode, start Thunderbird in Troubleshoot Mode - https://support.mozilla.org/en-US/kb/troubleshoot-mode-thunderbird

If problem does not reduce, please obtain a performance profile https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Comment 2

[duparchy]

As written, I've found a workaround and gave you a clue : Display rendering.

i.e changing the focus to another (empty) mailbox and then the synchronization of the large mailbox, which is done in the background, is extremely fast.

This problem seemingly related to display rendering is maybe worsened when using Virtual Machines w/o dedicated GPU (which the most common). Display rendering is not as fast as dedicated computer.

Comment 3

[Wayne Mery (:wsmwk)]

In that case, please obtain a performance profile https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Comment 4

[duparchy]

I have uploaded a performance profile
https://share.firefox.dev/447OAjy

I synced a mailbox w/ 48000 emails. Headers only.

I kept the focus on that mailbox for a minute or two, up to 2000 message sync. Rather slow.
Then I changed the focus to the Draft mailbox and the remaining 46000 messages synced extremely fast.

Comment 5

[duparchy]

When the focus is on the syncing folder, depending on the sort order the speed is higher.
i.e. when nothing change on the pane the speed is correct. when synced emails are constantly updated on the screen the sync is extremely slow.

Syncing speed is tied to screen rendering which is probably not a good design. We're not watching a movie...;-)

Comment 6

[Wayne Mery (:wsmwk)]

Does it help to disable status bar at View > Toolbars ?

Comment 7

[duparchy]

Hi,
no.

Comment 8

[Wayne Mery (:wsmwk)]

To recap - faster when the message list is not displayed for a folder which is getting it's initial download sync of messages.

Questions:

1. Were you moving the mouse during that period? (other than to start and disable the profiling)
2. You mention "When the focus is on the syncing folder, depending on the sort order the speed is higher." Which sort order is slower?
3. Can you do another profile with 127 beta? We have a few fixes https://mzl.la/4dKv9ld since your profile with version 125 beta.

Comment 9

[duparchy]

Hi,

your recap is correct

1 - Moving the mouse during sync has no effect wheresoever.
2 - If the Date sorting does not produce change in the GUI, it is faster (but still very slow). My last quick test w/ 127.0b2 is not significant on that matter which is response to 3 question
3 - I confirm that the problem is exactly the same 127.0b2 beta.

(BTW I'll probably post another slowness bug. Recently the "quick-filter" is not quick anymore...Typing in there cause Thunderbird to freeze for a few seconds)

Comment 10

[duparchy]

Please find the Profile :https://share.firefox.dev/4dPUCK4
I synced an Inbox mailbox w/ 41660 emails

Synced the first ~3000 emails with focus on Inbox and Date sorting causing email list to constantly change. (sync extremely slow)
Synced ~3000 to ~10000 by changing the Date sorting so the email list does not change (sync slightly faster)
Sync ~10000 to ~30000 with focus on another mailbox (sync reasonably fast)
Sync ~30000 to 41660 focus back on Inbox and Date sorting causing email list to constantly change (sync extremely slow again).

Comment 11

[Wayne Mery (:wsmwk)]

Thanks duparchy.

Comment 12

[Wayne Mery (:wsmwk)]

I did a "Joy of Profiling" session with Florian yesterday on the first profile from comment 4 and we came up with some juicy issues. He writes" it's actually a really good profile to learn, there are lots of interesting markers, different time ranges in the timeline showing different high CPU use patterns that are interesting to look at in the flame graph, etc..."

julienw later documented the most severe issue in matrix as follows

from the stackchart, when zooming in, we clearly see that rowCountChanged is called as the result of nsImapMailDatabase::AddNewHdrToDB;
and this produces a lot of consequences to adjust the scroll, and especially _ensureVisibleRowsAreDisplayed that get scrollTop that triggers
a layout each time. I can think of a few fixes.

1. The most important: batch the changes so that new headers are added only once every second (for example).
2. make the action of "pin to bottom" itself more effective, with the help of overflow-anchor => https://css-tricks.com/books/greatest-css-tricks/pin-scrolling-to-bottom/. Although from my test it still does a layout approximately the same duration, so definitely not a silver bullet.

You can see the layout flushes (which should be avoided) in a subset of the profile https://share.firefox.dev/4bpCCEz - in the marker chart scroll down to the "Layout" section.

Also, a side effect is lots of CC as seen in https://share.firefox.dev/4aw98no

Another issue, less severe but a major power drain, is thousands of calls to prefs. The suggestion is to cache prefs (I think it was for tags prefs).

Comment 13

[Wayne Mery (:wsmwk)]

Other bug reports:

• Folder component bugs may have one or two matching bug reports https://mzl.la/4dRqQo4. For example Bug 1858030 - Message list flickers and fills slowly when scrolled or via mouse wheel
• Bugs in other components https://mzl.la/4bqNuCe (most of which do not yet have profiles) probably also has some matches. For example Bug 1869260 - Massive jank in Daily in the main thread, due to nsImapMailFolder::InitiateAutoSync (with server side folders)

p.s. you can see the hits on rowcount in the stack chart of https://share.firefox.dev/3WO3iKI

Comment 14

[duparchy]

Please find another Performance Profile, related to Quick Filter (which became slow only recently).

I applied a filter on a 10K emails mailbox and then removed it.

https://share.firefox.dev/4bPKBuB

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to duparchy from comment #14)

Please find another Performance Profile, related to Quick Filter (which became slow only recently).

I applied a filter on a 10K emails mailbox and then removed it.

https://share.firefox.dev/4bPKBuB

Thanks. See bug 1893571

Comment 16

[Wayne Mery (:wsmwk)]

Next week, please try version 128

Comment 17

[duparchy]

I tried 129.0b3

Looks like there is some improvement. From the status bar I estimate a headers sync rate at ~1200 headers/s following a check-mail on a large mailbox (50K emails) . Still not blazing fast, but acceptable. We "see" that the emails list panel is not constantly refreshed anymore during headers .

Comment 18

[Wayne Mery (:wsmwk)]

Thanks for the update. Please reopen if you see unacceptable behavior

Comment 19

[duparchy]

Sorry for being an annoyance but I reverted from the Beta channel to the regular, now ESR, channel.

128.2.3 is still painfully slow.