1973165 - Large load times on first launch when trying to view messages from Inbox (and other UI blockage) while messages are downloading

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect, P1)

Description

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Attachment: Screenshot 2025-06-20 at 09.22.33.png

Affected versions

• 140.0(20250617192444)

Affected platforms

• macOS 15.5(MacBook PRO mid2019)

Steps to reproduce

1. Launch TB.
2. Login with valid credentials.
3. Go to your Inbox folder.
4. Select any email.

Expected result

• All emails are opened immediately with a very small load time.(if any)

Actual result

• Most emails have a very big loading time.

Notes

• This issue does not reproduce on the affected machine on other TB versions (latest 141.0a1, 128.11.1, 140.0b3)
• This issue does not reproduce on other OS's (Ubuntu/Windows) with the same affected account used to report this issue.
• Was not able to reproduce this issue on macOS 15.4.1 , nor on another machine running macOS 15.5(so this seems to be specific only to my device)
• The issue persists for ~10 minutes and after that it runs better (it could be because of the Downloading messages process?)
• Attached what is displayed in the Error Console when encountering this issue
• Screen Recording of the issue.

Comment 1

[Wayne Mery (:wsmwk)]

I have seen this while messages are downloading.

1. Does it happen when global indexing is disabled at Settings > Indexing?
2. Does it happen when "Keep messages" is disabled in Account Settings > Sync & Storage?
3. Does it happen with status bar disabled?

If none of these help, do https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Some potentially related issues from https://mzl.la/4k523Px:

• Bug 1941019 - Indexing messages gets stuck, Thunderbird becomes sluggish
• Bug 1937931 - UI randomly freezes up to 30 Bug 1935903 - Extremely slow to load messages and getting notifications (not responding) while downloading new mail or clicking emails
• Bug 1934197 - Jank with massive periods of CC after Event processing delay - Layout : Redraw / Graphics : DisplayList in Reflow about:3pane - even with status bar disabled

Comment 2

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

So I tried again with the newly released 140.0-build2(20250701221112)using macOS 15.5:

• Tried with global indexing to off
• With Keep Messages disabled
• With status bar disabled

Also tried using different combinations of the above on/off or all off, the issue reproduces the same way as it did in the description.
Important notice: I encounter this issue ONLY with this release/build. 140.0esr-build7 works good and the issue is not present.

Comment 3

[Eleanor Dicharry]

It's possible I was able to reproduce this (or something like it) on my Mac (MacOS Sequoia 15.5, 2025 Macbook Pro M4).

Connecting to an account with about 20k messages in the inbox, it takes noticeably longer for 140.0esr to load the inbox and display the number unread than it took 128.9.2esr. Viewing individual messages takes longer while the inbox folder is downloading. I also noticed that the count displayed next to the inbox in the tree view stops updating if I have the inbox folder selected during the initial download. If I deselect it, it continues downloading, and it looks like maybe it's either running slower or blocking while the inbox folder is selected.

It takes about 15 seconds for 128 to load that inbox and it takes about 100 seconds for 140 to load that inbox.

I saw the same behavior in 141b2, so I'm not sure if this is the same.

Comment 4

[Eleanor Dicharry]

I popped over to my Linux system (2022 12th Gen Intel, elementary OS 8) to check this there as well, and I'm seeing a similar performance difference there too for the initial download. With 140.0esr it takes about 2 minutes to download the inbox headers. With 128.12.0esr it's maybe 15 seconds.

Comment 5

[Eleanor Dicharry]

mozregression pointed me at these pushes: https://hg-edge.mozilla.org/comm-central/pushloghtml?fromchange=5294de3db27a3871a4f1bf5905d1ff48841cfac7&tochange=9ba85c3b79059081a384ca3a06841582d05323bc

I'm going through the revisions now to see if I can figure out which one caused this.

Comment 6

[Eleanor Dicharry]

Commit that appears to have caused the regression (identified through local backout after mozregression): https://hg-edge.mozilla.org/comm-central/rev/9ba85c3b79059081a384ca3a06841582d05323bc

Comment 7

[Eleanor Dicharry]

I think this is a race condition of some sort. During download of headers for the inbox folders I get a bunch of the following message to the console:

[Parent 21074, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8055000A: file /.../thunderbird-central/source/comm/mailnews/imap/src/nsAutoSyncState.cpp:678

At that location we're trying to acquire a semaphore.

Comment 8

[Eleanor Dicharry]

Attachment: Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

Comment 9

[Wayne Mery (:wsmwk)]

Makes sense that autosync is involved.

Just for historical reference, the pref was added 14 years ago in Bug 464126 - imap should load and show newest headers first, especially in the new client account with existing server account case - with the comment "only does chunking if the folder is open in a window. I also got rid of the monitor wait+notification for header download, which simplifies that code a bit." But the chunking code existed prior to that bug.

Comment 10

[Eleanor Dicharry]

Thanks for the context. I talked to Magnus about two alternative fixes for this on the backend chat channel, one being what I went with and the other being just increasing the batch size. With a batch size of 1000, we recoup most of the lost performance, but since the original pref was only applied if the user opened the folder in a window, most folders were not batching at all prior to the regressing change, so we decided to remove the pref and the batching. With the regressing change, all of the folders began batching and the roundtrip latency became far more prevalent with lots more competition for the queue.

Magnus, maybe you were already aware of this context (I see you in the bug Wayne referenced), but does this change your opinion on which path is best here?

Comment 11

[gene smith]

Note: Probably nothing here that is not already known. But I was curious as to what this bug is about so I did some testing and research. But maybe this info is helpful.

With tb shut down I delete INBOX and INBOX.msf. With 128.12esr when I start up, all the 5000 "new" message in inbox are scrolling by and I can't grab or open any of them until all the headers finish downloading. When it starts downloading the message bodies I can quickly open any message with very little delay.
In IMAP:4 log, I see that ALL the headers are getting fetched at once. This must be because Inbox is not getting detected as open in a window for some reason, so the initial fetch requests all the headers.

Now I do the same thing with 142.0a1 (July 4 daily). I get a quick burst of header downloads and scrolling and then it stops scrolling before remaining headers download. I can go to a message and select it (while header downloads still in progress) and messages open kind of slow (maybe about like reporter's screen capture video). When download of headers finishes and download full messages is in progress with 142.0a1 (and 128esr), the message opening time is very short, almost instant.
In IMAP:4 log, I see that 200 headers are getting fetched first (the newest 200) so this allows the code a chance to open and fetch some messages between the each 200 header chunk. Things are pretty busy so it's not real quick opening a message.

The inbox I tested above has about 5000 message, so not huge. Also, only tested on linux.

So based on what I see, the header download of new messages is what is slowing down the message opening, not the download of the message bodies (by autosync).

In the previous comments by the reporter and Eleanor above, I'm not sure how many new headers and message bodies are being downloaded at startup. In my test, it is all of the messages in Inbox (5000) since I started with deleted INBOX and INBOX.msf. Typically it would just be the number of messages that have arrived since TB was last run.

I applied the patch in comment 8 to my old daily build (May 22nd last update and not optimized) and see about the same behavior as for 128.12esr: keeps scrolling message while downloading all the headers and then can open messages fairly quickly while downloading bodies.
Not sure no chunking is a good idea since it requests all the headers in one giant imap fetch command which might be too big for some servers.

Finally, as mentioned here in bug 464126 comment 13, looks like the header download order is usually oldest first. I think that might be true since if "running in a window" is not detected, ALL the messages will be fetched in one big imap command and the oldest will be first. Not sure why Inbox is not seen as running in a window when users typically always have Inbox open at startup (as I did). Of course, if header chunking is done regardless of window being open (as code is now and "open in a window" is not checked), newer header chunks will be fetched first.

Comment 12

[Magnus Melin [:mkmelin]]

(In reply to gene smith from comment #11)

Not sure no chunking is a good idea since it requests all the headers in one giant imap fetch command which might be too big for some servers.

I'm not too worried, as then we'd have bug reports for the non-chunking cases, and I don't think we have.

The alternative would have been to use a very high chunking number like 1000 or more.
But then... that would really only happen for the very first folder initialization which only happen once (or so) during the profile lifetime, as one normally does not get thousands of messages each time Thunderbird gets opened. That initial load after account setup is not particularly joyful to begin with but I think it's fairly reasonable that everything's not snappy before the profile is somewhat in sync.

Comment 13

[Eleanor Dicharry]

I agree with Gene that it is nice to be able to look at messages while headers are downloading. For small downloads, it doesn't really matter, but if you do a first-time connection to an account with, say, more than 50k messages in a folder, the folder is completely unusable until the download is done. Since it seems like chunking wasn't actually happening in 128 esr anyway, the initial download is an edge case, and we never got a bug report about it, maybe it doesn't really matter to users.

For context, on my Gmail account with 20k messages in the inbox here are the timings:

Without chunking: ~15s
With chunking + chunk size 200: ~100s
With chunking + chunk size 1000: ~20s

The other sort of emergent behavior that happens with chunking is that it allows downloading headers for multiple folders concurrently. This didn't happen prior to the regressing change, but since the regressing change had the effect of enabling chunking across the board, it started happening. There is a sort algorithm that prioritizes the inbox, but loading the inbox doesn't completely block loading other folders, and all non-special folders receive the same priority, so if you have a big folder, it doesn't completely block folders that are equal or lower in priority.

I just wanted to point out a couple of observations from my investigations on this issue to point out that there is some UX benefit to chunking, at the cost of the sluggishness in loading (that can be somewhat mitigated by tuning the chunk size).

I know I submitted the patch to remove chunking, but given all of this, is that still something we want to do?

Comment 14

[Magnus Melin [:mkmelin]]

I'm open to instead increasing the chunk size if others prefer so.
Re downloading header for multiple folders concurrently, I'm not sure that's desired as it could eat bandwidth and processing at a bad time.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to Eleanor Dicharry from comment #13)

I agree with Gene that it is nice to be able to look at messages while headers are downloading. For small downloads, it doesn't really matter, but if you do a first-time connection to an account with, say, more than 50k messages in a folder, the folder is completely unusable until the download is done. Since it seems like chunking wasn't actually happening in 128 esr anyway, the initial download is an edge case, and we never got a bug report about it, maybe it doesn't really matter to users.
...

Thanks for the great analysis.

Lack of bug report is not always a good measure of impact - users will sometimes tolerate non-optimal behavior for a period of time. (Me included - this issue really bothered me but I had other priorities to attend to.) We should seek optimal behavior regardless of reports and perceived user impact and sentiment.

Also consider, "larger users" should as much as feasible be able enjoy the same good performance as smaller users - they are not always well represented in bug reports and I have no idea why.

I just wanted to point out a couple of observations from my investigations on this issue to point out that there is some UX benefit to chunking, at the cost of the sluggishness in loading (that can be somewhat mitigated by tuning the chunk size).

Do you have a suggestion? Or, are you implying there may be some type of user who will want to adjust a preference?

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin [:mkmelin] from comment #14)

Re downloading header for multiple folders concurrently, I'm not sure that's desired as it could eat bandwidth and processing at a bad time.

Can you elaborate?

Comment 17

[Magnus Melin [:mkmelin]]

If you just opened one folder that needs to be synced, you'd want all resources focused on making that one usable ASAP (and other folders later). Of course for true concurrency it would not matter.

Comment 18

[Eleanor Dicharry]

Do you have a suggestion? Or, are you implying there may be some type of user who will want to adjust a preference?

It's already a preference (mail.imap.hdr_chunk_size) that defaults to 200, which gives us the 4-5x slowdown we're seeing. Realistically, I don't see average users tuning this, or even understanding its effect, so if we're going to keep chunking in place, we should tune it for them. Using a chunk size of 1000, we can get it down to almost the same as without chunking, so I'm guessing with 500 or so we can convert it to a 2-3x slowdown rather than the 4-6x we're getting now.

I agree with Magnus' concern that it may lead to more downloads than a user wants, especially on a metered connection or something. We can control that, but this all happens right at startup and we're subscribed to all folders by default, so a user would have to unsubscribe quickly to avoid everything starting to download.

Another thought from an a11y perspective is whether watching a big scroll with thousands of messages (as we get without any chunking) by default which leads to a lot of quickly flashing text rolling by on the screen could be an issue for some users.

Since we're debating a tradeoff that really ends up being a user experience thing, should we get any of the UX folks involved in this to help decide on the best course of action? If so, we might want to pick a small short term fix for this that can get uplifted into 140esr and decide on a longer term approach to making this phase of starting up or connecting to an account more manageable, as it's part of the overall onboarding experience.

Comment 19

[Wayne Mery (:wsmwk)]

I should have said "the preference". Definitely an average user wouldn't have a clue what to do. I read too much into what you wrote.

STM whatever gets us close to the 128 in terms of responsiveness and scrolling should be good enough for ESR ?

As for metered connections - I shouldn't think IMAP users are a target audience. Users who want low latency and efficiency, if they know what they are doing, typically choose pop.

Comment 20

[Eleanor Dicharry]

Thanks, short term would be to remove chunking entirely then. At least in 128, nothing was being chunked anyway because the "currently open in window" check wasn't working. So removing chunking will provide the exact same UX we had in 128.

The patch to remove chunking is already approved (and attached to this issue). I'll submit a try run and get it landed.

Comment 21

[Wayne Mery (:wsmwk)]

So with chunking removed, the current unresponsiveness will be gone (or minimal).

Comment 22

[Eleanor Dicharry]

(In reply to Wayne Mery (:wsmwk) from comment #21)

So with chunking removed, the current unresponsiveness will be gone (or minimal).

Yes. (With the caveat that it's harder to click on anything during the initial big download, so it's hard to observe the originally reported sluggishness opening messages caused by the chunking, but the overall download is much faster, and that's the behavior in 128esr anyway)

Comment 23

[gene smith]

harder to click on anything during the initial big download

With the continuous scrolling, I was never able to open anything while headers were downloading with 128esr. I'd never seen this before and I've done a lot of imap testing with INBOX/INBOX.msf deleted, just not recently.

Edit: I went back to the last 102 release. What I see there when I start TB selected on the account (Inbox not in a window) and with INBOX and INBOX.msf deleted, an imap fetch of ALL the message headers occurs and there is no crazy scrolling. However, when I click on a message, it doesn't open until all the headers finish downloading.
If I do this again, this time with Inbox selected, headers are fetched in chunks of 200, no scrolling and I can open messages (with typical delay) while download is in progress.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/516474d7a87d
Remove header download chunking. r=mkmelin

Comment 26

[Wayne Mery (:wsmwk)]

duparchy of bug 1965407, likely duplicate of this bug, confirms his issue is gone.

Is patch good for uplift to esr140?

Comment 27

[Eleanor Dicharry]

Yes, this should be good to uplift to esr140.

Comment 28

[Eleanor Dicharry]

Comment on attachment 9499311 [details]
Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: This was discovered in the 140 esr release. This patch restores the message header load behavior as it was in 128 esr.
• User impact if declined: Users will wait longer for initial mailbox loads as multiple folders compete for limited network connections to load messages in batches. This also makes reading messages take longer as the read requests are competing with the header batches for limited network resources.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Daily?: Yes
• Has the fix been verified in Beta?: Yes
• Needs manual test from QA?: Yes
• If yes, steps to reproduce: 1. Create a new account, preferably to an account with a nontrivial number of messages in the inbox.

2. When the account first loads, it should load all of the messages in the inbox prior to loading any other folders.
3. Once inbox messages are loaded, clicking an individual message to view its content should be a fast operation.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The change restores the effective behavior of 128 esr because the batching was never happening in that release.
• Does the fix cause any migrations to be skipped?: No
• String changes made/needed: N/A

Comment 29

[Corey Bryant]

Comment on attachment 9499311 [details]
Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

[Triage Comment]
Approved for esr140

Comment 30

[Eleanor Dicharry]

Attachment: bug-1973165-esr140.patch

Comment 31

[Corey Bryant]

Comment on attachment 9516980 [details] [diff] [review]
bug-1973165-esr140.patch

[Triage Comment]
Approved for esr140

Comment 32

[Corey Bryant]

Comment on attachment 9499311 [details]
Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

[Triage Comment]
This patch version didn't apply to esr140

Comment 33

[Eleanor Dicharry]

(In reply to Corey Bryant from comment #32)

Comment on attachment 9499311 [details]
Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

[Triage Comment]
This patch version didn't apply to esr140

Just to be clear, that's a reference to the old patch you're talking about, right? The new patch I attached today is good?

Comment 34

[Corey Bryant]

(In reply to Eleanor Dicharry from comment #33)

(In reply to Corey Bryant from comment #32)

Comment on attachment 9499311 [details]
Bug 1973165 - Remove header download chunking. r=mkmelin,#thunderbird-back-end-reviewers

[Triage Comment]
This patch version didn't apply to esr140

Just to be clear, that's a reference to the old patch you're talking about, right? The new patch I attached today is good?

Yes, that's correct. Thanks for the help Eleanor!

Comment 35

[Stephen King (:sking)]

Thunderbird 140.3.1:
https://hg.mozilla.org/releases/comm-esr140/rev/92eb5225318ba6425cd177e8ffda37917fbb9444