1808614 - gui freeze while moving large amounts of email messages (mbox, Local Folders only, offline)

Status: RESOLVED DUPLICATE of bug 1621868

(MailNews Core :: Database, defect)

Description

[bm_witness]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

Steps to reproduce:

Folder with 100+k messages in it, move them from one to another.
This is using the mbox formatted Local Folders only; no remote services involved.
I've also put TB into offline mode to prevent downloading of new emails (though that doesn't seem to effect the calendar syncing).

Background: I have been using TB for years, and have some high volume mailing list data and other information (e.g GitHub changes) from years of collection. I had an issue where TB duplicated the folders and I am deduping them with folders ranging from a few hundred to 10's of thousands of emails (40k-70k not uncommon; a few have 100+k).

Actual results:

TB goes through the steps of moving the messages from the source to destination. When the progress is within a few hundred of the total it seems to go off to do some cleanup activities on the source or reindexing/compaction in the destination (not sure as there's no info). For small folders (<10k) this can cause the TB GUI to hang for a few seconds to minutes - you can't scroll or otherwise use TB, but it returns relatively shortly. For larger folders (>40k) this can take longer and longer, with >70k messages taking an hour or more. 100+k messages seems to take several hours.

Initially I would end up killing TB; but then I discovered that it would eventually recover if I left it long enough (e.g overnight).

While I am running on Linux (PopOS! 22.04, 32GB RAM, AMD Ryzen9 proc) I don't expect this would be any different on other platforms. I'm still using the mbox formatted folders. All email is

Expected results:

Indexing, cleanup, compaction functionality would happen in the background and not impact the GUI interface, perhaps displaying a message on the selected folder saying what TB is doing if needs to wait for one of those background activities to complete.

Comment 1

[Thomas D. (:thomas8)]

I do believe that this is happening, and one day we need to bite the bullet and improve the backend even though this may not be everyone's usecase, but it shouldn't have this type of poor UX. Confirming to keep on the radar, but quite sure that this is an old problem which has been reported before.

Comment 2

[Wayne Mery (:wsmwk)]

This is using the mbox formatted Local Folders only; no remote services involved.

But is the source folder not an imap folder?

If it is an imap folder, there is still significant overhead beyond simply storing the messages to the local folder, despite setting Thunderbird offline:

• Thunderbird still collects information of every message for a potential "undo" of the move BY THE USER
• Thunderbird collects information about every message for replAy to the imap server when Thunderbird goes onlinE
• Both of the above can result in high memory usage

I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdx

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #2)

...
I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdx

And even if imap is not involved, still likely a duplicate. Same list of candidates.

Comment 4

[Wayne Mery (:wsmwk)]

(corrected some missing letters and words in comment 2)

Comment 5

[bm_witness]

(In reply to Wayne Mery (:wsmwk) from comment #3)

(In reply to Wayne Mery (:wsmwk) from comment #2)

...
I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdx

And even if imap is not involved, still likely a duplicate. Same list of candidates.

no, IMAP isn't involved in this. I'm moving data from one folder to another both under the "Local Folders".

Background: I had something happen where somehow my "Inbox" in my "Local Folders" got recursively copied into a sub-folder of the Inbox 18 times before I killed TB because of how long it was taking. I'm cleaning it up and deduping my email but due to the size (normally 20 GB, after this it exploded to 330GB) I have to copy messages up a level, dedup, and repeat. Many of those folders have 10's of thousands of emails in them, with some over 100k messages.

Background #2: I do have IMAP accounts configured; but my rules move from the IMAP inbox to my Local Folder Inbox (one per account), then I apply local rules to move them to appropriate folders. However, that doesn't effect this performance bug as I'm entirely operating offline (no new incoming messages) and only moving between folders in Local Folders.

Reading over the list, most of those are IMAP related in a way that doesn't impact this outside of any backend stuff for storing indexes locally; the following seem related though:

• bug 722183
• bug 1621868

Comment 6

[Wayne Mery (:wsmwk)]

bm_witness, do you still experience this when using version 128?

Comment 8

[bm_witness]

(In reply to Wayne Mery (:wsmwk) from comment #6)

bm_witness, do you still experience this when using version 128?

Apologies for the delay - I saw this but not when I was able to verify anything and it skipped my mind for a bit.

it's been a long time since I've tried, and I have such a massive task that I've been working on an alternate tool to fix my inbox.
That said, I just tried it - moving 700+ messages, and it still did hang up a little before reporting an error to me about being unable to write to something, wasn't sure which side it meant.

It wasn't as bad as it was before; I think the initial copy phase was fine but seemed like it hung more during the clean up of the origin folder.

I have one folder that has 119k+ unread emails (probably at least that much already marked as read). I clicked on it, it certainly needs to regen the cache, but it seems to be handling it okay. Certainly taking a long time to generate the cache, but that's expected. I could probably make a tarball of this one folder to provide for testing if I had somewhere to share it - it's essentially an archive of one of the OpenOffice Dev mailing lists, only with a ton of duplicate messages in it. (I'm slowly deduplicating my email, but have been using webmail interfaces for a while due to the issue since its hard to do it under TB.) After several hours it still hasn't loaded the new page, but the GUI is still responsive - the events dialog will come back if I dismiss it (I'm in offline mode; but it still comes back), and the GUI didn't freeze over and go white like it used to. So I'll call it progress.

NOTE: I moved my Laptop from PopOS to Kubuntu 24.10 so a little more up-to-date; I still need to do the 25.04 update.

Comment 9

[Wayne Mery (:wsmwk)]

Always a good idea to screen shot errors. :)

Do you see the error mentioned in https://mzl.la/4kWcQwB? Or remember more exactly than "unable to write"?