gui freeze while moving large amounts of email messages (mbox, Local Folders, offline)
Categories
(MailNews Core :: Database, defect)
Tracking
(Not tracked)
People
(Reporter: bm_witness, Unassigned)
Details
(Keywords: dupeme, perf, Whiteboard: [bulkoperations])
User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
Steps to reproduce:
Folder with 100+k messages in it, move them from one to another.
This is using the mbox formatted Local Folders only; no remote services involved.
I've also put TB into offline mode to prevent downloading of new emails (though that doesn't seem to effect the calendar syncing).
Background: I have been using TB for years, and have some high volume mailing list data and other information (e.g GitHub changes) from years of collection. I had an issue where TB duplicated the folders and I am deduping them with folders ranging from a few hundred to 10's of thousands of emails (40k-70k not uncommon; a few have 100+k).
Actual results:
TB goes through the steps of moving the messages from the source to destination. When the progress is within a few hundred of the total it seems to go off to do some cleanup activities on the source or reindexing/compaction in the destination (not sure as there's no info). For small folders (<10k) this can cause the TB GUI to hang for a few seconds to minutes - you can't scroll or otherwise use TB, but it returns relatively shortly. For larger folders (>40k) this can take longer and longer, with >70k messages taking an hour or more. 100+k messages seems to take several hours.
Initially I would end up killing TB; but then I discovered that it would eventually recover if I left it long enough (e.g overnight).
While I am running on Linux (PopOS! 22.04, 32GB RAM, AMD Ryzen9 proc) I don't expect this would be any different on other platforms. I'm still using the mbox formatted folders. All email is
Expected results:
Indexing, cleanup, compaction functionality would happen in the background and not impact the GUI interface, perhaps displaying a message on the selected folder saying what TB is doing if needs to wait for one of those background activities to complete.
|
Reporter | |
Updated•2 years ago
|
|
||
Comment 1•2 years ago
|
||
I do believe that this is happening, and one day we need to bite the bullet and improve the backend even though this may not be everyone's usecase, but it shouldn't have this type of poor UX. Confirming to keep on the radar, but quite sure that this is an old problem which has been reported before.
|
|
||
Updated•2 years ago
|
|
||
Comment 2•2 years ago
•
|
||
This is using the mbox formatted Local Folders only; no remote services involved.
But is the source folder not an imap folder?
If it is an imap folder, there is still significant overhead beyond simply storing the messages to the local folder, despite setting Thunderbird offline:
- Thunderbird still collects information of every message for a potential "undo" of the move BY THE USER
- Thunderbird collects information about every message for replAy to the imap server when Thunderbird goes onlinE
- Both of the above can result in high memory usage
I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdx
|
|
||
Comment 3•2 years ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #2)
...
I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdx
And even if imap is not involved, still likely a duplicate. Same list of candidates.
|
|
Reporter | |
Comment 5•2 years ago
•
|
||
(In reply to Wayne Mery (:wsmwk) from comment #3)
(In reply to Wayne Mery (:wsmwk) from comment #2)
...
I agree with Thomas, this is likely a duplicate, and therefore not likely a new bug. Some possible candidates in https://mzl.la/3GGGEdxAnd even if imap is not involved, still likely a duplicate. Same list of candidates.
no, IMAP isn't involved in this. I'm moving data from one folder to another both under the "Local Folders".
Background: I had something happen where somehow my "Inbox" in my "Local Folders" got recursively copied into a sub-folder of the Inbox 18 times before I killed TB because of how long it was taking. I'm cleaning it up and deduping my email but due to the size (normally 20 GB, after this it exploded to 330GB) I have to copy messages up a level, dedup, and repeat. Many of those folders have 10's of thousands of emails in them, with some over 100k messages.
Background #2: I do have IMAP accounts configured; but my rules move from the IMAP inbox to my Local Folder Inbox (one per account), then I apply local rules to move them to appropriate folders. However, that doesn't effect this performance bug as I'm entirely operating offline (no new incoming messages) and only moving between folders in Local Folders.
Reading over the list, most of those are IMAP related in a way that doesn't impact this outside of any backend stuff for storing indexes locally; the following seem related though:
Description
•