Moving big local folders around is very slow -- should me a simple "mv"
Categories
(Thunderbird :: Folder and Message Lists, defect)
Tracking
(Not tracked)
People
(Reporter: alexandre.ferrieux, Unassigned)
References
Details
(Keywords: perf)
Steps to reproduce:
Just move a big local folder, which is nested under another, to another one.
Actual results:
The move is very slow.
As this is Linux, we can use strace:
It shows that a file copy (many write()s) actually occurs, not just a move (rename()), which would be instantaneous.
Expected results:
Just a pair of rename()s: one on the message file, and one on the msf.
|
||
Comment 1•1 year ago
•
|
||
A folder (file) move / rename cannot be used in this example, because there must be logical moves of messages with their metadata.
But there are two projects in progress that can help:
- rearchitecting message storage - hopefully later this year.
- improving write performance, through a set of issues related to Bug 1242042 - Enabling buffering for file stream to write message for C-C TB (I'm collecting local folder performance in bug 1621868)
|
Reporter | |
Comment 2•1 year ago
|
||
I have just done an experiment: making a snapshot of both XXX and XXX.msf, then moving then around in TB, then comparing.
Result:
- the msf file (metadata) is indeed changed (but only by appending something it seems... never mind)
- the mbox file (messages themselves) is untouched
As a consequence, it is possible to do a move on the mbox at least. This would be a huge kick, as it is several hundred times larger than the msf.
|
|
Reporter | |
Comment 3•1 year ago
|
||
Please note this report is about moving whole folders around, not individual messages. Therefore, conflating it with 1621868 misses the point.
|
|
||
Comment 4•1 year ago
•
|
||
Thanks for testing. I understand the use case and from your perspective I am conflating. But from my larger understanding of the code I am not. It's not possible to just "move" a folder without non-trivial changes of the current database code. The code of current architecture won't be changed to accommodate this idea, hence the dupe.
In other words when the rearchitected database happens this may be less of an issue. If the problem still exists when the new code is tested and the other open bugs such as bug 1242042 are not resolved, that means those things still need to be fixed - which should still help with your use case.
|
||
Comment 5•1 year ago
|
||
Yes - we could definitely shortcut the case where folders are moved, by directly moving files/dirs about (both for mbox and maildir).
The main issue at the moment is just the state of the message and folder copy/move code - it's fantastically complicated and hard to work with at the moment. Over the years so many fixes and hacks and shortcuts have been added in that it's really really hard to change something without breaking 10 other things.
Currently it's reimplemented separately for each different folder type (local/imap/news/rss/ews/etc...), where the code to do the local part of such operations should be shared between them.
It needs a big refactoring, but that'll be much simpler once the database is sorted out, and once interfaces to synchronise with the server-side operations are clarified (most protocols have similar server-side short-cuts for moving/renaming folders, at least within the same server).
So. It'll definitely happen but it's part of a much larger refactoring.
|
|
||
Comment 6•1 year ago
|
||
(In reply to Ben Campbell from comment #5)
So. It'll definitely happen but it's part of a much larger refactoring.
Are there blocker bugs that you have in mind? Bug 1714472?
|
|
||
Comment 7•1 year ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #6)
(In reply to Ben Campbell from comment #5)
So. It'll definitely happen but it's part of a much larger refactoring.
Are there blocker bugs that you have in mind? Bug 1714472?
Yes, and ideally it'd be after we've moved from per-folder databases to a globalDB, and post-EWS, when we start factoring out a lot of the protocol-specific folder code.
90% of what you need to do for folder copy/move is the same no matter which protocol you're dealing with. That last 10% is talking to the server to perform the remote operation.
So that 90% can (should!) be shared between all the folder types rather than the current ad-hoc re-implementations all over the place (with very dodgy/flawed error handling).
|
|
Reporter | |
Comment 8•1 year ago
|
||
Sorry to be insistent, but it looks like there's a confusion here. I created this ticket specifically about move (not copy) of whole folders (not subsets of messages) across local folder (no variety of folder types or protocols). Indeed this is a very important special case in terms of scaling, as typically users will have much more local disk space than per-account space on a mail server. In this special case, having per-folder databases is obviously the best in terms of performance, as it supports the equivalence between a "folder move" and simple inode juggling (with possible update of small metadata), by minimizing the filesystem load. Moving away from that, to a big global DB, sounds scary...
Description
•