291733 - Excessive disk fragmentation

Status: RESOLVED WONTFIX

(MailNews Core :: Database, defect)

Description

[Jon]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 Firefox/1.0.3
Build Identifier: Thunderbird version 1.0.2 (20050317)

My mail files rapidly gets fragmented.  I have Thunderbird automatically
filtering my mail into several folders, including several medium to high volume
mailing lists & automatically moving spam to the junk mail folder.

If I do not regularly defragment, then my folders end up with hunderds of
fragments each, and this does slow down Thunderbird.

My folder sizes are ~150Mb for Junk, 6Mb for Inbox, 80Mb for Subversion-dev. 
I'm using NTFS on WinXP with a consumer-grade SATA disk.

Workaround: Run a disk defragmenter often.

I get the impression that Thunderbird is growing each file on disk as each
individual e-mail is moved to that mail folder.  Could Thunderbird automatically
grow it's files in larger chunks (e.g. growing a 50Mb mail folder in 1Mb chunks)
to reduce fragmentation?


Reproducible: Always

Steps to Reproduce:

Comment 1

[Matthias Versen [:Matti]]

how should TB write to the mbox file in 1MB chunks in an standard mbox file if
only one single 1kb message is avialible ?

Comment 2

[Jo Hermans]

see the discussion in bug 251876 (which was WONTFIX'ed)

Comment 3

[Jon]

The discussion in bug 251876 is useful, but the last reason given for WONTFIXing
it was:

"I'd hate downloads to have a wrong size when mozilla crashes, making
any reliable resuming impossible. so I'll mark this WONTFIX."

I.e. that bug was talking about files downloaded from the web & saved to disk. 
But this bug is slightly different, in that it's about the mail files.

Regarding the comments about portability in that bug, I believe it's possible to
do this fairly portably.  From
http://www.opengroup.org/onlinepubs/007908799/xsh/fseek.html :

"The fseek() function allows the file-position indicator to be set beyond the
end of existing data in the file. If data is later written at this point,
subsequent reads of data in the gap will return bytes with the value 0 until
data is actually written into the gap."

So a call to fseek() followed by a one-byte fwrite() should be sufficient to
grow the file - and the OS should be clever enough to allocate all this space
contiguously.

I don't follow Mozilla development closely enough to understand the comment that
"The file APIs in Mozilla are frozen"???

Comment 4

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 5

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 6

[Ryan Flint [:rflint] (ping via IRC for reviews)]

*** Bug 343786 has been marked as a duplicate of this bug. ***

Comment 7

[Simon Bünzli]

Saving all e-mails as individual files instead of into one massive mbox might be the better (although maybe even more time-costly) solution to this problem. While this would neatly allow individual messages to be found by desktop search engines and get rid of the issue where whole mailboxes are quarantined, it might not happen anytime soon (if ever).

Comment 8

[Konstantin Svist]

(In reply to comment #7)

I disagree. In Windows, this will result in even harder-to-control fragmentation problem. NTFS tries to insert a small file into the first matching hole, so emails will essentially be all over the partition - this would cause a huge performance hit if you for instance want to rebuild the index of the mailbox

Comment 9

[Wayne Mery (:wsmwk)]

we're not going to entertain optimizing for OS, are we?
(and there are other bugs that address doing other mail storage models)

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

I think we'd consider optimizing for the OS in some cases.  As you say, there are other bugs to track alternate storage models, though.

Comment 11

[Wayne Mery (:wsmwk)]

For the size files cited in comment 0 (or even larger) I can't see how there would be a noticable impact to thunderbird performance on current systems. Even for bigger size files, I'd bet there are other more impactful factors to improve performance.

Plus, we are headed slowly in the direction of maildir, which is what comment 7 is about