Closed Bug 101273 Opened 18 years ago Closed Last year

Alternative to mbox storage format to avoid "compact folders" and speed up message processing

Categories

(MailNews Core :: Backend, enhancement)

x86
Linux
enhancement
Not set

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 845952

People

(Reporter: dahechler, Assigned: mscott)

Details

(1) Each folder be a separate directory.
(2) With each folder have a status file that is nothing more than a list of
message "names"
(3) Each message comes in and is stored as a separate file.

                 Inbox
   ----------------------------
    |           |      |     |
status file   msg1   msg2   msgX

So, for the folder "Inbox"
(1) create a directory "Inbox"
(2) create a "index file" in the Inbox directory
---------------------------------
(1) As each email come in, store it in it's own file within the Inbox directory
(2) update the index file with the email name by appending the email name at the
end of the list
------------------------------
(1) As each email is deleted or moved, delete the email file from the Inbox
directory
(2) Delete the email name from the index file and delete the empty slot
(compressing the file)

Savings:
(1) This will completely eliminate the need to "compact folders"
(2) Since each email is it's own file, it makes it much easier to move the email
around from folder to folder and delete.
Not sure why this was filed under editor core ... over to MailNews
Assignee: kin → mscott
Component: Editor: Core → Mail Back End
Product: Browser → MailNews
QA Contact: sujay → esther
Or at least offer the option to TURN OFF comapct folder. I'm using NTFS
filesystem comression and that is sufficient enough for me.
Reporter : storing each mail as an individual file is a very bad idea when there
are more than a few hundred files involved, especially if the files are small.

For instance : my 20GB Fat32 hard drive has 16kb allocation units.  This is
quite normal.  Every file will occupy a multiple of 16kb on the disk.  A 29kb
file will occupy 32kb, a 33kb file will occupy 48kb, etc.  More importantly, a
1kb file (like, say, an e-mail) will occupy 16 kb.  2000 2kb files will occupy
approx 32000kb.  32 Megabytes of disk space to store 4 Megabytes of data.  This
is not very efficient.  

This can be a real issue with newsgroups, where it is not unusual to have
several thousand messages on disk at once for some groups.

In addition, once you get over several thousand files in one directory, the OS
itself will start to creak.  Have you ever tried to browse a directory with
several hundred thousand files in it?  I have.  The OS shells involved (WinNT4
and Win2k) didn't like it - the systems hung.  They might have started moving
again once they'd finished processing the file list, but after 20 minutes I gave
up waiting.  It's not a good idea to leave that kind of trap lying around for
users to blunder into.

Marking WONTFIX
Status: UNCONFIRMED → RESOLVED
Closed: 18 years ago
Resolution: --- → WONTFIX
Oh well, it sounded good to me since the mainframe world has been doing it for
over 20 years (before PC's).  At McDonnell Douglas, we had the whole campus on
mainframe email (PROFS) which handled 500,000 employees !!
Juicy tidbit of info:  Meijer (http://www.meijer.com) still uses PROFS for their
large grocery team's communications system.  Adding David, but I think this will
just get verified as won'tfix.
yeah, won't fix - we use berkeley mailbox format because it's a standard, and
the benefits outweigh the drawbacks. And switching would be an large undertaking.
verified.
Status: RESOLVED → VERIFIED
*** Bug 133172 has been marked as a duplicate of this bug. ***
Product: MailNews → Core
Product: Core → MailNews Core
This would be solved by bug 845952
Status: VERIFIED → RESOLVED
Closed: 18 years agoLast year
Resolution: WONTFIX → DUPLICATE
Summary: Eliminate the need to "compact folders" and speed up message processing → Alternative to mbox storage format to avoid "compact folders" and speed up message processing
Duplicate of bug: maildirblockers
You need to log in before you can comment on or make changes to this bug.