225280 - Mail messages are only really deleted when the folder is compacted

Status: RESOLVED INVALID

(MailNews Core :: Backend, defect)

Description

[Richard Neill]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007

Messages are never actually deleted from folders until the time that the folder
is compacted. They are merely not displayed.

Reproducible: Always

Steps to Reproduce:
1. Get a message in inbox
2. Delete it
3. Click trash folder, delete the message from there.
4. Now grep through the actual files corresponding to Inbox and Trash
5. Surprise! It's still there.


Expected Results:  
I think that Mozilla's strategy of not "defragging" the Mail Folder files is
actually quite a good one from a point of view of efficiency. But I did get
rather a shock. And I reclaimed 400 MB of wasted disk space.

There are 5 ways to treat this:

1)As a security bug, which should be fixed, and which has security implications
for users who believe that "delete" really means "delete" (at least so far as
the OS will allow it, especially if they then use secure-deletion tools to wipe
unused disk sectors)

2)As a really neat way to undelete emails that one deleted in haste.

3)As a documentation bug: in Edit->Prefs->Mail->Offline & Disk Space, the
compact folder option should be set below 50kB, with a warning that if this is
not true, deleted messages are unlikely to expire. 

4)As a performance bug: If mail folders are not compacted, then they can grow to
hundreds of MBytes. This wastes disk space and slows down performance. It also
means that deleting a corrupted .msf file will cause many deleted emails to
return from the dead. At a minimum, I'd suggest checking if a folder wastes more
than 5 MB, (regardless of the users pref in Offline and Disk Space) and popping
up a warning dialog.

5)It also suggests to me a single Tools->Cleanup menu that would give options
with checkboxes to clear {cache,history,bookmarks,...,compact email folders}

(I'm now using 1.5; this problem has been building up for me since the days of 0.8x)

Comment 1

[Matthias Versen [:Matti]]

yes, deleted messages are only deleted in the file itself if you compact the
folder. 
You can't avoid that without changing the used mail databse from mbox to single
files. Such a change is wontfix, deleteing them directly if you delete the
message is a big performance problem -> wontfix

There are enough other bugs where you can find this informations. Please search
the next time before you file a bug - thanks

1) Your mails are stored in the users home directory and that means that your
mail files should be secure and there should be also a note in the Mozilla help.

3) That preferences is not the documentation. The default value should be raised
to 1MB and it should be enabled by default (IMHO and we have a bug for enabling
this feature as default)

4) it's a performance bug if you would compact your mail fodler for every
deleted mail


This is no bug -> invalid

Comment 2

[Richard Neill]

I'm sorry, I disagree with you.

There should be a way to delete messages permanently, and it should be clearly
labeled "delete messages" Users expect that when the trashcan is emptied, the
messages are gone for good - this is certainly behaviour in *all* other
applications. Compacting the folder is a non-obvious way to do it! So if delete
doesn't actually mean delete, then there should be a menu option like "really
delete".

I don't agree that because mail is in the users home directory that it is
secure. They are not secure from root, they are not secure from a subsequent
user (of the same account/profile), they are not secure from someone walking
past an unattended terminal, and on Windows, they are never secure.  It is
important that a deleted email is really deleted. (yes, I know about data recovery.)

Also, its not enough just to turn on the preference for compacting files by
default unless the preferences dialog explains that the consequences of not
doing it include old emails "coming back to life." 

Also, what about rebuilding .msf files? 
What about the performance issue of wasting 400 MBytes of disk space?

I do agree that the mbox format is a good one, and that folders should not be
compacted too often. Also, I agree that documentation should be in the "help"
section, except that people actually only read the help section when they have
trouble. Most people learn by reading what they see in the menus and dialog
boxes. So it is important to explain things there too.

As for searching, I did search for "delete compact" - this ought to have yielded
other bugs if there were any. I am sorry.

I do think this bug is important, even if only as a documentation bug.
Personally, I have used moz mail for several hours per day for 2 years - and I
didn't know about this til today - which suggests that there is something not clear!

Comment 3

[Matthias Versen [:Matti]]

see all the dupes of bug 183837 for example.

You opened a bug with 
"messages are only really deleted when the folder is compacted" in Mailnews/Mail
Backend

This is no bug and that's the reason why this is invalid.
If you want to wait x minutes every time you delete a mail you can set
autocompacting to 1kb (there are a few known issues with it..).

You can file a new bug if the compacting function isn't in the help. (Use
Browser/Help or Product Documentation)
The preferences are the wrong place to explain a function and the default
setting is another bug (but I think there are a few problems with compacting
that shpould be resolved first)

The performance issue of wasting 400MB is a user error and/or a dupe of the
default enabled auto-compacting bug)

Comment 4

[WADA:World Anti-bad-Duping Agency]

Richard, see Bug 58308 : support qmail's maildir format.
This is the enhancement request for "changing the used mail databse from mbox to
single files".