Closed Bug 122373 Opened 23 years ago Closed 22 years ago

inbox & trash folders corrupted during delete of multiple mails

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: mozilla, Assigned: naving)

References

Details

(Keywords: dataloss, regression)

Attachments

(1 file)

proposed fix
1.10 KB, patch
cavin
: review+
Bienvenu
: superreview+
Details | Diff | Splinter Review 
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.7) Gecko/20011221
BuildID:    2001122106

Selected multiple spam emails in my inbox using ctrl-click.  Hit the "del" key.
Noticed that they did not disappear.
Chose "empty trash" - noticed that the trash folder disappeared form
the list of folders in the "dbharsh on bellsouth" group.  Using "explorer" I
verified that the "Trash" file was still present in the file system.

Status at this point: still had mulitple e-mails selected in the inbox folder. 
Under "local folders" there was still a "trash" folder.  I dragged the
highlighted e-mails toward that folder and released.  Nothing obvious happened.

I selected "compact folders" and some of the e-mails disappeared.

I had some good e-mails still in the inbox, which now could not be read (click
on them & get nothing in the content pane; or else get the body of one of the
trash e-mails for the wrong header line).

I closed Mozilla and attempted to rename the .msf files associated with Inbox
and Trash (there wasn't one for trash?).  Restarted, and this appeared to clear
the inbox since I was left with only the "good" emails' titles.  I moved a few
of these to their appropriate folders, then selected "compact" again - whoa! 
Some of the spam which had been deleted and emptied REAPPEARED in the Inbox.  I
moved them to trash, and once again, they would not move from the inbox.  I
closed mozilla & renamed the complete folders "inbox" and "trash" and their .msf
files to things like inbox_corrupt or t_backup, etc.  Started mozilla again with
a newly rebuilt Inbox, and am still seeing problems.

Reproducible: Didn't try
Steps to Reproduce:
1. Still present... 
2.
3.

Actual Results:  See detailed description

Expected Results:  Simply moved emails between folders.
looks like a problem with multiple delete.
Assignee: bienvenu → naving
Status: UNCONFIRMED → NEW
Ever confirmed: true
QA Contact: esther → sheelar
Just a note this was before the multiple delete optimizations were checked in. 
We have had other bugs about messages reappearing/coming back but we haven't
been able to reproduce them. sheela may also have tried to reproduce this. 
sorry, I didn't mean the view optimizations were responsible, just that you've
worked on multiple move/copy/delete issues before. One thing to look for with
messages coming back is if they don't have x-mozilla-status lines in the
messages, often because the original folders were created by importing from
eudora or OE. Also, losing the trash folder after empty trash might be related.
There are two possibilities - we lost the trash flag, so the trash folder was
there but just not in it's normal location in the folder list, or somehow the
trash folder was completely missing after empty trash. Also, I bet when the user
tried to drag messages to the local folder trash, we were trying to do a copy,
because it was across servers, which is why nothing obvious happened (assuming
obvious means the original messages should have been deleted).
I'm assuming this is a pop user, not imap.

That was a note to myself when I work on this bug. I have seen "trash" disappear
on cindy's machine but it has been elusive to reproduce. There have been recent
fixes where trash after empty trash moved up/down in the folder-pane.
Reporter is also using an older build from 2001-12-21-06. Navin fixed the empty
trash folder bug getting misplaced around 2002-01-16. 

Reporter,
Can you tell us based on Bienvenu's comments if you have renamed, copied or
moved folders from a different profile OR if you had imported folders through a
different mail application.
Original folders were imported from Netscape 4.7?.  
The trash folder was still present when viewing using Explorer; however, it was
not listed in the Mozilla pane.
When I said "nothing obvious happened" when trying to drag the folders across to
the "local" trash folder, I meant that "nothing visibly happened" - no mails
appeared to be deleted from the inbox, nor were they present in the local trash.

I have saved the corrupted boxes by moving them to Explorer-created folders
within the Mail folder.  Mozilla appeared to successfully rebuild the Inbox and
both Trash folders on it's own (however, my original inbox had a subfolder which
was used to hold messages from a particular mailing list - a filter moved them
to that box upon reciept.  

It is a POP server.

One other (possibly relevant) item is that all of my stored mail folders
(created in Mozilla from their Netscape 4.7 counterparts) appear under the
"dbharsh on mail.rdu.bellsouth.net" account instead of "local."  I personally
thought they should have been "local" since they are no longer on the server,
but just left them alone as I didn't want to interfere with the automatic
account creation.

Another comment: this is my first use of Bugzilla and MAN am I impressed.  This
is the absolute best bug tracking tool I have ever seen in my life.  The user
gets to see developer feedback and comments and can provide input.  I work in
telecom and we could sure use something like this!  Very, very impressed so far!
Forgot to continue the thought on the subfolder of Inbox - naturally this was
not recreated by Mozilla and I did that manually this afternoon.  I have
received mail on that list and the filter properly placed it in the subfolder,
so I think that's working fine.  That was the only manual "folder intervention"
I have done in the recovery process.
*** Bug 121167 has been marked as a duplicate of this bug. ***
Exact same behavior on my Windows 2000 Pro system. Using version 1.1 alpha.
I have seen some behaviour like this. I just submitted <a
href="http://bugzilla.mozilla.org/show_bug.cgi?id=162351">162351</a> which may
be related to this. The last time this happened - just now - I had just gone
through an incidence of 162351 (manually killed a hung mozilla process which had
persisted after closing mozilla) and now I am seeing the right message names in
my inbox but totally bogus message contents from messages I deleted weeks ago.
In the intervening time I have not connected to the server at all. ie, I think
that the .msf is still good but somehow my inbox data reverted to some old
version??? weird.

Anyway, I think that this bug merits a two-pronged attack. Fix the cause, sure;
but there will always be power failures or whatever that lead to corrupted mail
folders. There should definitely be some way to uncorrupt them, especially if
the data is still clean on the server. Backups, "restpre" commands, etc. As I
said, I've seen folder corruption several times, and I've always managed to
handle it (manually deleting the .msf file and copying over an .msf file from a
virgin folder - but that was when I had hundreds of msgs in my inbox and I may
have lost some without realizing. Now it seems that the raw data has only old
messages, not the new ones, so I'll have to choose some other strategy) but
there should be better safeguards & recover options.
Allow me to clarify this bug report:

Steps to reproduce:
1. Create a new profile (optional)
2. Create a fake account
3. Compose ten messages and Save them (they should now be stored in the drafts
folder)
4. Move these ten messages to the Inbox.
5. Mark them as unread (select all, then Message > Mark > As Read)
6. Now there should be 10 unread messages in the Inbox folder. Select the first
message. There should now be one read and 9 unread message.
7. Press and hold the Delete key.

Expected Results:
The ten messages should be moved to the Trash folder.

Actual Results:
The unread mail count rises to about 20-30 in Trash, and half of the messages
are left in the Inbox folder. The database seems to be corrupted too, with
emails without subjects and a weird date.

Reproducible: ALWAYS

Adding the keyword dataloss. OS -> All
URL: http://n/a
Keywords: dataloss
OS: Windows 2000 → All
Hardware: PC → All
David, what build are you using?
Always using the latest, or almost latest in this case: 2002091708. Windows XP.
Please try the steps to reproduce, they work! :)
ah, press and hold down the delete key? so that it does multiple deletes, one
after  the other? Yes, that didn't work so well. Navin, you should try that. It
exercises the copy service, I believe.
I tested the steps here on my Winxp with the 9-17trunk.
I used a valid POP account when I created my New Profile.
I followed the steps and when I held down the Delete key the Trash count kept
rising. I got up to a count of 195 while 4 of the 10 messages stayed in the Inbox.
a.) I released the Delete key and checked the 4 messages left in the Inbox, they
were accurate when I opened each one.  
b.) I opened the Trash and found the other 6 message, they too were accurate.
c.) I then Logged into my POP account and the incoming messages from the server
downloaded.  The 4 previous messages were still there.  
d.) I deleted one of the new msgs that came in, 2 of them appeared in the Trash,
e.) I deleted one of them from the Trash, both disappeared.
f.) I Deleted another newly downloaded message from the Inbox, it showed up in
the trash along with a blank message with a date of 12/31/69.   The trash count
is now at 196, with only 10 listed messages.  2 of those messages that were
moved there with the original scenario have duplicates.  Each time I delete one
of the duplicates,a blank msg appears with the 12/31/69 date.  

Basically at this point we have a corrupted .msf file. Exiting the app and
relauching doesn't help the Trash folder. The Inbox appears to be correct with
the count but it still has the messages I tried to delete in the first scenario.
Compact Folders, cleans up the Trash, giving the correct count (4) and no blank
messages. 

Now, with my POP Inbox and Trash corrected, I decided to hold down the Delete
Key with the 1st msg selected in the Inbox.  The same type of behavior.  The
messages stopped deleting from the Inbox, but the count for the Inbox continued
to decrease until it showed I had none. The Trash continued until it showed I
had 186.  I open the Trash, I only have 11 msgs 7 newly deleted and 4 from the
previous count.  The Inbox count show I have 0, but there are 139 msgs listed
and they all appear to be OK.  
Compacting folders again cleans up the msg count for the Trash.  



I have a suspicion that the same hdr is getting deleted multiple times from the
db, and this is causing all sorts of problems.
If you press Delete once, a message will be deleted properly, but if you hold
down the key, this problem occurs.

As a developer myself, I strongly believe that the reason for this is because
the delete key event is triggered while another message is deleted. The UI
doesn't get to update before the delete key event is triggered again, so Mozilla
tries to delete the same message again and again (which doesn't exist at the
original location anymore).

This is probably easy to fix by stopping the delete key event from being fired
while a mail is being processed. I'm a VB developer (don't flame me), so the
logic would go something like this:

Sub DeleteMessage()
    Static IsProcessing as Boolean
    If Not IsProcessing Then
        IsProcessing = True
        ' delete mail code goes here
        IsProcessing = False
    End If
End Sub

Just my two cents.
I just tried this with our released branch build and had no problem when I held
down the delete key.  I had 1162 msgs in the Inbox, held down the key until 238
msgs had been deleted.  The Trash had the correct count and the Inbox had the
correct account.  This appears to b a Trunk build only bug.
Note: In the branch test I did not do the move msg to Inbox before I had logged
in so I'm not sure if that step is necessary to see the problem. I tested this
as a typical user would.  I added the account, logged in and waited for the
download, then selected the 1st msg and held down the delete key.  
Branch build is OK with whole scenario in comments #11 & 15, marking this as
regression and nominating.
Keywords: nsbeta1, regression
QA Contact: sheelar → esther
This is strange that this bug has been around for a while, it was not resolved
as fixed in the branch, but it doesn't happen in the branch.  There must be
another fix that went into the branch that circumvents this bug. 
we have code to deal with multiple nearly simultaneous deletes, by queuenng them
up  in the msg copy service - my guess is that something's going awry in this
scenario,  I'm not sure where.
I'll take a look.
Status: NEW → ASSIGNED
Attached patch proposed fix — — Splinter Review 
The fix is to not do subsequent deletes if we are already in middle of deleting
last one.
cavin, bienvenu, Can I get reviews ? thx.
why can't we queue up the deletes, like the copy service does?
we can't do it because we remove the rows after delete is over in
OnDeleteCompleted    (perf gain) and we use the current selection over there to
remove rows.

The same delete was getting queued multiple times because we pressed the delete
key and didn't release it. So index was same but delete was getting fired. 
Could you say more about the perf gain you refer to? If we're not going to allow
deletes while a delete is in process, is it a fair question to ask what use is
converting everything to use the copy service?

For the user, the best performance is often given by removing the hdr from the
view, loading the next message, and then deleting the msg, because, often, the
user is deleting a single message and really wants to read the next message.
The way delete works right now is we copy the message and if successful then we
delete it from the view in OnDeleteCompleted. And we depend on the selection to
stay intact until OnDeleteCompleted. We can change the way delete works going
forward but I think let us fix this issue for now. 

One way we could do delete in future is to remove it from the view when we are
deleting (before copying) and then add it back if copy failed. 
The copy service change was done because OnStopCopy handler was not getting
called in some cases. In order to make it safer, I made everything go through
copy service. I think we can revisit the issue of queuing deletes as I said in
my earlier comment. 
Comment on attachment 99896 [details] [diff] [review]
proposed fix

sr=bienvenu. This change worries me because we're now turning off delete/move
when _deletingRows is true. If it ever happens that m_deletingRows gets stuck
on true because some operation that sets it doesn't clear it, we'll never be
able to delete messages in the view. But I tried all the different things I
could think of and it was cleared. So, since this causes data corruption, I'll
let this quick and dirty fix in while we work on a real fix.
Attachment #99896 - Flags: superreview+
Comment on attachment 99896 [details] [diff] [review]
proposed fix

r=cavin.
Attachment #99896 - Flags: review+
I'll log a separate bug to deal queuing deletes. marking this fixed.
Status: ASSIGNED → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED
QA Contact: esther → gchan
Using trunk builds 20030225 on winxp, mac and linux I don't see any problems
deleting multiple select messages in my POP accounts or in my Local Folders.  
Verified
verified
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: