Closed Bug 166617 Opened 23 years ago Closed 22 years ago

corruption after compacting imap folders configured for offline use

Categories

(SeaMonkey :: MailNews: Backend, defect)

Product:
SeaMonkey
Component:
MailNews: Backend
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: alecf, Assigned: Bienvenu)

References

Details

(Keywords: dataloss)

Attachments

(4 files, 1 obsolete file)

locking offline stores for auto-compact
2.23 KB, patch
Details | Diff | Splinter Review
part of fix
1.34 KB, patch
Details | Diff | Splinter Review
fix most common case
7.42 KB, patch
Details | Diff | Splinter Review
fix downloading selected and all folder cases
5.60 KB, patch
Details | Diff | Splinter Review
augh, I'm experiencing more corruption on IMAP.. this one just started happening after I my folders were compacted.. I'm getting messages where I click on the message in the thread pane, I get a mostly blank header area, and a message like this: ------=_NextPart_001_0001_01C2537A.5BB68900 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML><BODY STYLE=3D"font:10pt verdana; border:none;"><DIV>Makin'&nbsp; W= hoopie Pie, man!</DIV> <DIV>&nbsp;</DIV> <BLOCKQUOTE style=3D"PADDING-RIG= HT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px so= lid; MARGIN-RIGHT: 0px"> <DIV style=3D"FONT: 10pt Arial">----- Original M= essage -----</DIV> <DIV style=3D"BACKGROUND: #e4e4e4; FONT: 10pt Arial; C= OLOR: black"><B>From:</B> Marc Lauzon</DIV> <DIV style=3D"FONT: 10pt Aria= l"><B>Sent:</B> Tuesday, September 03, 2002 5:48 PM</DIV> <DIV style=3D"F= ONT: 10pt Arial"><B>To:</B> Marsh, Christian L.; Sandra Forrest; doug bel= l; alec flett</DIV> <DIV style=3D"FONT: 10pt Arial"><B>Cc:</B> Sara Harri= s; charlie murdach; kate sweetman; Craig McGinnis</DIV> <DIV style=3D"FON= other messages a blank, and others show the headers of other messages in the body.
these are messages you have saved for offline use? You probably need to remove your .msf file (and the offline store just to save disk space). If this happened after compact, then Navin should look at it first since he does the compact stuff.
Assignee: bienvenu → naving
yeah, I think they may have been downloaded for offline use. This time around, they were recent messages that had arrived within the last 2-3 hours. And yeah I figure removing my .msf files will fix it
Was this auto-compact or normal compact ?
I don't know what that means, but it did ask me and I hit "ok"
ok, that means it was auto-compact - it prompted you for compacting.
Severity: normal → critical
Keywords: dataloss
QA Contact: gayatri → sheelar
this is not dataloss... this is on IMAP, the messages are intact on the server.
Keywords: dataloss
change->gchan
QA Contact: sheelar → gchan
I noticed that there is this file "nstmp" in my mail directory.. and the only occurance of "nstmp" in the tree is in nsMsgFolderCompactor.. the file seems to be left over from an earlier compaction. here's one thought - are we compacting multiple folders at the same time and maybe they're running into the same file because we're hardcoding the temp filename? nsIFile has a CreateUnique() that would avoid this problem, but I'm guessing that this is nsFileSpec code, so I'm not sure what to offer there (aside from converting over to nsIFile :))
the only way we'd be compacting multiple files at the same time is if multiple auto compacts were running at the same time, and in that case, you'd get two prompts, I believe. I think Navin fixed that bug anyway...
nsIFileSpec MakeUnique would do that. But, we could just ensure that the file is of 0 bytes before starting to write there. That would explain why mismatch hdr body. David, I don't think 2 compacts were running here at the same time. Atleast alec, doesn't mention it.
In comment #8, Alec speculated that we might be compacting multiple folders at the same time. That was the context of my comment. You're saying that if someone left nstmp lying around with other stuff in it, and we asked for a tmp file, we wouldn't clear out the previous contents of nstmp? That would definitely be a bug though I'd be a little surprised if that were the case that we were not clearing out the nstmp file.
I realized MakeUnique would probably be better way to go, just so that user can recover some data from last failed compact.
no, I didn't get two prompts - but I often get the folder compaction message when I switching servers, which usually involves clicking on the INBOX of different server, so I don't know if that has anything to do with it..
probably true - ideally we would never need nstmp because we wouldn't crunch the compacted folder until the compact had succeeded, but better safe than sorry.
looking for r/sr ? thx.
Are you sure this can possibly fix the problem Alec is seeing? It looks to me like the folder compaction code sets the message offset by doing an tell in the local mail folder - how can that ever be wrong? Whatever junk is at the beginning of the nstmp folder would just be ignored, wouldn't it?
yeah, that was just a out-of-the-blue theory on my part.. (though this probably isn't a bad idea anyway)
It is just not good to overwrite any file in that directory. That is just dataloss! I am trying to recreate alec's problem.
Comment on attachment 98014 [details] [diff] [review] patch to use unique temp files. great, sr=bienvenu. I just wanted to make sure you were still trying to find the cause of the bug, because I don't think the nstmp theory is a sure thing (but you should try it out, if you haven't already)
Attachment #98014 - Flags: superreview+
Comment on attachment 98014 [details] [diff] [review] patch to use unique temp files. r=alecf
Attachment #98014 - Flags: review+
We are overwriting the temp file since we are opening output stream to it. I checked it the data gets over-wrriten so that is not an issue. Probably something else happened here. I'll check in this patch but not close the bug.
I am going to check in the patch to use unique temp files against bug 70322.
*** Bug 170149 has been marked as a duplicate of this bug. ***
QA Contact: gchan → esther
I found this bug while searching for duplicatas to add a new one. i think the info may help so i'm posting it here. I set mozilla to check for mail at every minute. And when i tell it to compact folders, it keeps downloading new mail! I'm pretty sure it's a bug (or just not the way it was planed) because the status bar just go inconsistent, showing the percentage bar of the compacting, but the text shows the downloading progress...
Gabriel, this bug is about IMAP. Are you an imap user or a pop3 user? I take it you have auto-download of new messages turned on.
One way this could have happened is when compact for imap offline stores was happening in the background, we were downloading bodies for offline use (selecting a msg). Both write to the offline store separately and could have written each other's data. I think both operations should get the folder lock.
Status: NEW → ASSIGNED
Summary: corruption after compacting folders → corruption after compacting imap folders configured for offline use
Comment on attachment 98014 [details] [diff] [review] patch to use unique temp files. a=rjesup@wgate.com for 1.0 branch. Please change mozilla1.0.2+ to fixed1.0.2 when checked in. BTW, having the patch on one bug and closing it in another leads to confusion...
Attachment #98014 - Flags: approval+
I was able to reproduce this w/ case I mentioned in last comment #27. David, do you think this is important to fix ? I think checking for lock before writing to offline store should fix this problem.
Keywords: mozilla1.0.2+
it would be good to have, but it's hard to get into this situation, since we only kick off auto-compact for imap folders if you're offline when you select the imap folder, and we can't be downloading messages for offline use while we're offline. Or, you could select a local folder while the download for offline use of imap folders is going on...is that what you did?
I moved locking code up into nsMsgFolderCompactor::Compact so that we don't have to grab lock twice. We release the lock when we are done w/ request just like local folders.
Attachment #98014 - Attachment is obsolete: true
s/nsMsgFolderCompactor/nsFolderCompactState s/twice/ in two places
*** Bug 191988 has been marked as a duplicate of this bug. ***
this is really torturous. I can't be the only one going through this! my folders get corrupted at least twice a week, and reading any non-new messages is impossible. Strangely, the corruption seems to go away the next time an automatic corruption^H^H^H^H^H^Hcompaction happens. I don't know what else do to do raise the priority of this bug.. is ANYONE looking at this?
Alec isn't the only one with this problem, I have it as well, exactly as Alec describes. Not sure how longit has been going on, but I've had it since at least all of 1.3, and probably 1.2, as well. I didn't really use Mozilla for mail before that. I have multiple accounts configured, and many -but not all- folders selected for offline use. When I switch between folders or accounts (while in online mode) and answer yes to compaction, my inbox.msf on my primary account always gets corrupted and I have to quit and delete. Some others get this, as well, like sent-mail. Although I keep my inbox pretty small, I don't compact often and it could be growing pretty large. Sent-mail is a large (in KB) mail folder. I haven't seen this happen on the smaller folders, like, say, where I have Mozilla move all mail sent to BugTraq. I keep those pretty small, with less than 50 all-text, no attachment messages. Just happened again, and I decided to search for the bug. Was running 1.3final and now running: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4a) Gecko/20030314
taking, but no immediate plans to dive into this (just too doomed)
Assignee: naving → sspitzer
Status: ASSIGNED → NEW
Keywords: dataloss
doomed? So, I guess that most people do not get this error? Do we know what themitigating factors might be, or a way to workaround it? I can compact an individual folder, but it is usually when I answer to the "do you want to compact all folders now" dialoge that this happens. If there's anything I can test or try, let me know.
I have been getting this problm as well, proabably since 1.2. It is a bit frustating, however now I know that there is a link with compacting and downloading messags for off line use, I will modify my behaviour. I am using imap server, and if I get the corruption, I switch back to using Netscape 4.79 and all is well with me messages. Is it correct that if no messags are downloaded for offline use, this issue will not happen? Or do I need to stop using message compacting, which I always say yes to? Is there an easy way to reverse the corruption of emails? how can I celan up this, is there a local file that I can delete that will fix this up?
If you use IMAP mode, you download any messages that you read, even if you're not syncing for offline use, so you run the risk of running into this problem. To fix the problem for a specific folder, shut down Mozilla and delete the associated .msf file for that folder from within your profile directory. This is the index for that folder, and the next time you open that folder, it will re-index and generate that file.
taking, changing component to offline. I'm going to look at using the folder lock for this.
Assignee: sspitzer → bienvenu
Component: Mail Database → Offline
Attached patch part of fix — — Splinter Review
if we're going to lock offline stores, need to override the nsImapMailDatabase code that sets the header flag not to use the nsMailDatabase code, which will try to get the folder lock.
this patch makes it so that downloading the whole folder for offline use grabs the folder lock, and downloading bodies of new messages also grabs the folder lock. With the previous patch to make compacting offline stores grab the folder lock, we shouldn't have compact colliding with download - if offline has the lock, auto compact will fail and put up an error message. If compact has the lock, offline will just not download the message bodies. This patch also cleans up some formatting, and makes it so clicking on an imap folder while offline will try to kick off an auto-compact, but double clicking get msgs while online will NOT.
I've checked in these fixes into the 1.5 trunk - these fixes should make it much harder to get this corruption. It's still theoretically possible to have this problem if you're downloading a single message or downloading selected/all messages for an imap folder and an auto-compact happens, but it should be much harder to make an auto-compact happen while you're doing that. I'll attach a patch to protect downloading selected/all messages for a folder, but it's trickier to protect the auto-downloading of a message while reading it. Please let me know if the newer 1.5 trunk builds still have a problem.
Status: NEW → ASSIGNED
this adds protection for the case of downloading selected/flagged messages, and downloading a whole folder, and an error message if we can't get the folder lock.
marking fixed; please let me know if this happens again with a trunk build from today or later.
Status: ASSIGNED → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED
Product: Browser → Seamonkey
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: