For a long time I have been using Netscape with 4.77 as my stable backup position after giving up on N6. I installed Mozilla and after successful use for several weeks, I started noticing that duplicates of emails would appear in my inbox. I deleted them of course. Over time the number of duplicates increased to the point that I would have 8 copies of emails and virtually every email in the inbox was duplicated many times. I treid to get help for this but got nowhere so I deinstalled Mozilla and went back to N 4.77. I jsut installed M1.1 and when I opened the email reader, the inbox was filled with hundreds of duplicated emails. It is not possible to delete all the duplicates, as they reproduce the next time the reader is opened. I will have to dinstall and go back to N4.77.
Are you using pop3 or imap? I'm guessing POP3. Did you import your mail from 4.77 or some other mail client? Is this happening with old messages or newly arrived messages or both? Do you have pop3 leave on server turned on? If so, you could be running into a problem where your popstate.dat file is getting lost/corrupted/deleted so that we think we have to re-download messages. Can you tell if these duplicates are re-downloaded from the server (i.e., a whole bunch of messages get downloaded from the server, but most of them are duplicates)? Or do they just magically appear? Do you have mail filters? Do you have auto-compact turned on?
Assignee: bienvenu → naving
Status: UNCONFIRMED → NEW
Component: Mail Database → Mail Back End
Ever confirmed: true
Assignee: naving → sspitzer
Very similar: Bug 208526 pi
i apologize if the following gets off-topic, but i'm hesitant to go and open new bugs for it all since it seems very related to me. i've been using mozilla since 1.0, and after recently installing 1.5, this "mail duplication" bug has started to affect me (again) too. just now, i deleted some junk mail from my junk mail folder, went into the trash and emptied the trash, and then ... i go into my sent messages folder ... and 500 messages from a couple of my other folders appear there. duplicates. it also happened to my inbox recently as well (like within the couple of days). note that when i tried to "delete mail marked as junk in folder" from the menu, it stopped responding so i had to go and do it all by hand. i don't know if this is related, but also in the past couple of days, i've noticed it says on the status bar that it is receiving (downloading) new messages, but then they don't appear in any folders!! what's going on? am i losing mail too? i'm using a pop3 server like i have for ages, and i'm not leaving any messages on the server. i think i'll have to use some third party software (Mailbag Assistant can do it, i think) to eliminate duplicates, and downgrade to 1.4 again. i noticed a similar duplication problem whenever i try to import messages from outlook express, although i haven't done that since before about 1.2 iirc. i'm sorry that i don't have more explicit information about what's going on here, but this seems like a pretty serious problem. my guess would be something to do with one of (a) multithreading problems (b) some disk storage / caching problems, (c) problems with junk mail controls. in fact, i noticed some multithreading errors in the filtering code as well a while ago (bug 193162), it could be related. i just checked, and for many of my folders all of the messages have been remarked as "unread". this has happened in the past as well, and seems to co-occur with this duplication problem. i think it could be some freaky multithreading bug that happens randomly.
oops. i meant to refer to bug 193612 in my last comment.
Is this a dupe of bug 90422? pi
hi pi, perhaps, i don't know whether it's to do with the msf file getting rebuilt or not. all of the reports in that bug seem related to getting duplicates in the inbox, whereas my problems are more severe. i'm getting duplicates in other folders (e.g. sent items) of messages that were filtered to other local folders. rephrased: i now have 500 new messages in my sent folder that also exist in several other folders. recently happened to my inbox as well. concurrently (as in, i think it's all the same bug), i'm getting the problems described in bug 57660, all of my messages are marked unread. finally -- this may be helpful -- when i go to different folders now, the progress meter on the status bar freezes in a different position, even though nothing is happening. really weird. to reiterate: i really think now that it's a threading problem, since (a) it's random, (b) it has several different manifestations, and (c) it's sort of a systemic problem. a list of bugs that seem all related: bug 92763 bug 171110 (note: this just shows that there are in fact synchronization problems, although the symptoms are quite different) bug 90422 bug 57660
sorry, i'm really tired. i meant (again!) to say bug 193612 instead of bug 171110 at the end of my latest comment (talking about synchronization problems)
The messages getting marked unread is caused by a combination of two problems: 1. Importing messages from Outlook/Outlook Express doesn't add an x-mozilla-status line to the messages. This status line is used to keep track of a set of flags about the message, including the read state. 2. The timestamp for the mailbox file is getting out of sync with the timestamp stored in the .msf file - to Mozilla, this makes it look like some other app has changed the mailbox file, in which case, we need to delete the .msf file and and regenerate it, using the flags stored in the x-mozilla-status line. Daylights savings time transitions cause us to think the timestamp is out of date on some machines (but not all - it's never happened to me). So, if there are no x-mozilla-status lines, we lose the read state when we regenerate the .msf file. This regeneration of the .msf file might be involved in your problem. It might have either caused the problem, or, it might have simply exposed a long standing problem. Because we use Berkeley mailbox format, when messages are deleted, we don't remove them from the mailbox, we just mark them as deleted in the x-mozilla-status line, and remove the msg entry from the .msf file. But, if there's no x-mozilla-status line, when we regenerate the .msf file, deleted messages will re-appear. So you're dying to know how to get the x-mozilla-status lines into your mailbox files. If you compact a folder (assuming there's at least one deleted message in the folder so that we think compacting should do something), the compaction process will add the x-mozilla-status lines to the messages.
Hi David, Okay, that's a nice and coherent explanation for what's going on. Of course it was the daylight savings switch yesterday, I don't manually compact folders, and in the past OE was my email client. Is it difficult to fix the import process to add the X-mozilla-status line for all messages? Maybe you can additionally add an "add X-mozilla-status to all messages in this account" command to the Tools menu. Also, I had just noticed what you pointed out: messages are (most of the time?) not actually deleted from any folder. I don't know if this is actually a requirement for using the mbox format or not (my guess is not). However, I think many users out there would be quite upset if they knew messages were not actually getting deleted, or even moved(!), for two reasons: 1) Security 2) Space loss I had a 600 Mb mail folder, and when I used Mailbox Assistant to eliminate duplicates, the size shrunk to 400 Mb, with 4000 out of 19000 messages being duplicates. I deleted the .msf files before importing into MA, and discovered I had 500 deleted files lying around too (hopefully, some of these are the deleted files from my problems in bug 193612 that you read about). Also, when I was doing some testing for that bug, to see if I could get the filters to reproduce the problem, I quickly ran out of hard drive space (I was copying large numbers of messages back and forth between different folders). I would suggest the following further fix to this "duplicates" bug: 1) Make all messages with an X-mozilla-status of 'deleted' visible (maybe highlight them in red or something) 2) Make the delete operation actually move messages to the trash. 3) Make move operations actually move messages to the target folder. 4) Make the trash operation actually delete messages, if it doesn't already. You could let users have a "keep compacted backups of trashed messages" option available, and this would provide a more organized solution to the accidental dataloss problem that you currently solve by keeping hidden copies of deleted and moved messages spread out over the entire email account folder. If you do that, then people aren't going to get this problem at all, and the mail folder will be a lot smaller. For now, this is my workaround to prevent against the duplicate and unread messages problem (for anyone else who cares enough): 1) Delete all .msf files in account directory 2) Import all mailboxes into Mailbag Assistant (sorry to spam this product, I don't work for them or anything, I just couldn't find comparable tools) 3) Use MA to remove duplicates 4) Export sent and received messages to different generic mailbox files, 1000 messages at a time (MA assistant chokes on requests to export large numbers of messages). 5) Open mailbox files, remove all X-mozilla-status lines with an editor (or use a script). 6) Copy edited temp-sent and temp-inbox mailboxes to mozilla account 7) Set up and run filters to move all messages to other folders 8) Delete temp-sent and temp-inbox mailboxes by hand. 9) Compact all folders 10) In the future, make sure that all new messages from the Inbox and Sent folders are filtered into other folders or deleted (i.e. they contain zero messages), and then go into the account directory and delete the mailboxes by hand. 11) Never move a message once it leaves the Inbox or Sent folder! If you could tell me where to look for information about the X-mozilla-status numbers, then I could change step (5) to remove only those lines that pertain to "hidden" messages. Cheers, and thanks for the replies, Chris
I've looked into fixing the import process. I will try to fix it. Understanding the code was a bit of a challenge.
David, I guess this is not the right place (please let me know where this would be or if I should file a new bug on it). Anyway: When Mozilla feels it is time to rebuild the .msf file, it throws away information (like how to sort messages). But since the old .msf file is still there, we could extract all that information and use it in rebuilding. pi
has there been any progress on fixing the importation process so mail duplication won't occur at daylight savings?
I fixed outlook express import - I think Eudora import might not add the x-mozilla-status lines, but I think I also fixed the daylights savings time issue as well.
are the import fixes already in Mozilla and TB, or are they expected in a certain future release?
already in tbird .4 and Moz 1.6
is there any way you'd consider actually removing duplicates / deleted messages from mailboxes, instead of just hiding them? if i move a message with a 10 megabyte un-compactable attachment, my mail folder grows by 10 Mb ... and, like i said before, people worried about security might be surprised to find messages they thought they had deleted were still there ... i do realize that this might not be a big priority, i'm just curious as to whether or not i can hope for this feature some time in the future (otherwise, i will continue trying to rebuild my mailboxes myself). cheers, chris
I'm not under any sort of illusions that Mozilla and TB are the only email progs to leave behind "ghost" emails until they are compacted away, but I would like to know that the bugs resulting from these ghosts are being identified and eliminated. So the problem with importing emails has been fixed. However, if I go through another daylight savings, and I've recently deleted mail, but not compacted, will those deleted emails reappear? Just a thought.
Firt of all, I apologize for writing a rather bad english. I am not sure this place is the good one, as i rather new as a Mozilla-user. I did migrate from OE to Mozilla1.6 some days ago, and had no problem with the counting of read/unread messages. Nevertheless I experimented the problem when I moved one of my profiles from disk C:\ to another disk, for saving and security reasons (I never keep datas on system-disk). To access to sub-folders, I modified "local folder" values in "Mail and Newsgroup Account settings", OK. But then I discovered a strange mix of read/unread countings : Some folders have good counting, but very few ; others have inverted countings ; a lot of sub-folders have huge numbers or question-marks : I had to click to each sub-folders to obtain credible numbers, but still inverted read/unread. As I use 187 subfolders, it taked much time. Afterward I saw that my preference on date sorting was lost, all the sub-folders needed to be visited one by one again to modify this. Is there a way of prevent this, or is this some leftovers of Bug 171110, fixed for OE migration, but still resilient when moving a profile to another folder location ? related with "x-mozilla-status lines" ?
do you still see this problem? try TB 2.0 beta if you are able http://stage.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8/
Summary: Mail inflation -- run away duplication of everything in the inbox → Mail inflation -- run away duplication of everything in the pop inbox
sorry for the spam. making bugzilla reflect reality as I'm not working on these bugs. filter on FOOBARCHEESE to remove these in bulk.
Assignee: sspitzer → nobody
I believe this was fixed a while ago...
Status: NEW → RESOLVED
Last Resolved: 12 years ago
Resolution: --- → WORKSFORME
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.