Closed Bug 171110 Opened 22 years ago Closed 17 years ago

Mail inflation -- run away duplication of everything in the pop inbox

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: gnelson, Unassigned)

References

Details

For a long time I have been using Netscape with 4.77 as my stable backup
position after giving up on N6. I installed Mozilla and after successful use for
several weeks, I started noticing that duplicates of emails would appear in my
inbox. I deleted them of course. Over time the number of duplicates increased to
the point that I would have 8 copies of emails and virtually every email in the
inbox was duplicated many times. I treid to get help for this but got nowhere so
I deinstalled Mozilla and went back to N 4.77.
I jsut installed M1.1 and when I opened the email reader, the inbox was filled
with hundreds of duplicated emails.
It is not possible to delete all the duplicates, as they reproduce the next time
the reader is opened.

I will have to dinstall and go back to N4.77.
QA Contact: gayatri → esther
Are you using pop3 or imap? I'm guessing POP3. Did you import your mail from 
4.77 or some other mail client? Is this happening with old messages or newly
arrived messages or both? Do you have pop3 leave on server turned on? If so, you
could be running into a problem where your popstate.dat file is getting
lost/corrupted/deleted so that we think we have to re-download messages.

Can you tell if these duplicates are re-downloaded from the server (i.e., a
whole bunch of messages get downloaded from the server, but most of them are
duplicates)? Or do they just magically appear? Do you have mail filters? Do you
have auto-compact turned on?
Assignee: bienvenu → naving
Status: UNCONFIRMED → NEW
Component: Mail Database → Mail Back End
Ever confirmed: true
Blocks: 92973
mass re-assign.
Assignee: naving → sspitzer
Very similar: Bug 208526

pi
i apologize if the following gets off-topic, but i'm hesitant to go and open new
bugs for it all since it seems very related to me.

i've been using mozilla since 1.0, and after recently installing 1.5, this "mail
duplication" bug has started to affect me (again) too.

just now, i deleted some junk mail from my junk mail folder, went into the trash
and emptied the trash, and then ... i go into my sent messages folder ... and
500 messages from a couple of my other folders appear there.  duplicates.  it
also happened to my inbox recently as well (like within the couple of days). 
note that when i tried to "delete mail marked as junk in folder" from the menu,
it stopped responding so i had to go and do it all by hand.

i don't know if this is related, but also in the past couple of days, i've
noticed it says on the status bar that it is receiving (downloading) new
messages, but then they don't appear in any folders!!  what's going on?  am i
losing mail too?  

i'm using a pop3 server like i have for ages, and i'm not leaving any messages
on the server.  i think i'll have to use some third party software (Mailbag
Assistant can do it, i think) to eliminate duplicates, and downgrade to 1.4 again.

i noticed a similar duplication problem whenever i try to import messages from
outlook express, although i haven't done that since before about 1.2 iirc.

i'm sorry that i don't have more explicit information about what's going on
here, but this seems like a pretty serious problem.  my guess would be something
to do with one of (a) multithreading problems (b) some disk storage / caching
problems, (c) problems with junk mail controls.

in fact, i noticed some multithreading errors in the filtering code as well a
while ago (bug 193162), it could be related.

i just checked, and for many of my folders all of the messages have been
remarked as "unread".  this has happened in the past as well, and seems to
co-occur with this duplication problem.  i think it could be some freaky
multithreading bug that happens randomly.
oops.

i meant to refer to bug 193612 in my last comment.
Is this a dupe of bug 90422?

pi
hi pi,

perhaps, i don't know whether it's to do with the msf file getting rebuilt or
not.  all of the reports in that bug seem related to getting duplicates in the
inbox, whereas my problems are more severe.

i'm getting duplicates in other folders (e.g. sent items) of messages that were
filtered to other local folders.  rephrased: i now have 500 new messages in my
sent folder that also exist in several other folders.  recently happened to my
inbox as well.

concurrently (as in, i think it's all the same bug), i'm getting the problems
described in bug 57660, all of my messages are marked unread.

finally -- this may be helpful -- when i go to different folders now, the
progress meter on the status bar freezes in a different position, even though
nothing is happening.  really weird.

to reiterate:  i really think now that it's a threading problem, since (a) it's
random, (b) it has several different manifestations, and (c) it's sort of a
systemic problem.

a list of bugs that seem all related:

bug 92763
bug 171110 (note:  this just shows that there are in 
                   fact synchronization problems, although the symptoms are
                   quite different)
bug 90422
bug 57660
sorry, i'm really tired.

i meant (again!) to say bug 193612 instead of bug 171110 at the end of my latest
comment (talking about synchronization problems)
The messages getting marked unread is caused by a combination of two problems:

1. Importing messages from Outlook/Outlook Express doesn't add an
x-mozilla-status line to the messages. This status line is used to keep track of
a set of flags about the message, including the read state.

2. The timestamp for the mailbox file is getting out of sync with the timestamp
stored in the .msf file - to Mozilla, this makes it look like some other app has
changed the mailbox file, in which case, we need to delete the .msf file and and
regenerate it, using the flags stored in the x-mozilla-status line. Daylights
savings time transitions cause us to think the timestamp is out of date on some
machines (but not all - it's never happened to me).

So, if there are no x-mozilla-status lines, we lose the read state when we
regenerate the .msf file.

This regeneration of the .msf file might be involved in your problem. It might
have either caused the problem, or, it might have simply exposed a long standing
problem. Because we use Berkeley mailbox format, when messages are deleted, we
don't remove them from the mailbox, we just mark them as deleted in the
x-mozilla-status line, and remove the msg entry from the .msf file. But, if
there's no x-mozilla-status line, when we regenerate the .msf file, deleted
messages will re-appear.

So you're dying to know how to get the x-mozilla-status lines into your mailbox
files. If you compact a folder (assuming there's at least one deleted message in
the folder so that we think compacting should do something), the compaction
process will add the x-mozilla-status lines to the messages.
Hi David,

Okay, that's a nice and coherent explanation for what's going on.  Of course it
was the daylight savings switch yesterday, I don't manually compact folders, and
in the past OE was my email client.

Is it difficult to fix the import process to add the X-mozilla-status line for
all messages?  Maybe you can additionally add an "add X-mozilla-status to all
messages in this account" command to the Tools menu.

Also, I had just noticed what you pointed out:  messages are (most of the time?)
not actually deleted from any folder.  I don't know if this is actually a
requirement for using the mbox format or not (my guess is not).  However, I
think many users out there would be quite upset if they knew messages were not
actually getting deleted, or even moved(!), for two reasons:

1) Security
2) Space loss

I had a 600 Mb mail folder, and when I used Mailbox Assistant to eliminate
duplicates, the size shrunk to 400 Mb, with 4000 out of 19000 messages being
duplicates.  I deleted the .msf files before importing into MA, and discovered I
had 500 deleted files lying around too (hopefully, some of these are the deleted
files from my problems in bug 193612 that you read about).  Also, when I was
doing some testing for that bug, to see if I could get the filters to reproduce
the problem, I quickly ran out of hard drive space (I was copying large numbers
of messages back and forth between different folders).

I would suggest the following further fix to this "duplicates" bug:

1) Make all messages with an X-mozilla-status of 'deleted' visible (maybe
highlight them in red or something)
2) Make the delete operation actually move messages to the trash.
3) Make move operations actually move messages to the target folder.
4) Make the trash operation actually delete messages, if it doesn't already. 
You could let users have a "keep compacted backups of trashed messages" option
available, and this would provide a more organized solution to the accidental
dataloss problem that you currently solve by keeping hidden copies of deleted
and moved messages spread out over the entire email account folder.

If you do that, then people aren't going to get this problem at all, and the
mail folder will be a lot smaller.

For now, this is my workaround to prevent against the duplicate and unread
messages problem (for anyone else who cares enough):

1) Delete all .msf files in account directory
2) Import all mailboxes into Mailbag Assistant (sorry to spam this product, I
don't work for them or anything, I just couldn't find comparable tools)
3) Use MA to remove duplicates
4) Export sent and received messages to different generic mailbox files, 1000
messages at a time (MA assistant chokes on requests to export large numbers of
messages).
5) Open mailbox files, remove all X-mozilla-status lines with an editor (or use
a script).
6) Copy edited temp-sent and temp-inbox mailboxes to mozilla account
7) Set up and run filters to move all messages to other folders
8) Delete temp-sent and temp-inbox mailboxes by hand.
9) Compact all folders
10) In the future, make sure that all new messages from the Inbox and Sent
folders are filtered into other folders or deleted (i.e. they contain zero
messages), and then go into the account directory and delete the mailboxes by hand.
11) Never move a message once it leaves the Inbox or Sent folder!



If you could tell me where to look for information about the X-mozilla-status
numbers, then I could change step (5) to remove only those lines that pertain to
"hidden" messages.

Cheers, and thanks for the replies,
Chris
I've looked into fixing the import process. I will try to fix it. Understanding
the code was a bit of a challenge.
David,

I guess this is not the right place (please let me know where this would be or
if I should file a new bug on it). Anyway: When Mozilla feels it is time to
rebuild the .msf file, it throws away information (like how to sort messages).
But since the old .msf file is still there, we could extract all that
information and use it in rebuilding.

pi
has there been any progress on fixing the importation process so mail
duplication won't occur at daylight savings?
I fixed outlook express import - I think Eudora import might not add the
x-mozilla-status lines, but I think I also fixed the daylights savings time
issue as well.
are the import fixes already in Mozilla and TB, or are they expected in a
certain future release?
already in tbird .4 and Moz 1.6
is there any way you'd consider actually removing duplicates / deleted messages
from mailboxes, instead of just hiding them?

if i move a message with a 10 megabyte un-compactable attachment, my mail folder
grows by 10 Mb ... and, like i said before, people worried about security might
be surprised to find messages they thought they had deleted were still there ...

i do realize that this might not be a big priority, i'm just curious as to
whether or not i can hope for this feature some time in the future (otherwise, i
will continue trying to rebuild my mailboxes myself).

cheers,
chris
I'm not under any sort of illusions that Mozilla and TB are the only email progs
to leave behind "ghost" emails until they are compacted away, but I would like
to know that the bugs resulting from these ghosts are being identified and
eliminated.

So the problem with importing emails has been fixed. However, if I go through
another daylight savings, and I've recently deleted mail, but not compacted,
will those deleted emails reappear? Just a thought.
Firt of all, I apologize for writing a rather bad english.
I am not sure this place is the good one, as i rather new as a Mozilla-user.
I did migrate from OE to Mozilla1.6 some days ago, and had no problem with the
counting of read/unread messages.
Nevertheless I experimented the problem when I moved one of my profiles from
disk C:\ to another disk, for saving and security reasons (I never keep datas on
system-disk).
To access to sub-folders, I modified "local folder" values in "Mail and
Newsgroup Account settings", OK. 
But then I discovered a strange mix of read/unread countings : Some folders have
good counting, but very few ; others have inverted countings ; a lot of
sub-folders have huge numbers or question-marks : I had to click to each
sub-folders to obtain credible numbers, but still inverted read/unread.
As I use 187 subfolders, it taked much time.

Afterward I saw that my preference on date sorting was lost, all the sub-folders
needed to be visited one by one again to modify this.

Is there a way of prevent this, or is this some leftovers of Bug 171110, fixed
for OE migration, but still resilient when moving a profile to another folder
location ?
related with "x-mozilla-status lines" ?
Product: MailNews → Core
do you still see this problem?
try TB 2.0 beta if you are able http://stage.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8/
Summary: Mail inflation -- run away duplication of everything in the inbox → Mail inflation -- run away duplication of everything in the pop inbox
sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.
Assignee: sspitzer → nobody
I believe this was fixed a while ago...
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → WORKSFORME
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.