Closed Bug 90422 Opened 23 years ago Closed 17 years ago

Mail messages are duplicated, when msf file is rebuilt for pop inbox

Categories

(MailNews Core :: Database, defect, P2)

Product:
MailNews Core
Component:
Database
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: jaimejr, Assigned: Bienvenu)

References

Details

(Keywords: dataloss, fixed1.4.2, regression, Whiteboard: adt2)

Attachments

(2 files)

proposed fix
3.34 KB, patch
mscott
: superreview+
mkaply
: approval1.4.2+
Details | Diff | Splinter Review
subset of patch required for 1.4.2
678 bytes, patch
Details | Diff | Splinter Review 
Build ID: 2001-07-11-06-0.9.2
Reproducible: Sometimes

Steps:
1. Launch the browser
2. Open Mail window
3. Select and open POP

Results: Sometimes, for whatever reason, messages are sent down from the server 
to my POP account, duplicating the messages (Note: Must have, "Leave messages on 
server" selected.
Keywords: nsbeta1
I experience the same behaviour.

My pop server reports that it is running "QPOP 2.5".

It has occured around 10 times over around 4 months. Can't detect any rhyme nor
reason though :(
Blocks: 104166
Jaime, when you say open POP in step #3, I want to make sure we do exactly as
you do in this step.  
1. Are you clicking on Get Msg and getting the password dialog.  
2. The summary mentions "when the msf file is rebuilt", had you removed the .msf
file prior to this launch?  If not, does the summary file get build every time
you open this folder after a relauch of the app?
3. What is the size of your Inbox file?
4. Do you Compact this folder on a regular basis?
Jason, if you can answer these questions too, that would help.

answers ... maybe :-)

Q1. Are you clicking on Get Msg and getting the password dialog?
A1. Nope. the password dialogue comes up when I Select Mail & Newsgroups, and
the first window opens. For Step 3, I am openig a particular Account Folder.
  
Q2. The summary mentions "when the msf file is rebuilt", had you removed the .msf
file prior to this launch?  If not, does the summary file get build every time
you open this folder after a relauch of the app?
A2. Yes, and no. Sometimes I delete .msf files, and sometimes the damn thing
just starts rebuilding itslef for some reason.

Q3. What is the size of your Inbox file?
A3. Size matters? lol ... the msf is 38K, and the inbox is 3.6MB

Q4. Do you Compact this folder on a regular basis?
A4. Yes, because I get that annoying dialogue all the time these days. ;-)

lemme know, if you need addtional information.

Jason also answered these question however he's no longer using the same
platform or mail client.  1.Not certain, probably NO.  2. I didn't manually
remove .msf file, not sure if summary file rebuilding or not, can't tell.  3.
200-500 msgs  4.Never compacts folder.


With this new info will try to reproduce in lab.
Keywords: nsbeta1+
I was having exactly the same problem using build 2001110815 on Linux. I'm
connecting to a Cobalt Linux box running some unknown version of qpop. However I
looked at /var/log/maillog while clicking the 'Get Msgs' button, and I spotted:

-ERR Error updating primary drop. Mail left in temporary maildrop (28)

associated with my connection. This was caused by the amount of disk space free
being smaller than the size of my mailbox. After I cleared out a lot of old
messages with pine and made more space available, the problem went away.
Priority: -- → P2
Status: NEW → ASSIGNED
Target Milestone: --- → mozilla1.0
Blocks: 122274
Keywords: nsbeta1+ → nsbeta1-
Target Milestone: mozilla1.0 → mozilla1.2
Jaime,
Have you been experiencing this problem in the recent builds? Navin has fixed so 
many issues regarding the msf file corruption. Please comment.
I know longer am using POP for an account, so i can't tell you, if this is still
an issue.
Blocks: 92973
I have the same problem, I use Mozilla 1.2b on WinXP and Win2000.
I sometimes receive all my mails from the pop server, not only new e-mails.

I've also noticed, for an unknown reason, my login (jotak) has been changed to
jotak/pop.free.fr (pop.free.fr being the mail server).
Renominating, since this is being seen by at least one person out in the wild.
Keywords: nsbeta1- → nsbeta1
Whiteboard: adt2
QA Contact: sheelar → esther
This happens to me ALL THE TIME! I am running M1.1 on Win/ME. I am using POP
with "Leave Mail on Server". Running "Compact Folders" sometimes brings this on.
Other times it just happens randomly.
I'm not sure this is 100% a Mozilla issue.  I used to have this problem many
times before using Outlook Express.  Of course, that doesn't absolve us from
solving the problem, but it's more wide-spread than just a single client.

I too have my preferences to leave mail on the server.  While IMAP would be a
better solution to my reason for doing this ("roaming" between work, travel,
home, etc) sometimes that's not an option.

My inbox has grown to near unmanagable proportions, holding (at present) just
over 19,000 messages (not a typo!) and consuming over 388 megs of disk.  Disk
isn't an issue but poor Moz dealing with such a massive file.

I have no idea how the inner workings of Mozilla Mail handle the mail indexing
and such, but I'm curious if there would be a way for it to index the message
ID's downloaded (if it doesn't already) and reject duplicates from the server? 
Maybe it does this already and it's not quite air-tight?

Also, I have my client set to "Delete messages on server when they are deleted
or moved from Inbox".  Can anyone tell me *when* this happens?  As in, are they
deleted when I "Empty Trash" or when I "Compact Folders" or just when I hit my
"delete" key?

Best wishes tracking this down; I'd help if I had the skills.

--Lyle E. Dodge
lyleATlyledodgeDOTcom

OS: Win2k, SP3
Moz: 1.2.1 (Gecko/20021130)
Unsure of Mail server type/version
James E Cottrell 3rd, does the problem still occur in a more recent milestone?
For instance Mozilla 1.2 or you could try 1.3b when it is available.
Bug 111787 the last comment dated 9/16/2002 mentions this same scenario and that
he's been using mozilla for 2 years without a problem and this just started
happening the past week (meaning around 9-10-2002).  That bug was resolved as
wfm before the additional information.  I will resolve it as a dup of this one
and add the last comment in that bug added to this bug because it has more info
to help reproduce this.  Here is the comment from that bug:

------- Additional Comment #18 From Tino Wuensche  2002-09-16 17:46 -------

I've been using mozilla mail client for almost 2 years and never had this
problem until last week.
I have "Leave msgs on server" checked as well as "Delete msgs on server when
deleted locally". When everything is working fine I have a popstate.dat file
with as many entries as I have msgs on the remote server.
Then, sometimes (but not every time), after a new message has arrived in my
remote inbox I click "Get New Messages" but the mail client says wrongly "There
are no new messages on the server".

At that point, I can see that my popstate.dat file has lost all its entries.
Now, clicking on "Get New Messages" again will naturally download all messages
from the remote server (incl. the one I had just received) because the
popstate.dat file is empty, I guess.

After this happened a few times, I emptied my remote and local inbox (I was
getting tired of downloading ~1,300 msgs every time). Same thing with the fewer
and different messages.

I also tried downgrading to 1.0 and 1.0.1 to no avail. The weird thing is this
started happening at home and in the office around the same time last week so I
figured it might be a 1.1 bug (I had just upgraded from 1.0). But after having
changed back to 1.0 and the problem not going away I am wondering if something
weird is happening on the server side (am using Yahoo! mail, so a lot of other
ppl should have this problem, too).

I can see how this is a tough one to reproduce for you guys as this has never
been an issue for me until now. Maybe you can give it another shot. This is so
bad that I will soon have to stop using mozilla mail :-(

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/2002082.
Mail triage team: nsbeta1-
Keywords: nsbeta1 → nsbeta1-
I don't know how many more examples you want but I'm getting it now and then.
Pop to groupwise v5 server, leave messages on server, compress folders, etc.
I'll check the headers and everything is the same - it is the same exact
message(s). Sometimes it dups more current emails other times it dups something
from several months ago - not really consistent (which makes it hard to debug).
I've been getting these dups for many months and I do try to keep current on
the nightly builds + watch the CVS checkins for items to try out. I'm not
totally sure that rebuilding msf's are part of the equation. I've blown them
away and left them there for different tests - it didn't seem to make any
difference.

I'm currently running:

Mozilla 1.3b
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3b) Gecko/20030214

I too have been seeing this problem.  I have three email accounts set up:

servers:
1. yahoo
2. an exchange server (work)
3. ?? don't know

clients:
Mozilla 1.3b, win2k ( on 2 different machines )

I get duplicate email randomly from all three servers, and I am getting quite
sick of it.  My settings are:
a. leave messages on server
b. check every 10 minutes
c. download automatically
d. check at startup

I have been seeing this problem for quite a while ( v1.2 i think ).


I too have been seeing this problem.  I have three email accounts set up:

servers:
1. yahoo
2. an exchange server (work)
3. ?? don't know

clients:
Mozilla 1.3b, win2k ( on 2 different machines )

I get duplicate email randomly from all three servers, and I am getting quite
sick of it.  My settings are:
a. leave messages on server
b. check every 10 minutes
c. download automatically
d. check at startup

I have been seeing this problem for quite a while ( v1.2 i think ).


mass re-assign.
Assignee: naving → sspitzer
Status: ASSIGNED → NEW
Hi

I'm using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4b) Gecko/20030507

I have 3 pop3 accounts (2 of them are the same server with different user
accounts - pop.free.fr - and for these 2 accounts I leave messages on the server)
When I start Mozilla the first time of the day, with 10-50 new messages on each
account, I happen, some days, to have a few messages being downloaded twice
(inbox of the account with the more messages and being left on the server).

It isn't really an issue with .msf since the message is really present twice in
the message file (check by opening in a text editor), and rebuilding the .msf
file doesn't change anything (by suppressing .msf and starting again).

Inbox sizes : 114 MB, 40MB, 2MB
Pasword is stored, and messages are collected automatically on startup on the 3
accounts
I don't use "compact mail boxes"
I have more than 100 messages left on the server
I also use "Delete msgs on server when deleted locally" (which seem to work now
and didn't seem to work in 1.3)

Reading the previous comments, I suggest to change the summary to "Mail messages
are duplicated, when multiple pop3 accounts checked at startup with "leave on
server" option"
I seem to have "fixed" my problem by checking "get new messages at startup" for
only ONE(1) mail account, and not checking this button for all of the remaining
accounts.  All mail accounts have check for mail every so often, download new
messages automatically, and leave messages on server checked.
I am not sure, that bug 208526 is he same. If so it is OS=ALL and critical due
to dataloss. I actually use POP and leave mail on server. But in
/var/log/maillog there is nothing which shows a problem.

The only thing which helps is restarting. If you move mails to different
folders, things get worse, actually mails duplicate really, not just in the display.

I'd assume this is Mail Back End not POP3, though.

pi
Can anyone who sees this bug check, if bug 208526 looks the same to them? Who is
seeing it in which version now?

Especially, the following problem showed up for me already two times today:
Mails multiply (not just duplicate!). No matter how many mails are in one thread
before the problem, after it only one is accessible (several times then).

pi
I also seem to have this issue.

I'm on Windows XP Home, using:
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624

I am using 2 seperate POP3 accounts.

Both are configured as:

- Check for new messages on Startup
- Check for new messages every 5 minutes
- Automatically Download new messages
- Leave message on server. 
- Delete messages when deleted or moved from inbox.

Current inbox sizes are :

Account One   :  203 MB
Account Two   :  1.7 MB
Local Folders :  211 MB

I do not compress my folders.

The thing about it is that I don't just recieve duplicate/multiple copies from
the server. I have a practice of archiving older messages to a seperate "Local
Folder" for later reference. I often get messages that where archived months or
even years ago re-appearing in my inbox as duplicates (they are not removed from
the archival folder) and marked as "new". Some of these archived messages where
imported from a previous mail program, and never existed on my current POP3
accounts.

This problem always seems to occur after a "hang" in the mail client on startup
.. sometimes long, sometimes breif ... it continues to happen until I restart
after it has started in any particular session. 

Frequency of occurance is perhaps once or twice a week.

This has been occuring since I first installed and switched to Mozilla ... v1.2
I believe.

I would like to add that I have a hack that "fixed" my problem.

I only set "get messages at startup" for ONLY 1 account.  My other 5 accounts
get their messages after a while....

This has solved my duplicate message problem.

Oh... winxp/mozilla 1.5b
Mark, your solution does not work for me, I only have one account.

pi
*** Bug 208526 has been marked as a duplicate of this bug. ***
Carrying over properties from duped bug, see there for details.

How about the component? I'd guess it is Database or Back End.

pi
Severity: normal → critical
Keywords: dataloss, regression
OS: Windows 98 → All
Hardware: PC → All
Boris,

What is your configuration? (OS, mozilla ver, hardware, etc)

MB
Mozilla 1.6a, Copyright (c) 2003 mozilla.org, build 2003092413
Using Linux (RedHat) on a PC. Also see earlier versions in bug 208526.

pi
Hello,

I'm having a similar problem. I pop my mail (from one account) every 10 minutes.
At least once an hour, when new messages arrive, every non-new message in my
inbox is duplicated. If I delete one of the messages, the duplicate will go
blank. To fix the problem I compact my inbox. The Inbox mail text file seems to
be intact (ie. no duplicates), so I assume it must be the msf that is getting
corrupted.

I'm using Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5) Gecko/20031007.
Clearing outdated target milestone, removing dependency to invalid bug 104166,
asking for blocking, since this here is a dataloss.

pi
No longer blocks: 104166
Flags: blocking1.6b?
Flags: blocking1.6a?
Target Milestone: mozilla1.2alpha → ---
1.6a is nearly done and this is an old bug with no developer activity. drivers
would consider a reviewed patch if it happens in time, but not gonna block for this.
Flags: blocking1.6a? → blocking1.6a-
msf file would be Mail Database. Moving.

pi
Assignee: sspitzer → bienvenu
Component: Networking: POP → Mail Database
Do any of you who see this problem have junk mail controls turned on for your
multiple pop3 accounts that check for new mail at startup? I fixed a bug that
caused all sorts of problems with that configuration that made it into 1.6a
final, I believe, on 10/28/2003
I am not use the junk mail feature at all.

pi
I don't think this is junk-mail control related since we've had this bug for a
couple of years now. Unless we get reproduce-by instructions, it's not going to
be fixed for 1.6
Flags: blocking1.6b? → blocking1.6b-
I had this problem already three times in one hour today! Having the machine do
some work seems to help it happen. I only see it in the inbox. What can I do to
help tackle the problem? Dataloss is not funny:-(

pi
Flags: blocking1.6?
what build are you running?
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6b) Gecko/20031114, but it is
happening since my first report (or a bit earlier) in bug 208526: Mozilla/5.0
(X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030514

So for me it looks like a regression in early 1.4 development. Others have much
earlier report, though.

pi
if a patch appears on the horizon renominate this for 1.6.  looks like one we
should spend some cycles on for 1.7
Flags: blocking1.6? → blocking1.6-
I just have 1.6 (release) here for less than one hour. The bug is still around:-(

pi
Flags: blocking1.7a?
OK, I figured out what's going on here - I just need to figure out how to fix
it. Basically, when biff fires, and it notices that the Inbox.msf file is out of
date, it forces it to get reparsed. But, if there's a view open on the INBOX,
that results in the view getting notified of all the reparsed headers as "new"
headers, so it adds those headers to the view. 

It's tricky to fix - I could force the db closed, which would clear out the
view, but the view wouldn't get re-opened, so the thread pane would stay blank.
I could fix the code in the view that adds new headers to the view to check if
the headers were already in the view and if so, ignore the notification. That
gets a little involved if the view is threaded, and in general would slow things
down a bit. I'd much prefer to have the view not get notified of new headers
during reparse.
David, I am glad you found the problem. Some things I don't understand:

1) I have never observed the problem in any other folder then Inbox. Is that
systematic or by accident?

2) Why does the view get notified about *new* mails? Why is it not notified that
there is some problem and it just needs to refresh?

pi
this is specific to the inbox, because the code that causes this is the code
that starts downloading new pop3 mail to the inbox.

It's cheaper to notify the view of new mail (e.g., add this one line), then it
would be to say - go rebuild your view of 10,000 messages because we added one line.
Attached patch proposed fix — — Splinter Review 
I've finally given in and added a boolean var to nsLocalMailFolder.cpp that
says whether we're reparsing the summary file. If we're reparsing the summary
file, we don't generate a folder loaded notification, which fixes clicking on a
folder with an out of date summary file to postpone creating the view until the
reparse is finished. And then I made reparsing the summary file not generate
notifications of new headers added, since no one should be listening for those
notifications during reparse.
Attachment #139736 - Flags: superreview?(mscott)
Comment on attachment 139736 [details] [diff] [review]
proposed fix

looks good. do you think this should be an m4 branch candidate?
Attachment #139736 - Flags: superreview?(mscott) → superreview+
Comment on attachment 139736 [details] [diff] [review]
proposed fix

I do think this would be a good 1.4.2 candidate, but I'd like to give this a
chance to work on the trunk for a little bit, which might not fit in with the
1.4.2 schedule.
Attachment #139736 - Flags: approval1.4.2?
fixed on trunk.
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
thank-you thank-you thank-you :)
Comment on attachment 139736 [details] [diff] [review]
proposed fix

bienvenu - I trust you :)
a=mkaply
Attachment #139736 - Flags: approval1.4.2? → approval1.4.2+
it's a -uw diff, so you don't see the whitespace cleanup I've done.
d'oh, wrong bug.
because 1.4 and the trunk have diverged signficantly, 1.4.2 only requires this
patch (which is part of the earlier patch).
Keywords: fixed1.4.2
checked into the m4 branch.
Blocks: 230700
Flags: blocking1.7a?
I am running 1.7a for some days now and the problem did not show up again. So
the fix seems to work for me.

pi
I am sorry that I have to report that I just had another incident of this
problem with 1.7b. But it is much less frequent now.

pi
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I'm using 1.7b and have that problem since 1.5 on different clients on Win2000
and WinXP. Sometimes getting messages twice on accounts that using option "Leave
mail on server". 
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040514

Just one more.

pi
We are experiencing this problem here too, first we thought that it's related to
a nfs problem, because all our homedirectories are served via nfs, but after all
these comments, I doubt that this is the reason.

BTW: We're using dovecot as pop-server
Product: MailNews → Core
Patrick, Boris - do you still see this problem?  note other open issues in bug 240049 and list in bug 92973 comment 15.  

David - was this specific problem completely fixed?
QA Contact: esther → database
Sorry, can't reproduce the error anymore, because we switched to IMAP
finished off by bug 9413 and bug 334664?
bug 92973 could also use cleanup love.
Summary: Mail messages are duplicated, when msf file is rebuilt → Mail messages are duplicated, when msf file is rebuilt for pop inbox
I believe this has been fixed - please re-open if you can reproduce this with a trunk (or even 2.0.0.4) build.
Status: REOPENED → RESOLVED
Closed: 21 years ago → 17 years ago
Resolution: --- → WORKSFORME
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: