461195 - Duplicate messages with pop3 account with Leave messages on server preference

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Wayland Moncrief]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3
Build Identifier: version 2.0.0.17 (20080914)

When the Leave message on server option is selected, Thunderbird downloads duplicate messages each time the get mail button is executed.

Reproducible: Always

Steps to Reproduce:
1.Select Leave messages on server in the account settings
2.Select Get Mail multiple times
3.Check the inbox and see duplicate messages


Expected Results:  
Thunderbird need to see if the message has already been downloaded.

Comment 1

[Rob]

Attachment: a screenshot of the duplicate mails

Comment 2

[Rob]

I actually signed up to bugzilla to file a bug report for this.

For me, it doesn't 'always' happen, but tonight it happened twice. That's twice 29 mails, downloaded from a pop server. (I only leave the mails on my server for a week, so I can also access my most recent mail when I'm away from my computer)

If someone has an idea what might be causing this, feel free to ask me to try stuff etc...

Regards,

Rob

Comment 3

[Magnus Melin [:mkmelin]]

Should normally work. 

A pop3 protocol log might shed some light... see http://wiki.mozilla.org/MailNews:Logging

Comment 4

[Rob]

Hi,

It's really not that simple: this only happens once in a while (I've been using Thunderbird for about 4 years now, and I've had this happen only a few dozen times).

I can't go about logging every single thing happening in Thunderbird just so I might be lucky enough to get the duplicate messages problem again, can I?

Regards,

Rob

Comment 5

[Rob]

Well, it did it again. See screenshot. I woke up to 55 new old mails :)

Comment 6

[Rob]

Attachment: another screenshot with duplicate messages

Comment 7

[Pat]

I have a similar problem on WinXP SP3, TB 2.0.0.18 (20081105) (but has been going on for over a year at least).  It occurs in only one of my 4 POP3 accounts.  It usually only happens a few of times a year without any apparent cause.  My computer is always on and TB checks for new mail every few minutes.

The scenario is similar to others here... look at TB first thing in the morning and there are a bunch of new messages which are mostly duplicates of existing read messages and others that are copies of old deleted messages.

I have messages sorted by date order and yet some of the duplicate messages, which show a received date which corresponds to when the message was originally receieved (e.g. many months ago), are not sorted correctly.  See attachment.

If the duplicate messages are deleted and TB restated the duplicates are returned to the inbox as unread.  I can't recall how I cleaned this up last time (shows how long between occurances) but I think it involved copying the account folders, deleting the account and copying the folders back in.

Comment 8

[Pat]

Attachment: Thunderbird erroneous duplicate email messages

Note the incorrect sorting.

Comment 9

[Wayne Mery (:wsmwk)]

screen shots are not helpful.
need a log per comment 3

Comment 10

[Milan Kerslager]

I have the same problem since some update early this year. Some messages in my
IMAP inbox are duplicated even they are not (checked locally by mutt).
Duplication hides next message in folder. When trying to delete on of the
duplicate message, the Subject:, From: and Date: changes to be empty.

The problem is the same on my Windows XP box and in my Fedora 10 or 11 too
(both 64bit). If I try to use another client (webmail, Evolution) via IMAP,
there is no inbox corruption so I'm not able to blame IMAP server (dovecot on
RHEL5). I also tryed to use older dovecot with no change.

This serious bug has been introduced into 2.0.0.x version early this year (both
Windows and Linux versions) and is present in current 3 beta (as of Fedora 11). I have no antivirus present on my systems. So don't blabe Zonealarm too.

See bug #365949 too.

Comment 11

[Martin Willcocks]

I have been having the same problem on Thunderbird and also in Netscape 7.2 Mail & Newsgroups, which used not to have the problem.  In this case, the duplicate, triplicate or even quadruplicated messages appear above the original version and are all corrupted, with no header information and apparently pointing into some other email messages.  I find the best way out is to move all new messages into a subfolder e.g. 'Inbox 2009', and then delete the unwanted duplicates. It's a long job.  I have Norton Internet Security on a Windows Vista system.  My T-bird version is 2.0.0.23.  The Netscape version is 7.2 (20040804) so it should not be getting this bug if it is a recent issue in T-bird.  I wonder if there is a trojan or similar that escapes detection by NIS 2009?  The programs are both set to download every ten minutes and delete messages from the server after 30 days, so as not to retain them for ever, but I don' seem to get the issue of re-downloading messages - it's more about messages already downloaded and read being duplicated - these show as read not new.

Comment 12

[Rob]

I've had it again. I'm in the habit of leaving the last two weeks' worth of mail on the server.

Yesterday, I received duplicates of the full content of my work mailbox once, and my private mailbox 4 times yesterday, and twice today (and it's 10 in the morning).

Will someone please take up this bug, it's extremely annoying.

I see 3.01 is available now, I'll install it and hope for the best :) I'll also make a habit of starting thunderbird with logging enabled (however, I've seen that the log contains the mail messages as well, so please advise which part of the log is interesting, so I can cut and paste)

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
> I'll also make a habit of starting thunderbird with logging enabled (however, > I've seen  that the log contains the mail messages as well,
> so please advise which part of the log is interesting, so I can cut and paste)

mail # & size of LIST response, mail # & uidl of UIDL response, mail # of RETR, and QUIT command is required. mail data itself is irrelevant. UIDL and status of mail is held in postate.dat. UIDL is written in X-UIDL: header of mail data in local mail folder file(can be viewed by View/Message Source). 
Do next, please.
(0) Keep backup of popstate.dat.
(2) Start Tb with logging enabled.
(3) Before each Get Msg, keep backup of popstate.dat, execute Get Msgs,
    and keep backup of popstate.dat after Get Msgs.
(4) Repeat (3).
Timestamp of log may help your log analysis.
  set nspr_log_modules=timestamp,pop3:5

Comment 14

[Rob]

I'm sorry, but since I can not duplicate the bug reliably I can not do this. I'm doing my job, and anything that's running automatically (like the log) I can enable, but I can't backup files before and after every get message...

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #14)
> but I can't backup files before and after every get message...

Oh, sorry for too hard data gathering work.
Keep backup of popstate.dat before start of Tb, keep backup several times if possible, when you are aware of download of new mails while you are using Tb with logging enabled(there is no need of "always". sampling is sufficient.), and, if you are aware of "duplicated download" keep backup, and keep bacup after termination of Tb, please.
If no problem occurred, discard log and backup of popstate.dat, please.
History of popstate.dat content with POP3 log will help problem analysis by developers. If life of a mail(a UIDL) can be tracked, guessing of cause of problem becomes easy.

Comment 16

[Rob]

I'm sorry WADA, but I'm in and out of meetings all day. 

I may be able to do the popstate.dat thing some time in the weekend.

However, in case anyone's interested, I had the issue again while I was logging, and I'll clean up the log and post it here.

Comment 17

[Rob]

Attachment: pop3.log showing incorrect download. Sorry, no popstate.dat

Comment 18

[Rob]

The sad (or happy) truth is, that since last comment, I've created a .bat file which does the logging for me (including saving the pre- and post- session state), but I haven't had any duplicates since.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> pop3.log showing incorrect download. Sorry, no popstate.dat

LIST and UIDL normally ends upon each POP3 access.
Write open of popstate.dat was interfered then Tb failed to read popstate.dat? Do you use auto-backup for Tb's mail directory? Do you run virus-scan peridiocally on Tb's mail directory?
Get process monitor, and check file I/O for popstate.dat. 
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
> Filter : Path ends with \popstate.dat, Include 
I think NSPR logging is not required while checking file I/O for popstate.dat

Comment 20

[Wayne Mery (:wsmwk)]

wada are you saying there is nothing useful in the protocol log?

Comment 21

[Wayne Mery (:wsmwk)]

For the record, as far as i can tell there are no other open bugs on this issue http://bit.ly/a1eu2G

not many have commented on this point, so for clarity, does anyone disagree this happens only with "Leave messages on server" enabled?

has anyone reproduced this in version 3?

Comment 22

[Rob]

Since what it does is, repeatedly download messages left on server, even though you have already got them, it would not happen at all if 'Leave messages on server' were not enabled.

However, it will probably not be the _reason_ that the problem happens.

Comment 23

[Rob]

By the way, yes, I was on 3.0 the last time I had this problem. Received duplicates about 10 times for both mail boxes, that day (jan 21), and haven't had it since... I'm sure it'll happen again, and when it does I'll have the popstate logs.

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> wada are you saying there is nothing useful in the protocol log?

No. The log was useful - by log, we could know that no communication error, which can cause wrong LIST/UIDL response handling, occurred in your case.

Comment 25

[Rob]

Okay guys, I've had it again, and have popstate 'before and after' for you to look at. However, the pop3.log is about 28MB big unzipped. It 'll take hours to clean it up (the pc and e-mail program had been running for a long time)

Can someone please suggest a quick and easy way to only give you guys what you need from that file, without any mail contents and in a small  filesize?

thanks

rob

Comment 26

[WADA:World Anti-bad-Duping Agency]

Rob, attach two backups of popstate.dat first, please.

Comment 27

[Rob]

Attachment: before and after popstate.dat

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #27)
> before and after popstate.dat

On which account, did re-download of all mails happen?

By the way, I could observe re-download of mail by next.
  1. Open popstate.dat by text editor who opens file with write mode.
     (I used Sakura.exe)
  2. Send a mail to a POP3 account
  3. Get Msgs by Tb. mail is downloaded. (leave messaegs on server).
     Write to popstate.dat is prohbited by write open of text editor.
  4. Get Msgs by Tb. mail is downloaded again.
  As Tb opens with write mode, I think keeping "read open" by other program
  also interferes Tb's write mode open/read and following popstate.dat update.
Rob, did you get process monitor log for popstate.dat? (see comment #19)

Comment 29

[Rob]

(In reply to comment #28)
> (In reply to comment #27)
> > before and after popstate.dat
> 
> On which account, did re-download of all mails happen?
> 

On the Pandora account

> By the way, I could observe re-download of mail by next.
>   1. Open popstate.dat by text editor who opens file with write mode.
>      (I used Sakura.exe)
>   2. Send a mail to a POP3 account
>   3. Get Msgs by Tb. mail is downloaded. (leave messaegs on server).
>      Write to popstate.dat is prohbited by write open of text editor.
>   4. Get Msgs by Tb. mail is downloaded again.
>   As Tb opens with write mode, I think keeping "read open" by other program
>   also interferes Tb's write mode open/read and following popstate.dat update.
> Rob, did you get process monitor log for popstate.dat? (see comment #19)

I didn't. Could you clarify whether that's something I can set and forget, or is that something which I need to trigger when I expect the re-download to happen?

Comment 30

[WADA:World Anti-bad-Duping Agency]

What we need to know first : Does someone access popstate.dat? (other than Tb)
  Filter of Process Monitor:
    Process isn't thunderbird.exe, Exclude
    Pass ends with popstate.dat,   Include
There is no need of waiting for re-dowonload, POP3 log, backup of popstate.dat. Get log for several hours while you are using Tb. Open/close of popstate.dat is seen? If not seen, get log for one day, please.

Comment 31

[Rob]

(In reply to comment #30)
> What we need to know first : Does someone access popstate.dat? (other than Tb)
>   Filter of Process Monitor:
>     Process isn't thunderbird.exe, Exclude
>     Pass ends with popstate.dat,   Include
> There is no need of waiting for re-dowonload, POP3 log, backup of popstate.dat.
> Get log for several hours while you are using Tb. Open/close of popstate.dat is
> seen? If not seen, get log for one day, please.

Do you mean 'path ends with'? I'm doing that right now, will leave it running overnight.

Comment 32

[Rob]

While I'm running it, I realize that maybe you mean: process IS thunderbird.exe, exclude (don't you want to know what other processes access popstate.dat?)

Well, I see quite a number of accesses by ekrn.exe, which is my antivirus program.

Comment 33

[Rob]

Attachment: half an hour's worth.

I gave you half an hour's worth. Will let the program running during the night.

Comment 34

[WADA:World Anti-bad-Duping Agency]

Thanks for your correction. 
>     process IS thunderbird.exe,  exclude
>     PATH ends with popstate.dat, include

> half an hour's worth.

Read open by ekrn.exe every two minutes, and read data at offset=0.

(a) Can you get process monitor log for access to a popstate.dat by any process for a while?
> PATH ends with \Mail\<a_POP3_server_name>\popstate.dat, include
> "for a while" :
>   Longer than (two minutes * 2), at least two accesses by ekrn.exe
>   Longer than (NN * 2) where NN = NN of check for new messages every NN ...
>               at least two accesses by Tb upon new mail check

(b) While you are using Tb, activate process monitor log, open popstate.dat
    for a POP3 account by text editor, and keep text editor active.
    (POP3 account of smallest number of mails kept at server)
    And, send a mail to the POP3 account, execute Get Msgs several times.
    Is the sent mail downloaded multiple times?
    Does "re-download of whole mails" happen?

(c) Remove Tb's mail directory from directory list of anti-virus software for periodical virus scan, and watch for several days or a few weeks.
Will frequency of "redownload of whole mails" be reduced?

Comment 35

[Rob]

Attachment: Scenario B

Scenario B: I did not get the re-download. However, as long as I kept the popstate.dat file open, I received the warning that the folder was being processed.

Comment 36

[Rob]

(In reply to comment #34)
...
> (a) Can you get process monitor log for access to a popstate.dat by any process
> for a while?
> > PATH ends with \Mail\<a_POP3_server_name>\popstate.dat, include
> > "for a while" :
> >   Longer than (two minutes * 2), at least two accesses by ekrn.exe
> >   Longer than (NN * 2) where NN = NN of check for new messages every NN ...
> >               at least two accesses by Tb upon new mail check

I can try, but: while I was leaving the process monitor to run overnight, I got an out-of-memory error. I think this might be due to the fact that even though the filter shows only what I want, in the background every single process is being kept account of. So I'll not be able to run this program for more than a couple of hours at a time... Anyway I'm doing it, will take a look in a couple of hours.

> (c) Remove Tb's mail directory from directory list of anti-virus software for
> periodical virus scan, and watch for several days or a few weeks.
> Will frequency of "redownload of whole mails" be reduced?

I'll do that after scenario A.

Comment 37

[Rob]

By the way, I hope it's allright that I kept the standard process monitor filter, and added the above ones to it? Or do you think I should _only_ filter on what you request? (in that case I'll have to start over)

Comment 38

[Rob]

I left Process Monitor running for a long time now in scenario A. It doesn't register anything (not even when I open popstate.dat in notepad for a long time).

Will try scenario C now, but you'll have to be patient, I haven't had a double download between jan 22 and march 5... 

If you like, I could give scenario B another try with another program than notepad (but which one? feel free to suggest one)

Comment 39

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #35)
> Scenario B: I did not get the re-download. However, as long as I kept the
> popstate.dat file open, I received the warning that the folder was being
> processed.

No log line for thunderbird.exe. When did you open popstate.dat by notepad? Before first successful GetMsgs by Tb?

Notepad's access was next. This may permit write open by other program and write data to the file. Probably delete/rename only is prohibited.  
> "10:57:09,3813830","NOTEPAD.EXE","2212","CreateFileMapping",
>   "C:\Documents and Settings\Rob\Application Data\Thunderbird\Profiles\6s48t9ig.default\Mail\in.pandora.be\popstate.dat",
>   "SUCCESS","SyncType: SyncTypeOther"
> "10:57:09,3816020","NOTEPAD.EXE","2212","CloseFile",
>   "C:\Documents and Settings\Rob\Application Data\Thunderbird\Profiles\6s48t9ig.default\Mail\in.pandora.be\popstate.dat",
> "SUCCESS",""

> If you like, I could give scenario B another try with another program than
notepad (but which one? feel free to suggest one)

"Write open" and/or "Read open" by script is best, I think.
Can you install "Open Object Rexx" or download PHP/unzip/set PHP library in path of Win? You can request open interactively by typing Rexx code or PHP code, or keep open by batch type script by prompting after open in the script.

Comment 40

[Rob]

I have a rather surprising thing to report: the bug might not be due to a bug in thunderbird at all (at least for me): I just received a whole load of duplicate mails, while my colleague sitting next to me began to curse because he also received a whole load of duplicate mails. However, he's using Outlook.

I think we'll file a bug report with our ISP ;) probably forgets its guids once in a while or something like that...

Comment 41

[Rob]

(In reply to comment #40)
> I have a rather surprising thing to report: the bug might not be due to a bug
> in thunderbird at all (at least for me): I just received a whole load of
> duplicate mails, while my colleague sitting next to me began to curse because
> he also received a whole load of duplicate mails. However, he's using Outlook.
> 
> I think we'll file a bug report with our ISP ;) probably forgets its guids once
> in a while or something like that...

by the way, you other guys might try and setup a second mail client next to thunderbird (if only for a while), and check whether the bug is really thunderbird specific for all of you (or you get the double download in other clients too).

Comment 42

[Wayne Mery (:wsmwk)]

Milan, Martin, do you still see this issue in version 3?  And how large is your popstate.dat file?

Note: we won't be hearing from the reporter, Wayland's address is dead.

Comment 43

[Wayne Mery (:wsmwk)]

(In reply to comment #3)
> Should normally work. 
> 
> A pop3 protocol log might shed some light... see
> http://wiki.mozilla.org/MailNews:Logging

Pat & anyone who still sees this in version 3.1, please attach protocol log.
(and noting rob thinks his is a server issue)

Comment 44

[Rob]

Haven't seen the issue in a long while, and I have noticed that the ISP has moved to a different webmail backend (or at least front-end) lately. I never got any response on problems re. duplicate downloads, but maybe the mail guys are behind several levels (layers?) of support guys (cannon flesh) ;)

Comment 45

[Joshua Cranmer [:jcranmer]]

RESOLVED WFM per comment 44.

Comment 46

[Miles]

Have been receiving dupe emails since a week ago when upgraded from 3x to 5 from a pop server.   Very annoying to say the least.  This is the first time this has ever occurred since started with Netscape in the 90's.  Using Win7 64bit, pop server is Att/Yahoo.  Duplicates are not seen on the their mail website in Firefox.

See there is an extension that should alleviate this, but it's a bit dumb to install an extension to fix a bug.  A mozilla reads that it can be corrected by not leaving msgs on the server, but this is an action I don't want to use.  At this stage, the best choice for me is to regress to 3.x and forget about 5 until this is fixed.