Last Comment Bug 359847 - Moving lager number of Messages from one imap folder to another resulted in multiple copies/duplicates
: Moving lager number of Messages from one imap folder to another resulted in m...
Status: NEW
Product: Thunderbird
Classification: Client Software
Component: Folder and Message Lists (show other bugs)
: 2.0
: x86 Windows XP
: -- normal with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
Depends on:
  Show dependency treegraph
Reported: 2006-11-07 11:18 PST by Scott MacGregor
Modified: 2011-10-03 04:20 PDT (History)
7 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description Scott MacGregor 2006-11-07 11:18:26 PST
Kerz was using version 2 beta 1 (20061106) on the Mac.

1) Ctrl-A on an Imap Inbox folder with 4000 messages in it.
2) Drag the selection to an online Archive folder to move them...


Many copies of the same message appeared in the archive folder. Instead of having 4000 messages he had about 40,000 messages. Each message was duplicated about 5-10 times in the archive. The number of dupes weren't consistent between messages.
Comment 1 David :Bienvenu 2006-11-07 11:54:37 PST
are these folders on the same server?
Comment 2 Jason Kersey 2006-11-07 14:46:30 PST
Same server.  The server is exchange, not sure what version, I can find out if you need it.  The only other thing not mentioned is after I did the move, I tried a number of times to go into the folder.  Each time I did, it would eventually time out downloading headers.  My only guess here is, each time it timed out, it added another copy of the messages it didn't download yet?  I can go back to the earliest messages I moved, which I assume would have their headers downloaded first, and there are only 4 or 5 dupes, where the messages I downloaded last have as many as 10-12 dupes, and took 3 or 4 attempts to download headers before I could see them.
Comment 3 David :Bienvenu 2006-11-07 14:57:35 PST
When we download headers, we only download headers for messages we don't have in the db. And since we store messages with the UID as a key, and you can only have one message with any given key, we can't get dupes of messages with the same UID. And when you select a folder, we look at the list of messages the server tells us about, and delete local headers for any message that is no longer in the server list of messages. 

So, it sounds like Exchange is doing something weird. An imap protocol log might tell us only did the move once?
Comment 4 Jason Kersey 2006-11-07 15:15:19 PST
Only did it once.  I'm on a mac, let me know if there's somewhere to pull logs from.
Comment 5 David :Bienvenu 2006-11-07 15:21:17 PST
ah, the mac...the only way I know to generate protocol logs on the mac is something like this:

open a shell
export NSPR_LOG_FILE=imaplog.txt

then cd to the Thunderbird directory and run thunderbird from the command line.

It should create imaplog.txt in the thunderbird directory - you can copy it away after selecting the archive folder, and we attempt to download messages from it.
Comment 6 Pavel 2008-09-29 15:27:26 PDT
Confirmed as repeatable with the following configuration:

OS: Windows XP SP2 + hotfixes
Client: Thunderbird
AntiVirus: Symantec 10
Mailbox: IMAP
Mail server: Courier IMAP 4.3.1

Case 1: User moved 7323 messages from their Inbox to a subfolder. I/O (writes) on the server pegged. IMAP server had to be stopped/restarted - it seems that enough commands were queued up. Quadruplicates were found.

Case 2: User created the subfolder and cleanly moved the same 7323 mail messages using Netscape 7.1 Mail client. User then accessed the IMAP mail account via Thunderbird and deleted the subfolder (User has "Move messages to Trash" option selected as the action for message deletion). I/O on server spiked. IMAP server was allowed to run for a long time (multiple hours). Sysadmin got tired of the shenanigans and restarted the IMAP processes after killing Thunderbird on the client's machine. 278 copies of each of the 7323 messages were found.
Comment 7 Pavel 2008-09-30 06:46:35 PDT
Additional data point:

Thunderbird, Windows XP, AVG A/V (email scanning disabled). Attempting to move 800 messages does *not* trigger the bug.

Will try to see if there is a magic threshold which triggers the shenanigans.
Comment 8 Pavel 2008-09-30 10:39:18 PDT
Test setup: mail folder with 15,794 messages. A subset was selected and then RH-Click -> Move To -> {Empty IMAP Folder} was selected.

Results: - Moved the 11,000 messages without an issue, albeit slow. Curious note: Claws Mail indicated that the original mail messages were marked for deletion LONG before the display fully updated on (on the order of minutes). - Created triplicates in the new folder, would likely continue making copies if allowed to run. Never finished marking old messages for deletion. - Created (roughly) triplicates in the new folder, would likely continue making copies if allowed to run. Never marked original messages for deletion.
Comment 9 Pavel 2008-09-30 10:58:50 PDT
Under logging (as per in, unable to duplicate bug.
Comment 10 Pavel 2008-10-06 10:53:58 PDT
Bug duplicated under Thunderbird on CentOS 5.2
Comment 11 Pavel 2008-10-08 06:45:37 PDT
Additional note: All these tests were done on multi-core boxen.
Comment 12 Jan Eldenmalm 2008-12-06 17:31:20 PST
Running: (20081105)
on Win XP with all latest service packs today.
Server: Communigate Pro 5.2.10

Moving messages creates duplicates when "the move" takes more than X (unknown) number of seconds, the status window flickers and it "restates" that messages are being moved.

Last test: attempted to move approximately 300 mails from "INBOX" archive folder on the same server/same account. Resulted in 15964 messages.
Comment 13 Jan Eldenmalm 2008-12-06 17:41:21 PST
Additional note: the "moved" messages were never removed from Inbox, and I manually interrupted the operation by shutting down thunderbird.

Moving up to approximately 150 emails work fine.
Comment 14 Pavel 2009-03-06 10:31:08 PST
2.0.19 under Vista (64bit) seems OK.
Comment 15 Marcel Bülles 2009-09-29 17:16:01 PDT
Confirmed as repeatable with the following configuration (see comment #6, similar):

Windows XP Home Edition SP 3
Client: Thunderbird.
Antivirus: Antivir, Comodo Free Firewall
Mailbox: IMAP

When moving emails from main folder to newly created "archive" folder quadruple copies exist. Applicable to all emails in folder irrespective of order. Inet provider states that issue is bug with Thunderbird not server-side.
Comment 16 Marcel Bülles 2009-09-30 02:10:56 PDT
Addition to comment #15:

Mail server: Dovecot 1.0.15
Comment 17 Tobias Bußmann 2010-09-01 23:47:17 PDT
I also can confirm this bug. 
Reproduced moving about 40k messages form INBOX to a subfolder. Messages duplicated about 3 times. 

TB on Mac OS X 10.4.11 with a courier-imap server 3.0.8

Note You need to log in before you can comment on or make changes to this bug.