610131 - Moving a large message or multiple messages between IMAP folders causes TB to hang/crash, results in duplicate messages

Status: RESOLVED DUPLICATE of bug 296453

(Thunderbird :: General, defect)

Description

[Chris Boshuizen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.11 (KHTML, like Gecko) Chrome/9.0.570.1 Safari/534.11
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101027 Lightning/1.0b2 Thunderbird/3.1.6

TB seems unable to move large messages (or a large volume of messages) between IMAP folders. The program essentially crashes midway through the process. If multiple messages are moved, some of them are copied to the new folder, but not deleted in the originating folder, and so duplicates are created. 

The behaviour is worse on slower internet connections.



Reproducible: Sometimes

Steps to Reproduce:
1.select a large message in message list (or select a large number of smaller messages)
2. type "A" for archive, or drag and drop to another folder
3. Thunderbird begins the copy process
4. Copy never finishes (large message is not copied, or the list is only partially copied)
5. delete of messages in originating folder never happens, so some messages are now in both folders.
6. Thunderbird becomes unresponsive, needs to be force-closed.
Actual Results:  
1) TB crashes
2) sometimes some of the messages are copied, but not deleted from the source folder, so there are now two copies of these. There is no easy way to tell which messages were duplicated.

Expected Results:  
Messages should be moved without a crash.
The move process should take into account the potential for network dropout or other interruption, and not duplicate messages. This means moving them one at a time, not doing a two step process involving a bulk copy then bulk delete.

Might be similar to:
https://bugzilla.mozilla.org/show_bug.cgi?id=583365

Comment 1

[WADA:World Anti-bad-Duping Agency]

I looks combination of next phenomena.
(1) Huge momery use by bulk move of IMAP folder due to bug 538378, then
    slowness like bug 583365 comment #1 happened by swap file expansion of OS.
    Unresposiveness of Tb occurs too in such stuation.
(2) You killed Tb in such situation, then crash like bug 583365 comment #0
    occurred(killed by OS in that case, because swap file can't be expanded)
(3) As you say, "bulk move" is roughly "bulk copy"+"bulk delete(store flag
    \Deleted)". As you killed Tb during "bulk copy" phase, "bulk delete" phase
    was not executed.

Can you observe crash without killing of Tb?

Comment 2

[Chris Boshuizen]

Actually it always locks up. In Win7 the main app window and all open compose windows freeze and become unresponsive. Clicking on them prompts the windows kill dialog. Checking the task manager shows either sustained 100% CPU, or 0% CPU -- I see either on occasion. I usually wait for several minutes to let it finish its thing, but it never comes back. If I leave TB running overnight, it is in an identical state when I check in the morning -- unresponsive, and sustained 100% or 0% CPU. If I check my IMAP folders online, no changes are made from the moment of the lock up.

Comment 3

[Chris Boshuizen]

I should add, I regularly receive email attachments between 1MB and 20MB, so I see this type of lock up daily.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)

Problem itself with IMAP is dup of bug 538378.
"bulk move of IMAP" looks to produce critical problem than "bulk move of local folder". If auto-sync is enabled for move target folder and IDLE is enabled, download by auto-sync can be initiated during the "bulk move". If Gloda is enabled, indexing after download may be initiated. They make situation worse.

(In reply to comment #3)
> I should add, I regularly receive email attachments between 1MB and 20MB, so I
> see this type of lock up daily.

I think mail size itself is basically irrelevant to problem and "number of mails" is key factor, although mail size can affect on required memory size if "viewing of moved mail" is involved.
How many mails do you move by one operation regularly?
How long does the "daily move" take?
What is Virtual Memory Size and Real Memory Size used by Tb while the daily bulk move is executed(and Tb is unresponsive)? No swap file expanding by OS during the daily bulk move?

bug 538378 is probably mainly for huge memory usage while "UID COPY ... target" and "UID STORE ... flag \Deleted".
What is your IMAP delete model? Do you enable option like "expunge after each delete"?
Is deep thread and/or long thread is involved in you daily bulk move?
If auto-expunge is scheduled and deep/long thread is relevant, bug 452221 also can occur after "UID COPY ... target" and "UID STORE ... flag \Deleted" and EXPUNGE.

Comment 5

[Michael Eilert]

Hi all,
I'm a sysadmin for several small enterprise environments.
We have massive problems with this known bug of Thunderbird.
 
In order to keep actual system configuration I am seeking help of a professional.

Is there at least any workaround, if the real cause itsself cannot be found??
My costumers would even make a donation to get rid of that problems.

No other mail client behaves like that. (tested pegasus, emclient, opera, even during the "catastrophies" these behave relatively normal. ...)

################ the problem #######################
 
One mail-users action trying to archive about 7000 old mails caused about 38000 mail duplicates and the server to hang.
This happens similarly almost every week. 

Restarting dovecot or the hole server doesn't help, in one case restart of apache2 helped a bit, I have no idea why.
 
During this situation every other user sending an email gets a message , that saving to sent-folder failed, asking him to retry, 
But this is not true, the mail is already in the right place in the sent folder. This causes more and more duplicates and more an more traffic.

Some user experience errors while writing mail bodies, that saving to drafts fails , others loose saved drafts.
 
We use Thgunderbird, because there's no alternative mail client supporting imap, caldav & ldap Adressbook
(evolution win32 was reported to do so, but doesn't function at all)
 
I've found many listed bugs in bugzilla that in my opinion are caused by one very basic misbehaving of TB.
 
If it does not get answer from the server within a certain time, it like a bull in a china shop it blindly shoots up to 5 times the same transaction to the server, not beeing able to handle any of the servers feedbacks.
 
This seems to be a very old bug , i found at least 3 year old duplicates, with TB 3 ist has become much more severe.
 
############## the system environment ###################
 
Server: Ubuntu 8.04 lTS with dovecot 1.2 maildir , postfix, openldap, davical, apache2
 
65 Mail Accounts , up to 25000 mail per user , some of them have mutliple cascading subfolders.
all together 200 GB and 600000 mails.
 
50 Clients  winXP Sp3   Thunderbird 3.1.9 with Lightning
Several Clients are over 5 years old
 
network: 100Mbit/s , 2 main central switches have 1000 MBit/s

############### some of the bug IDs that sound similar to me #######
This is a lot of stuff , delete it if you like:
492672 	nor 	-- 	All 	[e-mail] 	NEW 	--- 	Need to adjust IMAP dynamic timeout algorithm to compensate for both slow servers and large emails
583365 	cri 	-- 	Wind 	[e-mail] 	NEW 	--- 	Memory leak ending in crash when moving 70k emails from inbox to archive (large real memory is required for bulk move/delete of mails, and real memory used by "select+all and move/delete" is not released until new Tb window open and old Tb window close)
476074 	maj 	-- 	All 	[e-mail] 	UNCO 	--- 	deleting or moving large numbers of messages at once very slow with high cpu usage
482368 	nor 	-- 	Mac 	[e-mail] 	UNCO 	--- 	Mail deleted from preview pane becomes unread in Trash folder
505443 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Deleted messages (two folders only) re-appeared en mass [pop]
519083 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	message duplicated when copied or moved from filter
537192 	cri 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Folders moved from Local Folders to Archive disappear, but can still be referred to in the menu MESSAGE>MOVE TO [?] AGAIN, and any use of that menu item causes more messages to disappear (even from the archive)
538591 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Copying imap folders to local folders by dragging doesn't copy all of mails in IMAP folder/subfolders
539494 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	IMAP Cache Should be Stored in "Local Settings" Application Data Folder (IMAP Cache: IMAP offline-store file by offline-use=on, never Disk Cache)
541669 	cri 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Moving emails creates 1000's of duplicate emails (IMAP)!
545082 	maj 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Filters filing to IMAP folders not running automatically (quarantine option=on/mailnews.downloadToTempFile=true, "MOVE to IMAP folder" action does nothing)
549405 	maj 	-- 	Wind 	[e-mail] 	UNCO 	--- 	inconsistent message list when using IMAP and a webmail interface successively
553564 	nor 	-- 	Mac 	[e-mail] 	UNCO 	--- 	Delete mail button stops working
556232 	cri 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Cannot move a message from a folder to another one.Sent email (or other emails) can't be copied to sent folder. Draft cannot be saved. "There was an error copying the message to the Sent folder. Retry?"
556749 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	deleting multiple mails at a time fails to delete all the messages you have selected
566342 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	When Unified view of several IMAP accounts switched on, a lot of the mails don't appear in global inbox - just the local ones
584655 	nor 	-- 	Wind 	[e-mail] 	UNCO 	--- 	messages can't be deleted or moved
590791 	maj 	-- 	Wind 	[e-mail] 	UNCO 	--- 	TB not consistently updating read/unread status (and other header flags) across multiple open TB clients viewing the same Inbox, despite all TB clients receiving the required real-time IMAP IDLE "push" of flag changes
610131 	cri 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Moving a large message or multiple messages between IMAP folders causes TB to hang/crash, results in duplicate messages
611515 	maj 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Moving folder with subfolders causes problems when subfolders have not been previously opened by TB
616694 	nor 	-- 	All 	[e-mail] 	UNCO 	--- 	Multiple running thunderbirds duplicate IMAP mails
620123 	cri 	-- 	Wind 	[e-mail] 	UNCO 	--- 	Freeze on send with PDF attachment
624082 	maj 	-- 	Linu 	[e-mail] 	UNCO 	--- 	IMAP: Moved messages disappear
216442 	cri 	-- 	All 	[e-mail] 	NEW 	--- 	group move of messages from IMAP Inbox to local folder loses mails
261837 	nor 	-- 	All 	[e-mail] 	NEW 	--- 	IMAP: messages marked as deleted are deleted permanently on exit
274576 	nor 	-- 	Linu 	[e-mail] 	NEW 	--- 	while moving more files from IMAP folder to local, only part is transferred
296453 	maj 	-- 	All 	[e-mail] 	NEW 	--- 	Racing CPU, slow performance moving/deleting/dragging large number of messages
227995 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	Fails to move copy to Sent folder, then claims message wasn't sent, but it was
328717 	nor 	-- 	All 	[e-mail] 	NEW 	--- 	IMAP folder being cached locally although not told to do so, seems never to be cleaned up
359847 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	Moving Messages from one imap folder to another resulted in multiple copies
398684 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	Virtual memory size of Thunderbird increases 40MB after each "Shift+Delete of all 40,000 mails and Compact folder"
488282 	maj 	-- 	Mac 	[e-mail] 	NEW 	--- 	Problem with copies placed in exchange sent message with special characters (é, à,..)
496254 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	New mail alert says "N new mails" even if all downloaded N mails is moved to local Junk folder by "Junk filtering for message-filter-moved mails" (POP3)
499634 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	"Copy/Move mails to [Gmail]/Trash" always creates local offline-store file
506509 	nor 	-- 	Wind 	[e-mail] 	NEW 	--- 	Memory leak when deleting with large amounts of mail
526490 	nor 	-- 	All 	[e-mail] 	NEW 	--- 	Mail not appearing when moving mail to IMAP folder and then immediately switching to it (race)
535070 	cri 	-- 	Wind 	[e-mail] 	NEW 	--- 	hung 100% cpu sending large attachment, UI thread blocked on http
551428 	cri 	-- 	Wind 	[e-mail] 	NEW 	--- 	crash moving folder to imap account from another account [@ nsImapMailCopyState::`scalar deleting destructor''(unsigned int)]
617839 	maj 	-- 	All 	[e-mail] 	NEW 	--- 	Switching to threads with a large number of emails is extremely slow
618553 	nor 	-- 	Mac 	[e-mail] 	NEW 	--- 	The compose window saves multiple drafts per message in the Gmail Drafts IMAP folder
506809 	nor 	-- 	Wind 	[e-mail] 	ASSI 	--- 	Slow IMAP delete immediately performance, shift+delete
378582 	nor 	-- 	Wind 	[e-mail] 	ASSI 	--- 	Filter which moves messages from IMAP account to Local Folder prevents new mail alert