849629 - Rule to copy mail on imap mail provider loses content, if action=Delete is requested in same filter rule

Status: RESOLVED DUPLICATE of bug 695671

(Thunderbird :: Untriaged, defect)

Description

[Peter Sherwood]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130215130331

Steps to reproduce:

I have created rules to copy some IMAP based mail to my local folders. These are messages I typically need to keep or I just simply don't want on my server. The rule is based on either the subject or the sender and is constructed as follows:
- Copy mail from imap server to local folder.
- Delete mail from imap server


Actual results:

For some messages (doesn't seem to be a reason which as they are very similar) the body is lost. The subject and sender are kept intact. The rest of the messages are copied properly. No error messages are seen or really any reason why it happened.
Sometimes it happens more, sometimes less again with no indication.
The previous rules on an IMAP server copied correctly. I use copy as move does not remove the  link on the server - perhaps move should? Anyway copy should be more reliable you'd think.


Expected results:

All messages should have been copied correctly.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to Peter Sherwood from comment #0)
> - Copy mail from imap server to local folder.
> - Delete mail from imap server

Filter action of "Delete" has problem. See bug 695671.
Many users perhaps use "action of Move from IMAP Mbox to local mail folder" for "Copy to local mail folder and Delete from IMAP Mbox" with auto-sync=Enabled/Offline-Use=On, because problem like yours is not reported frequently.
I suspect problem like next.
  - After header fetch, filter is invoked.
  - Copy is executed in Offline Copy mode(header informaion is copied).
  - Due to bug in Delete, filter execution abnormally ends.
  - Then "IMAP fetch BODY[] command to download mail to local folder"
    is not normally invoked.

Do following.
(1) Use IMAP delete model of "Just mark it as deleted" in Tb(Server Settings).
Unless EXPUNGE is executed for mails flagged as \Deleted, deleted mail is shown with strike-thru line. So, Undelete of the mail is always possible, unless EXPUNGE is executed.
See with Folder View=All, instead of Unified Foldeer view, to avoid unwanted problem, please.
(2) Show Order Received column(UID of mail if IMAP, Offset if local mail folder) to know UID of mail.
(3) To avoid repeated problem, change filter rule
    (split rule for different actions to avoid unwanted problem)
(3-A) If you want to continue to use Delete
  - Copy to other IMAP Mbox for safety in a rule with same condition
  - Copy from IMAP to local folder by different rule of same condition
  - Delete from IMAP by another different rule f same condition
(3-B) Change Delete to Move, with separate rule for Copy and Move.
  - Copy to other IMAP Mbox for safety in a rule with same condition
  - Move from IMAP to local by different rule of same condition
In any case, if copy to local was successfull, do Compact at Inbox, manually delete excess copy of mails in other Mbox(Shift+Delete, delete without copy to Trash) after a while, then do Compact at the Mbox.
If you have many rules, "edit of msgFilterRules.dat by Text Editor" may be an easy way to change many filter rules.

Questions about your environmnt, problem.

(Q1) When you experienced problem, was data in filterlog.html written correctly?
(See bug 695671 comment #11, and check bottom part of filterlog.html by Text Editor)

(Q2) Is auto-sync enabled? (Check via Tools/Options/Advanced/General, Config Editor)
  auto-sync enabled (N is server number of the IMAP account)
  - mail.server.serverN.autosync_offline_stores=true
  - mail.server.serverN.autosync_offline_stores doesn't exist
    mail.server.default.autosync_offline_stores = true
  auto-sync disabled
  - mail.server.serverN.autosync_offline_stores=false
  - mail.server.serverN.autosync_offline_stores doesn't exist
    mail.server.default.autosync_offline_stores = false

(Q3) Is Inbox Offline-Use=On folder? (Folder Properties/Synchronization)

Comment 2

[:aceman]

Peter, after the messages are copied, can you try right click on the target folder, choose properties and repair folder. Do the message bodies appear correctly now?

Comment 3

[Peter Sherwood]

Thank you for those suggestions, I will go through all the information and respond shortly. Ideally I'd like to repeat it with the actual messages and these will come through later today. Just in case it's message related. Please bare with me.

Comment 4

[:aceman]

Please backup the mail storage folders before doing the tests. See in account settings -> server settings -> local directory for the path where the mail is stored in the filesystem and backup (copy) those files somewhere safe.

Comment 5

[Peter Sherwood]

The message bodies do not appear when I repair the folder. The subject and sender friendly name disappear also. I am going through the other tips.

Comment 6

[Peter Sherwood]

The dual-filter-rule method is a suitable work around for me after about a couple of weeks of testing. It would be nice if the filter rules could be applied against a single folder, that would make the rule more accurate. However this has worked for me.

Comment 7

[Peter Sherwood]

And again thanks for the help!

Comment 8

[WADA:World Anti-bad-Duping Agency]

Comment #0.
- Copy mail from imap server to local folder.
- Delete mail from imap server
(In reply to Peter Sherwood from comment #6)
> The dual-filter-rule method is a suitable work around for me
> after about a couple of weeks of testing.

You look to have split rule for "Copy" and "Delete".
  initial: action=Copy to folder and action=Delete in same filter rule 
  current: action=Copy in a filter rule,
           action=Delete in different rule with same condition.
In design, "in single rule" == "in different rules", but they are slightly different in actual implementation, as seen in 398498, bug 742155. So, "split rule" may alter phenomenon.

My (3-A) "If you want to use Delete" in Commet #1 is;
  If you want to continue enjoying problems of that bug.

Problem of that bug is;
  When multiple mails arrive at same time,
  If filter rule of "action=Delete" fires on a mail#(N),
  action=Move and action=Delete
  (action of same level as MoveToFolder/Delete)
  by other rule or same rule
  on subsequent mail#(N*1) to mail#(Last),
  is not executed.

Because you don't look to use "action=Move to folder",
problem by that bug in your case is perhaps following only.
  When multiple mails arrive at same time,
  If filter rule of "action=Delete" fires on a mail#(N),
  action=Delete
  by other rule or same rule
  on subsequent mail#(N*1) to mail#(Last),
  is not executed.
So, problem is not so big.

However, if possible, change "action=Delete" to "action=Move to Trash folder", unless you are eager to enjoy problems by that bug.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Essentially dup of bug 695671. But defference between "Copy and Delete in single rule" and "Copy and Delete in different rules of same condition" is seen in this bug only.
Setting dependency to that bug for future analyss.