519083 - message duplicated when copied or moved from filter

Status: UNCONFIRMED

(MailNews Core :: Filters, defect)

Description

[toddgleason]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4

I'm using IMAP.  Messages that are moved from my IMAP account to my Local Folders are now getting duplicated.  Most messages I instead copy to my Local Folders and also move to a "recent" IMAP folder for easy viewing from my phone; these are also duplicated within the Local Folders, but not within the IMAP account.  (Within the filter, I do the copy before the move.)


Reproducible: Always

Steps to Reproduce:
1.  Create IMAP filter with a copy to Local Folders and a Move to another IMAP folder.
2.  Receive mail on that filter.

Actual Results:  
Get extra message in Local Folder.

Expected Results:  
Get one message in Local Folder.

This problem is new as of Thunderbird 3.0 Beta 4.  Beta 3 handled it properly.  Beta 2 had some issues with the reliability (some messages only go moved, and some only got copied).  It's very frustrating and I'm wondering if there is a possible misconfiguration.

Comment 1

[Kent James (:rkent)]

I was not able to duplicate this on my system.

The best thing that you could do to help confirm this would be to create a new profile with no extensions or existing filters, just create this one filter, and try some test messages. Use a very simple filter criteria, like Subject Contains. Then tell me the result.

Is that possible? If not, maybe you could just give me a few more details, such as exactly what the filter is, and what type of IMAP server you are using.

Comment 2

[toddgleason]

I think I have traced this down to having an unrelated filter present that searches for text in the body of the message.  So this extra filter would look like this:

name="BodyFilter"
enabled="yes"
type="17"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/Folders/BodyFilterFolder"
action="Move to folder"
actionValue="imap://user%40domain@server/Recent"
condition="OR (body,contains,somethinginthebody)"

With a filter that looks something like this added into the mix, I get the message duplicates.  I will for now just remove the part of the filter that searches based on the body text.  Can you confirm that this can be duplicated based on the above?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)

"Both copy & move in a filter rule" case. It's Bug 448337(fixed by Tb3.0b4), which you already knew.

Your case is "copy to local mail folder + move to IMAP folder".
There may be remaining issue in such case. Move to same IMAP account's folder? Or Move to different IMAP account's folder?
Bug 376235 is already fixed(Bug 448337 is problem even after fix of Bug 376235).
Does "copy and move in different filter rules" work well?

Comment 4

[Kent James (:rkent)]

IMAP body filters are new in TB3 beta 4, so I'll need to look at this more carefully.

I'm not sure what you mean by an "unrelated" filter though. The more details you can give of your precise setup the better, though I will try to duplicate anyway.

If you could narrow it down to 2 only 2 filters enabled that are necessary to create the problem, then give me the details of those filters, it would be helpful. I assume that comment 2 is one, what is the other?

Comment 5

[toddgleason]

The first filter involved would be the one that actually gets triggered.  In my case I made it act based on an exact "from" address, and its action was essentially the same as what is in comment #2.  Here it is, roughly:

name="From contains: sender@domain"
enabled="yes"
type="17"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/Folders/TestFolder"
action="Move to folder"
actionValue="imap://user%40domain@site/Recent"
condition="AND (from,contains,sender@domain)"

I don't believe it matters whether you put this before or after the filter from comment #2.  The filter in comment #2 is "unrelated" in that it doesn't get triggered, but apparently it gets evaluated in some way.

I also don't think this is a copy + move issue, because if you delete the "Move to folder" part of the rule above and change the copy to a move, you still get 2 copies.  At least this was what I saw for my catch-all whitelist rule that files everything unrecognized as spam.  (I started seeing duplicate spams, and those are only moved from IMAP to local spam folder, rather than being copied and moved.)  Because of this, I don't see a reason to split apart my copy and move steps into separate rules.

Instead I would characterize this as being a problem that occurs when you copy OR move a message from IMAP to a local folder.  So to reiterate how you reproduce and what you see:

1.  Have filter #1 that triggers based on body text (I doubt its actions matter, but maybe they do).
2.  Have filter #2 that triggers based on sender (I doubt it matters how this one is triggered), that copies or moves message from IMAP to a Local Folder.
3.  Receive message that triggers filter #2.
4.  Message appears twice in Local Folder.

Caveat:  I did not test this with just two filters.  I instead built up my original set of filters until the problem recurred, and experimented with the filter (#1 in the case above) that made the problem appear until it was obvious what was different about that filter.

Note that I did not see this in Thunderbird 3.0 Beta 3, but my rules that file based on body probably were never even getting triggered.  And those rules were written way back when I was using POP and Thunderbird 2.

Comment 6

[Kent James (:rkent)]

I'm seeing this now on my development system, for a simple case of a single filter Subject Contains "test", with actions Copy To (a local folder) followed by Move To (an Imap folder), receiving a matching message in an imap account.

I'll investigate.

Comment 7

[Kent James (:rkent)]

OK in my case, in the test profile I had enabled applyIncomingFilters to all IMAP folders (a hidden preference or folder property) as a test of another bug, so the second copy was correct: it made the first copy when the message hit the Inbox, and the second copy when the moved message hit the new folder.

I removed that preference, and I can no longer duplicate this. I also added the body filter as you suggested, and I still cannot duplicate. So I am unconfirming, and removing my assignment.

Todd, I can't imagine that you are using applyIncomingFilters to the IMAP folder, as this is an undocumented hidden preference. Or are you?

Comment 8

[toddgleason]

I didn't see applyIncomingFilters in my prefs.js.  The only prefs from there including the word "filter" are:

user_pref("mail.server.server1.useServerFilter", true);
user_pref("mail.warn_filter_changed", false);

Looking for "move" and "copy" I also find:

user_pref("mail.server.server1.moveOnSpam", true);
user_pref("mail.server.server1.moveTargetMode", 1);
user_pref("mail.server.server8.moveOnSpam", true);
user_pref("spamato.moveSpamMessages", "true");
(Note that Spamato is disabled)

And for "apply" I find:

user_pref("mail.server.server1.applyToFlaggedMessages", false);
user_pref("mail.server.server3.applyToFlaggedMessages", false);

I don't know if any of these are remotely helpful or not.

Comment 9

[Kent James (:rkent)]

(In reply to comment #8)
> I didn't see applyIncomingFilters in my prefs.js.

I didn't think so.

Todd, I would be happy to investigate this if I could reproduce it. Perhaps you could follow my suggestion in comment 1 to try to narrow down the conditions under which this occurs.

Comment 10

[toddgleason]

I agree that may be necessary.  Can you point me to the way to do this with the minimum effort, no data loss, and easy restore to the original profile/configuration?

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to comment #10)
> I agree that may be necessary.  Can you point me to the way to do this with the
> minimum effort, no data loss, and easy restore to the original
> profile/configuration?

todd, see  

http://kb.mozillazine.org/Testing_pre-release_versions#Using_profiles_to_manage_risk

Comment 12

[Mike Cowperthwaite]

I am seeing a similar symptom to this with 
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.9) Gecko/20100419 Shredder/3.0.5pre

In my case, the filter is moving bugmail from IMAP and storing it in a local folder.  I'm not 100% sure of this yet, but the wrinkle appears to be:  if a single message arrives, it's not duped; if multiple messages arrive (in the same fetch?) all but the first one is duped.

Comment 13

[Marcus Groeber]

Also seeing this here with "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9".

In my case, I use a set of filters to move e-mails with certain keywords from an IMAP4 inbox to Local Folders. In particular, there are two unusual filters that could be related to this:

- One (the second-to-last in the order, called "Macus") that checks whether the "To" or "Cc" contains my first name, and if so, stops processing rules

- Another one (the last in the list, called "Rest") which unconditionally moves every mail from the IMAP4 inbox to the Local Folders inbox.

The idea of the setup is to leave only "important" mails addressed directly to me on the server, and to filter out all bulk traffic into local folders. This works well, but, as described above, the "Rest" rule often creates multiple copies of an unread e-mail in the Local Folders inbox.

This is also logged in the filter protocol (unfortunately, in German) - here is the last session:

---
Filter "Bulk" auf die Nachricht Lidl Newsletter <[e-mail]> - Kein Aprilscherz - 14 Tage Thailand-Rundreise ab 799 Euro p.P. mit Lidl-Reisen am 01.04.2011 19:32:36 angewendet Nachricht (ID= E5-8o7aumek-nr10/1/[e-mail]) verschoben nach mailbox://nobody@Local%20Folders/Inbox/Low-bulk

Filter "Bulk" auf die Nachricht Lidl Newsletter <[e-mail]> - Kein Aprilscherz - 14 Tage Thailand-Rundreise ab 799 Euro p.P. mit Lidl-Reisen am 01.04.2011 19:32:36 angewendet Ausführung gestoppt

Filter "Bulk" auf die Nachricht ALDI Newsletter <[e-mail]> - ALDI Multimedia Angebote: 43,9 cm/17,3" Entertainment Notebook, 47 cm/18,5" LCD-TV mit integriertem DVD-Player, 8,89 cm/3,5" Navigationssystem, Digitaler HD Satelliten Receiver, 12.0-MP-Slim-Digitalkamera, DO, 07.04.2011 am 02.04.2011 09:31:25 angewendet Nachricht (ID= [e-mail]) verschoben nach mailbox://nobody@Local%20Folders/Inbox/Low-bulk

Filter "Bulk" auf die Nachricht ALDI Newsletter <[e-mail]> - ALDI Multimedia Angebote: 43,9 cm/17,3" Entertainment Notebook, 47 cm/18,5" LCD-TV mit integriertem DVD-Player, 8,89 cm/3,5" Navigationssystem, Digitaler HD Satelliten Receiver, 12.0-MP-Slim-Digitalkamera, DO, 07.04.2011 am 02.04.2011 09:31:25 angewendet Ausführung gestoppt

Filter "Rest" auf die Nachricht "Dr. Lara Koenig" <[e-mail]> - 50 Kilo in einem Jahr weg am 02.04.2011 10:43:58 angewendet Nachricht (ID= [e-mail]) verschoben nach mailbox://nobody@Local%20Folders/Inbox

Filter "Rest" auf die Nachricht - am 01.01.1970 01:00:00 angewendet Nachricht (ID= ) verschoben nach mailbox://nobody@Local%20Folders/Inbox
---

Note the last entry, which seems to represent the duplication from the e-mail from "Dr. Lara Koenig" (which happens to be spam).

I am not absolutely certain when this started happening, but I believe it coincided with switching the profile to the Thunderbird 3.0 release.

Comment 14

[Wayne Mery (:wsmwk)]

do you all still see this in current versions?  (9, 10, etc)

Comment 15

[toddgleason]

I see it intermittently...it seems like every several days or so, I see it happening, and only on some of my emails. They can be filed with the very same rule but some will duplicate and others won't.

Also, sometimes I see blank versions of emails. This can happen independently of whether the email got duplicated in the folder it was copied to. Sometimes it happens for one copy, sometimes for neither, and sometimes it happens when the email was not duplicated. However, I never see this on the IMAP server where I moved the original email.

Comment 16

[toddgleason]

Oh, I only just upgraded today to version 10 and have not yet seen the problem recur. I was running version 9 before and saw the problem yesterday and this morning.

Comment 17

[toddgleason]

I just saw the bug again, running the latest (10.0.1).  A message was copied (incorrectly) twice into the local folder, and moved (correctly) into the IMAP folder.

Comment 18

[x471]

TB 11.01 / Windows 7 prof. / roughly 50 filters / IMAP 

filter "g12" moves message from IMAP to a local folder - like expected.
Then it feels like From, To, ID and Date is lost or changed to 01.01.1970
and that matches to my last two filter "bcc before 2012" and "bcc before 2013"
Three filter are executed - only one should, 
because in IMAP no messages is older than 5 weeks 
and it is happening with a Testmail from me to me

Filter "g12" auf die Nachricht "LT" <[e-mail]> - Test am 04.03.2012 19:05:12 angewendet Nachricht (ID= [e-mail]) verschoben nach mailbox://nobody@Local%20Folders/2012

Filter "bcc12" auf die Nachricht - am 01.01.1970 01:00:00 angewendet Nachricht (ID= ) verschoben nach mailbox://nobody@Local%20Folders/bcc/2011

Filter "bcc13" auf die Nachricht - am 01.01.1970 01:00:00 angewendet Nachricht (ID= ) verschoben nach mailbox://nobody@Local%20Folders/bcc/2012 


Since 2008 everything was fine 
until I created my first filter "move-from-imap-to-imap-subfolder" (with Bug 735940)
msgFilterRules.dat is restored (mv to local only) but messages still got multiplied.

Comment 19

[x471]

run filter on folder  : only "g12" is executed - like expected.
run filter on message : "g12" and "bcc12" and "bcc13" is executed.
It happens with or w/o action "Filter stoppen" on a Test-Mail from me to me.
"bcc12" and "bcc13" should not be executed because both say "not-from-me AND not-for-me"
They are executed because no Sender nor Receiver is seen. ( see Comment 18 )

Comment 20

[x471]

workaround for the moment:
additional condition in each filter : "after 02.01.1970"
now I can "run filter on message" without duplication.

Comment 21

[HolgerG]

confirming this behavior for Linux TB 12.0.1 and IMAP:
I have ~30 filters defined a last "catch what's left over" filter that moves messages from the IMAP inbox to a local MyInbox folder. Anything already moved gets picked up.
The filter log shows the miraculous "at 01.01.1970 01:00:00 moved" messages from above. (Well, this is a English setup) 

This started to appear w/ TB 12 for me. TB 3x-TB 11 behaved well w/ the same filter setup of using a "Catch all" filter having "match all messages" enabled. even using one of the other two options (match following.. / match any of..)  doesn't change the behavior.

My take would be that the "Move To Folder" action doesn't correctly mark the message as deleted in the IMAP folder, but only sets some date/time fields....

Comment 22

[Marcus Groeber]

What works for me (inspired by comment 20) is to have a single rule at the top which checks for "Date is 01.01.1970" (turned into NaN.NaN.NaN when opening it again, interestingly). This rule has "Stop filtering" as the only activity.

In my case, this issue was already present in earlier versions of TB (also with an IMAP acount, but on Windows), and has been bugging me for several years now...

A similar issue, also with the "magic" date, is reported in bug 752237.

Comment 23

[HolgerG]

I actually can drive me somehow to see bug 752237 happening here as well. Didn't drill it down - it happend when trying to find a solution for this issue here.

Comment 24

[Todd C. Gleason]

I've never seen the dates get messed up with this happening to me, and it's not on every message, it's inconsistent. Restarting Thunderbird helps it go away for awhile.

Comment 25

[Chen Wei]

I have only one filter, which is for gmail, when match found, it will copy the new mail to a local folder, and move this new mail to Gmail/Trash imap folder so it will be deleted from gmail.  It works, except it always create at least 2 copies in the destination local folder. I suspect it has something to do with my slow internet connection to gmail server, but I can't prove it. Also, the filter should works regardless the speed of connection.

The system is thunderbird ESR 17.0.2 on Linux, the filter only check subject field started with "xxxxx"

Comment 26

[Chen Wei]

it looks like when Thunderbird moves mail to another IMAP folder, the mail arriving on the destination folder also triggers the Filter.

The possible solution might be restrict the filter to certain folder only.

Comment 27

[monperrus]

Happens to me on TB 45.0a2.

Interestingly, messages can be duplicated twice (IMAP account #1->IMAP account#2 -> local folder) which means I get 4 (2*2) versions of the same message in my local folder.

Comment 28

[monperrus]

Still this bug on EarlyBird 47.0a2 (2016-03-28).

In addition to bug 752237, it seems related to bug 557548 as well.

Comment 29

[Wayne Mery (:wsmwk)]

Can you still reproduce this issue?

Comment 30

[monperrus]

> Can you still reproduce this issue?

I am not able to try anymore. I solved all the filtering issues, incl. the duplicate message one, by moving back from maildir to mailbox.

Comment 31

[toddgleason]

It still happens reliably for me, though the exact behavior may have changed over these many years. I've been getting one copy in the IMAP folder, one in my local folder, and one in my Junk folder for a lot of emails. I still don't have any idea why. And I have no idea what maildir vs. mailbox means to Thunderbird.

Comment 32

[gitman]

I have a filter which does a "Mark as read" and then a move to another IMAP folder. I am sometimes getting duplicate copies of these filtered messages. This only happens on filters in which I have a "Mark as read" step.

I tried to switch the "Mark as read" to after the move step, but Thunderbird warns me that the execution order still dictates the "Mark as read" step will occur first.

I have a couple of older TBird installs (17.0.4, and 24.2.0) where the problem is not occurring. Only when I have my 52.5.2 install open do the messages get duplicated.

Comment 33

[toddgleason]

I think I may be seeing what's changed, in my case. I suspect the original problem went away, but I've been using Thunderbird for a long time, and I think at some point a filter option called "Stop Filter Execution" came into being. I missed it, for the most part. (I had just one filter that had it set.)

I ended up going through my 1200-line-long msgFilterRules.dat with a macro to add the filter into all my rules. I think the problem was that rules would match, but then it would fall through to my bottom-most catch-all white-listing rule and throw the message into junk, leaving me with either one copy in the right folder and one in Junk (if a rule matched), or sometimes two in Junk. I'll see whether this improves the Junk and duplicates situation. I'll see about updating again once I've seen how well this works.

Comment 34

[toddgleason]

So far, this isn't looking like a fix. I had an email go both into the destination folder and into Junk. The rule that puts it in its own folder says to:

1. Copy to the local folder
2. Move to an IMAP folder
3. Stop Filter Execution

The filter runs when Getting New Mail, with the setting of Filter before Junk Classification.

I have no idea what made the message also go into Junk.

Comment 35

[toddgleason]

Most of my filters don't end up with emails in Junk though. What I see mostly is that my final filter, which sends to Junk (whitelisting, in other words), results in 2 copies in Junk. The rule says to mark it as junk, move it to Junk, and stop filter execution. I tried breaking up the filter into one, to mark as Junk, and another, to move it and stop execution, but it didn't help.

Comment 36

[JimDeadlock]

This just started happening for me. I have a simple search-body filter that moves the message to a local folder (no other filters for that mailbox). It was working fine until yesterday when it suddenly started duplicating the messages in the local folder. I've made no changes to Thunderbird or filters lately. Restarting Thunderbird doesn't help. TB v. 52.7.0 (64-bit) on Ubuntu 18.04, server is running maildir. Adding the date filtering workaround seems to work.

Comment 37

[Stephen]

Still seeing this issue as described in my comment on Bug 752237.

I am not using a Unified Folder View so with hindsight this would have been a better place to post the comment. The workaround described above appears to solve the issue.

Comment 38

[toddgleason]

I am also still seeing this issue, but it is intermittent. Here are two messages that end up in the Junk mail. The first one appears just once. The second is duplicated. The rule execution in filterlog.html looks identical to me for the actual messages, but it looks as though for the second one, it believes it is also seeing a message with no from, no subject, and no date (or 1969 as date). The actual end result in my Junk folder though is the same message twice, with roughly the date at the beginning of the log message. There is no 12/31/1969 message in my Junk folder nor do I see a blank sender. This strikes me as a concurrency issue.

--- First the non-duplicated message:

<p>
[4/8/2021, 7:09:26 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Xeric <[e-mail]> - DRUMROLL PLEASE 🥁And the Instagram Photo Contest WINNER Is . . . at 4/8/2021, 7:07:52 AM
junk score
</p>
<p>
[4/8/2021, 7:09:26 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Xeric <[e-mail]> - DRUMROLL PLEASE 🥁And the Instagram Photo Contest WINNER Is . . . at 4/8/2021, 7:07:52 AM
moved message id = im5hlJgPS1uMf3ZSBoKscQ@geopod-ismtpd-1-0 to mailbox://nobody@Local%20Folders/Junk
</p>
<p>
[4/8/2021, 7:09:26 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Xeric <[e-mail]> - DRUMROLL PLEASE 🥁And the Instagram Photo Contest WINNER Is . . . at 4/8/2021, 7:07:52 AM
execution stopped
</p>

--- and now one that was duplicated:

<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Interesting Information <[e-mail]> - Why cursive handwriting is good for your brain - among other things. This handwriting by Prakriti Malla from Nepal won the title of “The Most Beau... at 4/8/2021, 11:05:04 AM
junk score
</p>
<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Interesting Information <[e-mail]> - Why cursive handwriting is good for your brain - among other things. This handwriting by Prakriti Malla from Nepal won the title of “The Most Beau... at 4/8/2021, 11:05:04 AM
moved message id = Aeq-VxDMfUa5PIHegKoOsIpkj_wOwtxtpXBmfpKwJrPjO2FtPqNpQ2HZeZ7fwrQ_c0OpAt5gQBXgD6SUBHkgYWyWeOOgs1aIzXfXKkHCirhvXBDS7fx_MS7VGAKpJCYBmbYU8w2Bn2vMmuT9TgABAgA=@t2.msgid.quoramail.com to mailbox://nobody@Local%20Folders/Junk
</p>
<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from Interesting Information <[e-mail]> - Why cursive handwriting is good for your brain - among other things. This handwriting by Prakriti Malla from Nepal won the title of “The Most Beau... at 4/8/2021, 11:05:04 AM
execution stopped
</p>
<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from - at 12/31/1969, 5:00:00 PM
junk score
</p>
<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from - at 12/31/1969, 5:00:00 PM
moved message id = to mailbox://nobody@Local%20Folders/Junk
</p>
<p>
[4/8/2021, 11:05:27 AM] Applied filter "Suspected Spam (unfiled messages)" to message from - at 12/31/1969, 5:00:00 PM
execution stopped
</p>

Comment 39

[Wayne Mery (:wsmwk)]

Todd, Jim, gitman (and anyone else)

Still see this after updating to version 128?

Comment 40

[toddgleason]

(In reply to Wayne Mery (:wsmwk) from comment #39)

Todd, Jim, gitman (and anyone else)

Still see this after updating to version 128?

Hard to say. The closest I have seen recently is that on 9/27 I had 2 messages listed in my junk folder from the same time, with the same-looking content except that one has the junk flag and the other doesn't. They're rendered differently possibly because of the junk flag. I can't find any message filters that either mark as junk or move to the junk folder without doing all 3 of marking as junk, moving to the junk folder, and stopping filter execution, so I don't see how the issue would be with my filters.

It feels like the bug is nearly but perhaps not completely squashed.