Open Bug 1368056 Opened 8 years ago Updated 3 years ago

After moving mails to another folder, a filter that is set to only work on received emails is fired

Categories

(Thunderbird :: Filters, defect)

Product:
Thunderbird
Component:
Filters
Version:
52 Branch
Type:
defect

Tracking

(Not tracked)

People

(Reporter: accounts, Unassigned)

References

Details

(Keywords: regression)

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0 Build ID: 20170518000419 Steps to reproduce: I have a filter which moves emails where I am only mentioned as a cc recipient to a folder "CC", which is a subfolder of the main inbox (of an IMAP account). The rule is set to only work on received mails. After having read such mails, I move them back to the inbox manually. Actual results: Since version 52.1.1. moving those mails to the inbox folder triggers the filter again, and the emails are moved back into the CC folder. Protocol says: [26.05.2017 16:12:46] Filter "CC" auf die Nachricht … am 23.05.2017 14:32:45 angewendet Nachricht (ID= …) verschoben nach imap://…/INBOX/CC What's interesting is that while the protocols timestamp (in square brackets) is the current timestamp, while the timestamp mentioned in the message seems to be the date and time when the email was received and moved by the filter for the first time. Expected results: The moved emails should have stayed in the inbox folder, as the filter is set to only work on newly received emails.
This looks like a regression from bug 308148 and bug 1320832. Can you set the preference mail.imap.filter_on_new to false (Tools > Options, Advanced, General, Config Editor) and see whether that solves the issue. Looks like the message moved back is considered "new" and treated again.
Flags: needinfo?(rkent)
Indeed setting filter_on_new to false prevents emails moved back to the inbox folder from being handled by the filter again.
Aceman, you reviewed https://hg.mozilla.org/comm-central/rev/73edb3ab0b80. So apparently a message moved back to the inbox has a UID higher than the current highest, so it's considered "new" and gets filtered. Same problem as in bug 1334943.
Blocks: 308148
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(acelists)
Keywords: regression
See Also: → 1334943
So we need to raise the "high water" UUID when messages arrive into the folder by moving/copying?
Flags: needinfo?(acelists)
Hmm, I didn't review that bug since I'm not an expert there. Which possibilities are there for a message to arrive in the folder? - Being received - Being moved/copied. When do these cases increase the high UID? They must do that at some stage, right? So you're saying you want to raise the UID when moving/copying and before the filter gets run so those ones are not detected as new.
Yes, that would be my idea. But I am also not an expert in this, I never used IMAP. Also there must be assurance that filters was already run on new messages in the folder we are copying into and playing with the highwater there.
(In reply to :aceman from comment #4) > So we need to raise the "high water" UUID when messages arrive into the > folder by moving/copying? No, that won't work reliable. You will not get any safe information in the destination box from where the mail came from. With IDLE mode there is a race condition against an incoming new Mail. Without IDLE mode, hundreds of new mails may have arrived in the meantime. The only information available to us are the copied flags and the 'Internal Date Message Attribute'. | RFC 3501 2.3.3. Internal Date Message Attribute [...] | [SMTP]. In the case of messages delivered by the IMAP4rev1 COPY | command, this SHOULD be the internal date and time of the source | message. If this timestamp is the same as in the source box, there is a high chance, that it is the same Mail. But it's just a 'SHOULD' requirement. The only reliable method would be to use the UIDPLUS extension. See: "RFC 4315 - IMAP - UIDPLUS Extension". There the server delivers the UID in the target box in the response to a COPYUID command. But I can not say how common this is among the servers.
(In reply to Alfred Peters from comment #7) > The only reliable method would be to use the UIDPLUS extension. See: "RFC > 4315 - IMAP - UIDPLUS Extension". In a bug search for UIDPLUS, I get lots of results. It seems that we are already supporting this. I did not know that yet.
I suggest for now we revert bug 1320832 until we devise a solution.
Flags: needinfo?(rkent)
See Also: → 1381865
See Also: 1381865 →
(In reply to Kent James (:rkent) from comment #9) > I suggest for now we revert bug 1320832 until we devise a solution. Almost a year into the release, I think we can probably say that ship has sailed.
Blocks: 1320832

Wayne, I don't understand what your are saying in your comment, so just in case, I want to confirm that the original problem still exists.

If a filter moves a message to a subfolder and when I decide it better sits in inbox and move it back there, the filter will be triggered again and move it back to the subfolder (60.8.0). It would be helpful if that could be avoided.

(In reply to klaus from comment #16)

Wayne, I don't understand what your are saying in your comment, so just in case, I want to confirm that the original problem still exists.

If you are asking about "Almost a year into the release, I think we can probably say that ship has sailed.", it means too late to reverse course/undo the change.

If a filter moves a message to a subfolder and when I decide it better sits in inbox and move it back there, the filter will be triggered again and move it back to the subfolder (60.8.0). It would be helpful if that could be avoided.

Avoided if you set mail.imap.filter_on_new=false, no? If not, what is your proposal?

yes, I read that somewhere. It seems it goes into conflict for people who read on mobile before TB can do the filtering.

How about using Gloda or messageheaderID to determine whether the email has already been processed by TB?

Considering the workflow:
-TB receives in inbox
-TB filter moves email to subfolder
-sometime later I see email and move it back to inbox
.. at that point, the email would hopefully be in Gloda. So if the inbox filter on receiving checks whether messageheaderID has been indexed and leaves those messages out, we should be fine.
I am aware there are some deficiencies about indexing on moving messages, but still, we might catch a few of those which are moved back. It probably works if there is sufficient time between receiving and moving back to inbox.

this is still not working in TB 78. Filters are reapplied when the moved message is redownloaded from imap inbox.

  1. I had a look at the code. What about this: first, we set highwater to 0, then we get the true value from msgdatabase.
    If that would fail or would be the value for the wrong folder, highwater might stay 0 and the if condition would always be true.

  2. it is in imap. Should bug component be filters or imap ?

Unfortunately, I cannot debug this without a full TB build environment for cpp, which I do not have.

Un-Setting the pref does not help if one uses mobile to read first (then the message is read and will not be moved upon first occurrance in inbox - another bug).

Also, this is pretty obscure to end users. Just installing a fresh TB on another PC for this account with the pref default (!) set will forcibly move all these emails out of inbox (who would remember to unset the pref before setting the email ccounts?). Just happened to me, 1 GB of emails mixed back into another folder. (at least, TB suddenly offered to regain 1 GB of deleted email storage, which probably is due to the move?).

Klaus

the idea of imap is to have access from multiple PCs to the same account.

We have a pref that is set to a value where we know it does not work.

The side effect of that pref being set may be a total reorganisation of inbox - thousands of emails might be moved a second time, if a new TB installation points to that account.

Should severity be higher than normal? Or the default be false? If it does not work on true, why set default to true?

I have the latest Thunderbird (78.6.1) and the same problem exists. If I set a filter to move an email automatically from Inbox to a folder A under a specific condition, then I manually move the email later to another folder B, the filter runs again and it's put back into folder A. I've added a filter on my IMAP server instead (which is probably best anyway).

I argue this can be solved by changing user workflow, hence the severity of this bug is low (which might also explain why nobody cared enough to fix this yet). Thunderbird by default does have no active filters, hence if users import/create filters that move e-mails from inbox to somewhere, actions by users are what trigger filter behavior.

  • Workaround:

    1. Manually drag e-mails not back to Inbox, but rather to a folder that is not the inbox.

    Expected outcome: Filters will only act on Inbox. Since these e-mails are not in the inbox, they will not be moved.

  • Workaround for installing new Thunderbird and accessing from remote:

    1. Create a new folder
    2. manually (or automatically via filters), move all e-mails from Inbox into this folder
    3. Install new Thunderbird; Import and run filters.

    Expected outcome: Filters will only act on Inbox. Since these e-mails are not in the inbox, they will not be moved.

This bud is still open: with TB (102.2.1 (32-bit)) and the flag set as recommended above, I still cannot overrule filters by manually moving mails from one folder to the other.

Thanks for your great work!

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.