Open Bug 1778064 Opened 2 years ago Updated 6 days ago

Messages Filters with "Move to folder" no longer run automatically

Categories

(Thunderbird :: Filters, defect)

Product:
Thunderbird
Component:
Filters
Version:
Thunderbird 102
Type:
defect

Tracking

(Not tracked)

People

(Reporter: olivier.jaquemet, Unassigned, NeedInfo)

References

(Blocks 1 open bug,
URL
)

Details

(Keywords: regression, Whiteboard: [filterfail])

Steps to reproduce:

I have several message filters, some moving to folder, some tagging, etc

Since the upgrade to Thunderbid 102, some filters with "Move to folder" action are no longer run automatically on the incoming messages.

Actual results:

  • Messages are kept in inbox
  • filter log does not contains any failure or indication

Expected results:

Messages should have been moved a specific folder

This problem was initially discussed in bug #1777731
But as it appears to be a different symptom, this bug report was created as requested by Ping Chen.

More information :

  • Filters failure : Selecting the "Inbox", and using the "Tools" menu to "Run Filters on Folder", has NO effect
  • Filters success : Opening "Message Filters" from the "Tools" menu, selecting all filters, and using "Run now" on the "Inbox" folder properly execute all the filters (this is my workaround for now)

Damned... It appears some of my actions on the filters have fixed the issue, but I'm not sure at all on which action did the trick.
Here is a complete list of actions I did on my filters :

  • reedit ONE of them, to re-select the destination folder. manually run on folder; no change
  • reedit ONE of them, to remove the "Delete From POP server"; manually run on folder; MAYBE this one did fix the pb
  • move ONE of them to top, manually run on folder; no change
  • disabled two of them, then re-enabled. manually run on folder; MAYBE this one did fix the pb

Also, ... it migtht be the same bug after all. Because I have looked at a backup of the msgFilterRules.dat (2 month old)
Here is an extract for one filer

  • Old (I was using Thunderbid 91) :
    actionValue="mailbox://jalios08@pop.jalios.com/1.%20Dev/SVN"
  • New (Thunderbid 102 since yesterday, but I dont know if this actionValue was modified before ) :
    actionValue="mailbox://olivier.jaquemet%40jalios.com@mail.exchangeincloud.com/1.%20Dev/SVN"

I HAVE NOT changed the username/hostname during that time

Blocks: tb102found

I can confirm the filter problem. This is with TB103.0b2 on Linux.
The behavior for me is slightly different than described above though.
What's different:

  1. When there are multiple messages in Inbox to be moved to the same folder by the filter, one of those messages is moved to the folder automatically, but all other messages are not moved, and remain in Inbox. This is reproducible. There are no related errors in the Error Console as far as I can see.
  2. Then, selecting Inbox, and using Tools - Run Filters on Folder works here. I.e. the filter is moving those messages to the correct folder. This action is also present in the filter log. So it does not seem to be a problem of the filter definition.

"Manually Run" is checked on all my filters.

Status: UNCONFIRMED → NEW
Ever confirmed: true

(In reply to Olivier Jaquemet from comment #2)

  • Old (I was using Thunderbid 91) :
    actionValue="mailbox://jalios08@pop.jalios.com/1.%20Dev/SVN"
  • New (Thunderbid 102 since yesterday, but I dont know if this actionValue was modified before ) :
    actionValue="mailbox://olivier.jaquemet%40jalios.com@mail.exchangeincloud.com/1.%20Dev/SVN"

I HAVE NOT changed the username/hostname during that time

This is expected, before bug 1483485 and bug 1769732, .hostname pref contains the hostname when setting up the account the first time. Then if you changed hostname, it's stored in .realhostname.
The actionValue points to the current hostname means migration code in bug 1769732 works.

I will test filtering multiple messages later.

Keywords: regression

The TB 102.2.0 update now also broke manually running filters via Tools - Run Filters on Folder. Very annoying.
Running filters manually still works with 104.0 b4 for me. Running filters automatically is broken for both, v102 and v104 beta.

Cannot reproduce comment 3 in a soft-fork of Thunderbird called Betterbird; Version 102.3.0-bb17 (64-bit) on Linux Mint (Cinnamon) connecting via IMAP.

using following filter (copied from msgFilterRules.dat):

version="9"
logging="no"
name="test4"
enabled="yes"
type="17"
action="Move to folder"
actionValue="imap://nobody%40gmx.de@imap.gmx.net/INBOX/test"
condition="AND (subject,contains,test00)"

Multiple messages that arrived at the inbox at the same time were moved successfully and automatically (when I clicked on Inbox) from inbox to imap sub-folders.

To test this, I created a new TB profile, but connected to an old E-mail account.

I will check if same holds for Thunderbird Daily

Thunderbird Daily 107.0a1 (2022-09-28) (64-Bit) works correct and behaves similar to what I experienced in comment 6

Christian Riechers, can you still reproduce? If not, then I would suggest to close this issue.

If you can still reproduce, then I would like to ask you to provide additional detailed data and instructions, as I have failed to do so.

The problem as per comment #3 still happens with TB 106.0b1 as well as with BB 102.3.0-bb17 (64-bit) on Linux.
You can ignore comment #5, that was a different problem.

There is no additional information I can provide other than what's already been mentioned in comment #3.
These are existing profiles, which went through several migrations. So it may well be some sort of migration issue. In any case, there is nothing I'm actively doing, the problem just started at some point. I'll be more than happy to provide more information if someone can tell me what that information should be.

Information that could help:

  1. Do you use pop3 or Imap?
  2. a copy of relevant filters in your msgfilterRules.dat (you can remove sensitive information via "find and replace" of a texteditor)
  3. Some example messages

You could try to recreate some of your filters that in theory should, but do not work, on a new Thunderbird profile and check if it works with the new profile.

(In reply to Thilo.Ertel from comment #10)

  1. Do you use pop3 or Imap?

IMAP

  1. a copy of relevant filters in your msgfilterRules.dat (you can remove sensitive information via "find and replace" of a texteditor)

version="9"
logging="yes"
name="tb-planning"
enabled="yes"
type="17"
action="Move to folder"
actionValue="imap://xyz@imap.xyz.com/Read/tb-planning"
condition="OR ("list-id",contains,planning.discuss.thunderbird.net)"

  1. Some example messages

messages to the tb-planning mailing list

I've created a new test filter for the IMAP account, which is supposed to move messages received from a different sender email to folder 'Read'.
I sent myself three email messages from the above sender email account. The problem is reproducible with that setup.
The first two messages sent have been received, but remain in Inbox. The third message is automatically moved by the filter to folder 'Read' as it should (confirmed by filter log).
I can move the first two messages received and still in Inbox to 'Read' by running the filter manually (also confirmed by filter log).

Great! I have two questions:

  1. "list-id" is not a standard condition. How did you add it to Thunderbird? Via addon or simply via "customize..." ? (Simply being curious here)
  2. Could you click on one of the e-mails that were NOT moved and press CTRL+U and then cross-check if the content of the list-id of these particular e-mails indeed contains "planning.discuss.thunderbird.net"? (CTRL+F and search for "list-id")

(In reply to Thilo.Ertel from comment #13)

  1. "list-id" is not a standard condition. How did you add it to Thunderbird? Via addon or simply via "customize..." ? (Simply being curious here)
    via "Customize"
  2. Could you click on one of the e-mails that were NOT moved and press CTRL+U and then cross-check if the content of the list-id of these particular e-mails indeed contains "planning.discuss.thunderbird.net"? (CTRL+F and search for "list-id")
    I'm pretty certain that all tb-planning list emails do have the 'List-Id' line in the header. The messages do get moved by the filter eventually, either automatically or manually. That wouldn't have happened if 'List-Id' was missing.

I've now slightly modified the filter to see if that makes a difference:
condition="OR ("list-id",contains,<planning.discuss.thunderbird.net>)"

Have you crosschecked? - I don't know your filter setup, but if you have multiple filters and conditions running at the same time, you might as well have triggered this bug

(In reply to Christian Riechers from comment #14)

I've now slightly modified the filter to see if that makes a difference:
condition="OR ("list-id",contains,<planning.discuss.thunderbird.net>)"

Modifying the filter did not make any difference.

(In reply to Thilo.Ertel from comment #15)

Have you crosschecked?
Crosschecked what?
I don't know your filter setup, but if you have multiple filters and conditions running at the same time,
I have multiple filters, but only one condition (e.g. matching List-Id), and one filter action (move to sub-folder).
you might as well have triggered this bug
I'm not sure whether it's related. I do not copy messages, only move. I do not copy or move to Local Folders, only moving to IMAP sub-folder.

Good. Then Bug 892424 seems likely not to be the cause.

With crosschecked I mean this:

"Could you click on one of the e-mails that were NOT moved and press CTRL+U and then cross-check if the content of the list-id of these particular e-mails indeed contains "planning.discuss.thunderbird.net"? (CTRL+F and search for "list-id")" Since you are at it, you can also cross-check, if the foldername is the same as the one in the filter action.

Although, "I can move the first two messages received and still in Inbox to 'Read' by running the filter manually" seems to point to a real bug...

(In reply to Thilo.Ertel from comment #18)

With crosschecked I mean this:
"Could you click on one of the e-mails that were NOT moved and press CTRL+U and then cross-check if the content of the list-id of these
particular e-mails indeed contains "planning.discuss.thunderbird.net"?

The List-Id of these particular e-mails indeed contains "planning.discuss.thunderbird.net".

Since you are at it, you can also cross-check, if the foldername is the same as the one in the filter action.

I'm not sure what that means. When running the filter (either automatically or manually) the messages are being moved to the correct folder, as set in the filter action.

Although, "I can move the first two messages received and still in Inbox to 'Read' by running the filter manually" seems to point to a real bug...

I don't disagree with that one.

Note, Bug 1769732 - filters move/copy rules involving accounts that had realhostname do not work

(In reply to Wayne Mery (:wsmwk) from comment #20)

Note, Bug 1769732 - filters move/copy rules involving accounts that had realhostname do not work

However, after rereading prior bug comments, 1769732 may not be a good fit here

See also Bug 1801464

See Also: → 1801464

I was referred here from the forum at support.mozilla.org by user christ1.

Here is the thread I posted there: https://support.mozilla.org/en-US/questions/1404869

I see this bug report was posted 7 months ago and has been inactive for 3 months, yet my issue began sometime after 5pm on February 8, 2023 as indicated by the filter log. At that time messages from my windstream email account that should be filtered into various folders are remaining in the in box. However, messages from my gmail email account are still being filtered as expected. Both accounts are imap.

I am on version 102.7.2. I see where version 102.7.2 was first offered to channel users on February 7, 2023. It seems likely that my instance updated sometime after 5:36:11 pm on February 8, 2023, which was the last log entry in the filters log until I manually ran the filters on February 10th. This strongly suggests that the update to version 102.7.2 was responsible for triggering this behavior.

I manually initiated the update to version 102.8.0 on Friday February 17, 2023. Within 24 hours, emails were again being filtered into the proper folders when getting new mail.

(In reply to Thilo.Ertel from comment #22)

See also Bug 1801464

If bug 1801464 is specific to message body, then that's not this bug?

FWIW, in the past year we have, I think, a growing list of reports of filters failing https://mzl.la/3JCz4lV

Whiteboard: [filterfail]

Summary:

This topic deals with multiple issues from three different bug reporters.

@Christian Riechers @Olivier Jaquemet @Michael Purcell:

  1. "Folder remain in inbox issue" → Please try one or multiple steps outlined below
  2. "Folder name, folder cache and msf corruption issue ala Bug 1777731" → Probably and hopefully resolved. Olivier has gone quiet too. But this could mean that one or all of you have some leftover msf mailbox corruption → try to Test with a new and clean installation of Thunderbird without using your old profile. Check if results are similar.
  3. "When there are multiple messages in Inbox to be moved to the same folder by the filter, one of those messages is moved to the folder automatically, but all other messages are not moved, and remain in Inbox" → This is probably Bug 1781792 → and depends on Bug 662056 → Workaround is to use "move later" filter action from filtaquilla addon.

In general, while the root cause may be different, the symptoms ("messages remain in inbox") are a little similar to bug 1801464. We've had a breakthrough in that bug and now know, there are conflict/timing issues with "filter before junk classification" when Maildir is enabled and "move" filter action has a "message body" condition. Maybe one of the workarounds will work here too.

Things to try

a) use "filter after junk classification"
b) use "move later" filter action from filtaquilla addon
c) Switch your account from Maildir to Mbox format.
d) Avoid placing filters that will likely trigger on the same message right next to each other. Place some other filters inbetween or in front of this filter to give Thunderbird time to finish earlier operations. This will hopefully avoid any race conditions, if there are any.
e) Test with a new and clean installation of Thunderbird without using your old profile. Check if results are similar.

Thanks for continuing to work on this. The problem still exists for me in both, release, and beta versions of TB.
Use "filter after junk classification" did not have any effect on beta.
I do not want to install the filtaquilla addon.
I'm not using maildir format on any account.
There is only a single filter action per message - move the message to an IMAP sub-folder. I do not want to move messages to a local folder.

See Also: → 1865598

( v115.113.0 ESR / Windows 10 / IMAP )

;TL/DR;
Can someone point to code lines where a filter match is true and the code then does an "if" to process actions? It seems like that code or some lines following it are involved in this.

Given that this issue is a few years old, I'll try to help with a few more details about the symptoms:

  1. I can run all filters on a folder with two mail items. One of them will properly move to another folder, the other will not.
    Given this, it seems the issue is not related to the content of the filters. Execution of specific filters or actions might trigger the condition, but the condition does not exist until one or more given mail items have been processed and the loop moves on to another.

  2. The issue is not with processing the filters to determine a match, but processing the actions.
    When this condition is present: All filters run on a folder. A filter condition can match. Its actions do not run at all. The next filters are run - because obviously the Stop Filter Execution action is not running on any of them.
    Given this, using the Move Later option in filtaQuilla doesn't seem to be a valid diagnostic, as even that action will not be executed. (I'm exchanging notes with Axel Grude (of quickFilter/FiltaQuilla) about this.)

  3. Compacting does not change the state of filtering on a folder when this happens.

  4. Once this happens, the only remedy I see now is restarting the Thunderbird application. The success rate on this is 100%. This is another confirmation that the issue is not with specific filter processing, specific mail items, or with folder corruption, but with some event that occurs that seems to affect the application state.

  5. This happens frequently. Many times per day. I'm doing a lot of maintenance right now on accounts, folders, and filters. I get the sense that this is provoking the condition. So this is a "somewhat reproducible" case.

I will be doing more testing on this to see:

  • If messages are copied into a folder that has an issue with one item, will other items process?
  • If an item is processed and then put back into the original folder, is it processed the same or at all?
  • Can anything other than a restart influence the state?
  • Might this be a timing issue? Maybe a lock contention?
  • Is this unique to IMAP or is POP3 (and local folders) affected?
  • Is there any other pattern?

Note that the "effect" that we are reporting might have multiple causes that occur independently, or as a result of a sequence of operations.

It would help to know whatever debug flags/prefs might be useful to get verbose diagnostic messages in the Error Console. And there might be no messages if some bit is set erroneously and no error condition exists.

But I really think the/a issue is in the code lines where it decides what to do on a successful filter match.

See Also: → 1909401

(In reply to Tony Gravagno from comment #29)

( v115.113.0 ESR / Windows 10 / IMAP )

;TL/DR;
Can someone point to code lines where a filter match is true and the code then does an "if" to process actions? It seems like that code or some lines following it are involved in this.

https://searchfox.org/comm-central/source/mailnews/search/src/nsMsgFilterService.cpp
https://searchfox.org/comm-central/source/mailnews/search/src/nsMsgFilter.cpp
https://searchfox.org/comm-central/source/mailnews/search/src/nsMsgBodyHandler.cpp

Flags: needinfo?(xh8r2nbu9q)
You need to log in before you can comment on or make changes to this bug.