Open Bug 1534153 Opened 7 years ago Updated 1 year ago

Mail not filtered on startup. No error messages

Categories

(MailNews Core :: Filters, defect)

Product:
MailNews Core
Component:
Filters
Type:
defect

Tracking

(Not tracked)

People

(Reporter: wsmwk, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: [dupme?])

A few days ago, on startup, none of my messages were filtered for vseerror imap account - they stayed in the Inbox. Not all messages had the same destination folder. Later messages started being filtered.

The day before that I had one message not be filtered when I checked for new mail in the morning. I don't 100% recall if that was a startup - I don't think it was.

working through bug list https://mzl.la/2VPhq3S

Seen again a couple days ago. Just before it occurred I had done two things: was in a newsgroup folder and just left it, had sent an email

Just saw this in normal operation (not startup), and the circumstances are that TB was being very slow after Windows came out of sleep mode, and while trying to scroll through new messages it would be light years behind displaying them - like imap mesage retrieval was super confused and not keeping up. So perhaps related to bug 168648 and bug 714489, and perhaps also something imap related.

Severity: major → normal
Status: UNCONFIRMED → NEW
Depends on: 168648
Ever confirmed: true
See Also: → msgreader-jumble

Comment 2 is not confirmed, because the message that "didn't get filter" and stayed in the Inbox did in fact filter correctly - it had CJK in the subject because I have a filter with "stop filter" action to keep them in the Inbox.

Status: NEW → UNCONFIRMED
Ever confirmed: false

(In reply to Wayne Mery (:wsmwk) from comment #2)

Just saw this in normal operation (not startup), and the circumstances are that TB was being very slow after Windows came out of sleep mode, and while trying to scroll through new messages it would be light years behind displaying them - like imap mesage retrieval was super confused and not keeping up. So perhaps related to bug 168648 and bug 714489, and perhaps also something imap related.

I've noticed this before and see it now: When I quickly scroll through messages with down-arrow eventually I see a reconnect and re-login occurring in the status bar. Don't know why this happens. Could cause messages to be "years behind." But the last few times I get a message to enter the password. Retry with stored password doesn't login and prompts again. I have to click enter new password and then the stored password (shown as asterisks) works. This is 60.7.1. Problem probably with my isp's imap server (charter/openwave) but not sure.

I've seen this again in recent weeks with 68.x (rare of course)

I still see this, rarely.

Seen this morning on Mac. Clicked get messages on gmail account after waking Mac. No new messages in Inbox. Clicked again. Checked activity manager. Nothing significant there. Clicked on "Moz" folder where most messages get filtered to - beach ball (wait). Checked the account directory - nothing strange there. After a couple minutes messages started flowing - not due to anything I did.

It is still unclear what blocked message activity - but whatever it was in this case was temporary. (It has not always been temporary)

See Also: → 1719072
Severity: normal → S3
See Also: → 1852786

Still seeing this occassionally

See Also: → 1905064
Blocks: 1790862

Seen several times in recent weeks.

Status: UNCONFIRMED → NEW
Ever confirmed: true

(In reply to Wayne Mery (:wsmwk) from comment #9)

Seen several times in recent weeks.

Re: comment 7, you wrote:

Seen this morning on Mac. Clicked get messages on gmail account after waking Mac. No new messages in Inbox. Clicked again. Checked activity manager. Nothing significant there. Clicked on "Moz" folder where most messages get filtered to - beach ball (wait). Checked the account directory - nothing strange there. After a couple minutes messages started flowing - not due to anything I did.

It is still unclear what blocked message activity - but whatever it was in this case was temporary. (It has not always been temporary)

I assume you are seeing something like you described in comment 7 for mac.

So computer was sleeping and you woke it up. Not sure if matters, but does your mac do a full sleep or a partial sleep like default on windows? FWIW, my linux laptop does a hibernate (powers off completely) and on wake-up and click get messages, I always see my bug mails get moved to their folder by the filter. Never seen any not retrieved or just stay in inbox on click of "get messages" (I don't use biff) nor bring in new mail at all. That's the only filter I use consistently for everyday use.

After a couple minutes messages started flowing - not due to anything I did.

This was probably the first biff or maybe IDLE response to occur after wakeup. I assume by this the new mail was retrieved and filtered (moved) to destination folder(s)?

i'm seeing this problem with the filters basically every day ever since i migrated (about 2-3 weeks ago) from Thunderbird ESR 128 to the standard / stable branch (v137.0.1 currently). The message filters are no longer applied at startup and i have to apply them manually.

The account message store structure is set to maildir, to help with processing/filtering the many incoming messages, but even so, every morning when i start Thunderbird i have to run the filters manually.

My use case is a bit on the heavy side though - i have Thunderbird set to monitor (and clean weekly/monthly) an internal mail server that is used by the ANPR / LPR system as a buffer to notify the access control system that keeps track of access tickets... and the system generates literally hundreds/thousands of email messages per day.
Each vehicle is notified only once every 24h (tickets are valid for 24h, starting from the notification time) but even so, we get a lot of them - in a typical 30-day month the system usually generates 20+ thousand notification messages.

Had to look up "ANPR/LPR" and see it is maybe a car/vehicle license plate reader? That's sounds like a unique way to use TB!
Can you be more specific as to what the filters do and how they are set up? E.g., message arrives in Inbox and filter moves or copies to another folders. How many filter are in effect? Etc.
How does the filter "clean" the mail server? Maybe moves message to a Trash mailbox? Is TB used at all by a person to "read" the messages/tickets?
Also, is your system using POP3 or IMAP? What is type of mail server in use, e.g., dovecot, cyrus, courier etc.
When you apply the filter(s) manually every day, about how many messages get moved/copied by each filter?

yes, a vehicle license plate reader - the mail server mailbox is set as a message buffer between the actual LPR server and the accounting server that parses the notifications about access tickets activation.

Thunderbird is not used to process the actual ticket accounting info, only to manually monitor the messages flowing between those two systems and perform random checks that license plate recognized by the LPR system is correct and corresponds to the vehicle photo that is included with the message. (The messages between those two systems are easily readable by a human)

The mail server cleanup is also done from Thunderbird because those two systems are not designed to delete old mail messages - i have a filter set in Thunderbird that after a message is older than 7 days then a message from the LPR server is supposed to be moved out of the inbox to another folder on the server.

After about 25+ days have passed, i move the messages out of that folder on the mail server into the Local folders (this practically frees up space and deletes the message from the server) - i move them into a local folder that corresponds to its month (e.g. 2025-02, 2025-03, 2025-04, etc).

Also, at the start of a new month, before starting Thunderbird i also delete the N-2 folder from disk and the associated .msf index file (where N is the current month that has just started) - this practically purges from the local storage all messages from the LPR server that are more than 31 days old.

The mail server is an inherited system running hmail (https://github.com/hmailserver/hmailserver), we're using it with SMTP+IMAP but it can also do POP3, and is (by design) not reachable from outside its network.

This morning before applying the filters manually i enabled the filter log and then aplied the filters - 180 messages have been moved - all arrived since last evening (i.e. in the last 12 hours).

note: i think i have described the 7-day filter wrong... the messages in inbox are moved immediately to the 7+days folder, but only when they are marked as read by the accounting server - i have set Thunderbird to not modify the read/unread message state flag so that only the accounting app sets the 'message was read' flag, from there i move them to another folder for the remainder of the month.

Adi, thanks for the detailed info.
If I understand correctly, the only issue with TB is that the filter is not detecting when the message is marked read (imap \SEEN flag set, I think). The filter is only looking to see if the new and existing messages have the \SEEN flag and is NOT looking at the age of the message (i.e., more the 7 days old)? So is your filter is set like this:

Apply filter when:
(x) Manually run
(x) Getting new mail [Filter before junk classification]
( ) Archiving
( ) After Sending
( ) Periodically, every 10 minutes

(x) Match all the following
Status      is      Read

Perform these actions:
Move message to             7-day-folder

The problem is nothing is getting moved to 7-day-folder unless you manually run the filter. It is not occurring when new mail is checked or when TB is started.
This brings up the question: how is your imap server settings set? I.e.,

( ) Check for new messages at startup
( ) Check for new messages every X minutes  <---- what is X?
( ) Allow immediate server notifications when new messages arrive <--- probably N/A for your imap server type (no imap IDLE support)

Re-read your first comment and you say it's only a startup problem (subject of this bug). So important to have the 1st item above selected.
But if it is already selected, does the filter then automatically work after X minutes (assuming you have the 2nd item selected)?
Also, when you start TB and BEFORE you manually run the filter (with first item above previously selected), do you see new messages (read and/or unread) in Inbox without clicking "Get messages" or pressing F5?

Anyhow, I'll try to duplicate this on a test account using the info you have provided and my assumptions on how your are set-up.

Adi, are you sure this worked on older TB versions? What I'm seeing is that the filter (at startup or when checking for new mail every X minutes) only looks at the new mail. The filter, when run automatically, ignores existing messages in Inbox. I set up a filter to copy only READ messages to another folder. My Inbox has lots of existing read messages. With TB shutdown, I then add a new READ messages to the server's Inbox. When I start TB, only the new READ messages gets copied by the filter. The existing READ messages are NOT copied. Only if the filter is manually run on Inbox are the existing READ messages in Inbox copied.

I've observed this before but with other filter criteria such as when filtering on message "Subject", only newly arrived messages are affected and any existing messages matching that "Subject" are not affected unless the filter is manually run. (Until now, I've never tried to filter on READ status but I'm seeing the same behavior.)

So, if I understand your set-up, for it to work automatically, the "accounting system" would need to delete any existing unread messages in inbox and replace them with copies that are marked read (imap \SEEN) so they appear to TB as new messages on startup or when checking for new mail. If the accounting system just adds the \SEEN flag, TB will not see the message as new and the TB filter system will not move or copy the message (unless manually run, as you are doing).

you are right, i am not sure that it ever worked on startup - i am so used by now with this problem that i tend to ignore it and just run the filters manually.

Also, the hmail server supports IMAP IDLE quite well, Thunderbird is regularly pinging in real-time for each notification message that arrives in inbox.
https://github.com/hmailserver/hmailserver/blob/master/hmailserver/source/Server/IMAP/IMAPCommandIdle.cpp

Also, the accounting system just sets the SEEN flag, it does not replace / change the message in other ways - the internal Message-ID: header is not modified.

This morning i have decided to re-organize the filters.. when starting Thunderbird i started it in offline mode, (so that it does not connect to the server and fetch new mail) and reconfigured the filters to be just one filter.

posted a screenshot of its configuration here: https://i.imgur.com/xHQ0kO5.png
short filter description: i gave up on the intermediate folder and will now just move all LPR mails from inbox that older than 7 days directly to the local folders.

After changing the filter i restarted Thunderbird (this time in online mode) and even though it fetched the new mails (already marked as read by the accounting server), the filter log remained empty until i clicked "Run Filters on folder".. and only then it started to move inbox messages older than 7 days ...
As said though, those messages were already downloaded by Thunderbird since a few days ago - the filter is designed on purpose to avoid touching recent messages... but the filter was never run on startup - and the messages not moved.

Note: sometimes i also enable the "run filter every x minutes" checkbox - probably this is a good work-around but i found that it can cause heavy load both on the client and on the server, especially when i come back from vacation and nobody has cleaned the server mailbox while i was away.
Having the filter handle 10+ thousand messages at once is not fun.. as it can bog down either the local machine or the server.
(when come back from vacation, i first start Thunderbird in offline mode and set the filter to run only manually, so that i can catch up manually with the cleanup)

Not sure how the "run filter every 10 minutes" checkbox works. Does that do the same as "run manually" and look at ALL messages in folder? If it does "look at them all" there is a pref parameter that will allow you to change the 10 minute to whatever you want: mail.server.default.periodicFilterRateMinutes defaults to 10 minutes. So you could set it to maybe 60 to lessen the server load. However, not sure if this runs immediately at startup like you want.

yes, the checkbox to 'run periodically, every X minutes' seems to do the same thing as running the filter manually, and even with it enabled, the filter does not run at startup, only after the first period passes.

it is a useful work-around though... thank you for mentioning that periodicFilterRateMinutes preference... now i have set it to run the filter every 2 minutes and it seems to be a good replacement for running at start and/or when it receives mail.

The suggested 60 minutes would probably be OK too if i were to leave Thunderbird run unattended, but this duration is not really what could cause the server load. That problem comes from running the filter at all after it was not run for a few weeks - such as when i come back from vacation.

During such a time of no cleanup activity the volume of messages pending in the inbox on our server can easily reach 10+ thousand and Thunderbird has no possibility to specify that the filter should only process a maximum of X messages for each run of the filter. (with X an adjustable amount of 100, 500, 1000 or other).

This is why when i come back from vacation i disable the filter first and then manually move messages - about 2500-3000 messages for each group of messages moved seems to be the threshold for queued IMAP EXPUNGE batches... if i go above that our server starts to get dangerously close to the maximum RAM memory limit for 32bit processes - i have even observed it once crashing when it reached the memory working set limit. ( i think i was trying to move about 5000 messages to Local Folders when it crashed)

We have 32GB of RAM in that server (which is also used for other things, and usually only about 8 GB are in use), but even so, hMailServer.exe is a 32-bit app = 2 GB maximum memory working set... so i have to limit the number of messages moved for each run of the filter.

Ok, thanks for info.
So actually your issue turns out to be more of an enhancement request rather that literally "mail not filtered on startup" which is subject of this bug. So maybe you should file a new bug (or bugs) requesting these changes:

  1. Document that when filters run at startup and on new mail checks, only newly arrived messages are checked and old messages satisfying filter criteria are ignored.
  2. Document that ALL messages are checked for filter criteria when filter manually run or run via 10m (default) timer.
  3. Add option to allow ALL messages to be checked for filter criteria at startup and new mail checks (just like manual and timed filter runs)
  4. Add option to adjust the auto check timer without having to set the pref. (May need to be timer specific rather than server specific.)
  5. Add option to limit the number of messages filtered per filter run
See Also: → 1770247, 1760816, 931303
Summary: mail not filtered on startup → Mail not filtered on startup. No error messages
You need to log in before you can comment on or make changes to this bug.