Junk Mail is not processed before Message Filters

RESOLVED WORKSFORME

Status

Thunderbird
General
RESOLVED WORKSFORME
11 years ago
8 years ago

People

(Reporter: Matthew Toledo, Assigned: Scott MacGregor)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: closeme 2008-03-01, URL)

(Reporter)

Description

11 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Build Identifier: version 1.5.0.10 (20070221)

First off, Thunderbird's spam filter gets rid of 95% of my spam mail... eventually.  It takes me three steps to accomplish this.  I am absolutely convinced this is happening because I have subfolders in folders, and Thunderbird is not processing the spam PRIOR to moving the messages to subfolders.

First, examine the image I have included.  It shows that I have chosen to put all my main into the INBOX. I then use message filters to organize the mail into appropriate sub-folders.  I check about 20 email accounts and have several more email aliases.

My mail server is a POP mail server.

1) Open Thunderbird  - The Thunderbird opens and starts downloading my email messages. At this point, it automatically throws only some of the spam directly into the trash. 80% of the spam has still gotten through. 

2) Click on a folder - When I click on a folder for the first time after new mail arrives.  Thunderbird suddenly starts eliminating the previously spam in only that folder.  This leads me to believe that spam filters are not applied BEFORE message filters. After clicking on the folder, 80% of the remaining spam starts to go away to the junk folder one message at a time.

3) Run a menu item - There is still junk mail in the folder. These messages are definitely spam, but have not been detected as spam.  I have to click on "Tools Run junk mail controls on folder."  Most of the remaining spam is marked as spam and moved to the trash. It leaves me with only one or two clever spam messages, which is acceptable.

If Thunderbird is capable of detecting spam at any of these steps listed above, it means that somewhere in the code, there is a reliable algorithm to detect spam.  The problem is, all the email is not being forced through this algorithm when it is initially received.  

SPAM FILTERS SHOULD BE APPLIED FIRST before any other user defined message filters.  This is not happening.


Recap:
Somehow, email messages bypass the spam filter when they are initially retrieved from the POP server. I think it bypasses the filter if you have "message rules" to place email into different folders. To fix this, Thunderbird needs to apply the junk filters BEFORE it processes the message filters.

Thunderbird may know about this fundamental bug, because they put in a total of 3 hooks to delete spam (listed above, and summarized below)  The hooks are:
1) Upon initial download
2) Upon viewing a folder that contains fresh email
3) By manually selecting "Tools Run junk mail controls on folder"

None of these hooks would be necessary if Thunderbird processed all email for spam BEFORE doing anything else.

Reproducible: Always

Steps to Reproduce:
1. Have several pop email accounts go into a single inbox
2. Create message filters to MOVE the email into sub-folders of the inbox
3. Retrieve your email
4. Some spam is filtered, more is marked as junk but still moved to the sub-folder.
5. Some messages bypass the spam filter altogether and are moved to the sub-folder.

Comment 1

11 years ago
I would not want this behavior changed.  The adaptive filter mechanism GUESSES whether a message is junk based on which messages you've manually marked as junk.  That's fine for cases you haven't covered in your filter list, but if I set up a filter to specifically mark certain messages as Junk or Not Junk, the adaptive filter has no business butting in and changing that judgment.  An explicit decision by a person (in setting up a filter rule) must always trump a guess.
(Reporter)

Comment 2

11 years ago
The problem isn't if or if you don't manually create a filter to mark a message as junk.  If you specifically want to mark a message as junk or whitelist it, then put that functionality in the spam trigger.  My complaint is that there are currently THREE triggers and none of them gets rid of all the email.  There should be ONE trigger called that filters all email for spam. I should not have to do three separate steps to accomplish the one goal of getting rid of spam in a sub-folder.  People create sub-folders for organizational reasons.  Keeping things as is might work for grandma, but not for advanced users.  Keeping things the same punishes folks who take advantage of Thunderbird's features.  It is a huge bug, plain and simple.  I never had to do this when I used to use outlook express and a third party spam killer.
To Matthew Toledo(bug opener):
Although sequence of filering is still (1)Message filter, (2)Junk filter, as designed, Tb 2.0 already has functionality to apply Junk filter to moved mails by message filter. See Bug 329569 for this change.
(When IMAP, problem still remains when mails moved to local mail folder)

It is working well, because fixing of Bug 329569 produced new bug reports of "Filtered mail was moved to Junk after upgrade to Tb 2.0, even though I moved the mail to a mail folder!", and I also experienced this phenomenon.
Workarounds of this new issue are;
 - add action of "Mark as Not Junk" in message filter
 - add mail address to address book

Comment 4

11 years ago
Yes, it would be great to have this automatic-junk processing after moving emails to loval folders for IMAP too, because I have this problem (see bug 389098).

One issue I asked myself is: Why there are so many differences in message-handling between POP3 and IMAP? when sorting email to a local folder or to a folder in a different "mail account" then the filters work the same, or ?!
I maybe understand a special handling for the "move inside one IMAP account" case, but I want to have eliminated spam there too when I endable this in the account settings ...

Comment 5

11 years ago
Matthew, have you tried v2 yet?  And, can this be summarized in a couple sentences.  (Bearing in mind developers are more likely approved something that works within the current paradigm rather than moving junk mail processing to before message filters.)


(In reply to comment #4)
> Yes, it would be great to have this automatic-junk processing after moving
> emails to local folders for IMAP too, because I have this problem (see bug
> 389098).

problem in that bug iirc is not a problem with the spec, but a bug which wada's comment may be a workaround for.


> Why there are so many differences in message-handling between POP3 and IMAP?

some references
http://www.mozilla.org/start/1.5/extra/using-junk-control.html 
http://wiki.mozilla.org/Thunderbird:Help_Documentation:Dealing_with_Junk_E-mail
... and a reading of wada's citation, Bug 329569, may help. 
OS: Windows XP → All
Hardware: PC → All

Comment 6

11 years ago
Matt?
Whiteboard: closeme 2008-03-01
(Reporter)

Comment 7

11 years ago
TB2.0 now removes junk email from messages that trigger a message filter.  This is the behavior change I was hoping for.  Thanks to this change: the count of unread email is accurate, TB doesn't make a beep or set a status bar icon when only spam is downloaded, I don't have to manually remove any spam (save for one or two real sneaky ones)

As far as I'm concerned, the POP version of this bug is fixed. If the IMAP guy has issues, he may want to split this into another ticket.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → WORKSFORME

Comment 8

11 years ago
There already exists a bug for IMAP -> #389098

And additionally the Bug #389096 handles one more difference between POP and IMAP that was "fixed" for POP earlier but not for IMAP.

Both bugs are still "UNCONFIRMED" :-(
You need to log in before you can comment on or make changes to this bug.