If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Automatic junk (spam) detection on messages moved by a filter.

RESOLVED FIXED

Status

MailNews Core
Filters
RESOLVED FIXED
12 years ago
9 years ago

People

(Reporter: jst, Assigned: Bienvenu)

Tracking

({fixed1.8.1})

Trunk
fixed1.8.1

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

12 years ago
Right now if you get a spam message and the message gets filtered into a folder we won't run spam detection on the messages until the user selects the target folder. This means we show in the UI that there's new messages in the target folder (even if it's spam and we'd know it's spam if we'd check) and we fire off notifications about the new message(s).

Seems like the user experience would be much better if we'd simply run a spam detection pass on the target folder after a filter has moved messages into the target folder, and then only once that's done, we'd notify the user about the new messages if there really are new non-spam messages in the folder.
(Assignee)

Comment 1

12 years ago
I think for imap we're doing this right now on the trunk and 1.8 branch, if showPreviewText is turned on, which it is, by default...for pop3, we're not doing this yet, and we should.
Status: NEW → ASSIGNED

Comment 2

12 years ago
Is this by any chance the same as Bug 200788?

Comment 3

12 years ago
Perhaps this is a duplicate, but has this bug been unfixed really for 3 years??? This bug has annoyed me everyday since I switched to Thunderbird 1.0.7.  Every single time Thunderbird reports new mail, I check it, only to have it junk-filter every message after I selected the individual-folder for each of my accounts.  Please, I want to reclaim my inbox! Filter junk, then report if I have mail, in that order. :)  This bug makes new-mail notification pointless.  Even if this is a duplicate, it needs to be duplicated, reopened, and reassigned. 2003?!?!?!?? I waited for the 1.5 release and didn't switch to outlook only because it seemed impossible that this wouldn't be fixed.  The previous bug report utterly failed it purpose and us users.  Please assign this bug to someone else. :)  This made the 1.5 release a real let-down. 
(Assignee)

Comment 4

12 years ago
My mistake. We're not doing this for IMAP quite yet either. We do update all the target folders of filter moves after each move completes, which will cause us to analyze the new messages for junks status, but we fire biff when all the updates are complete, i.e., the headers have been downloaded and mail filters applied, but before junk mail analysis has completed on those folders. The target folders will get analyzed and messages moved to the junk folder (if you've set it up that way) as part of the get new mail process, but biff will still fire in some situations where incoming mail is all junk. It's an improvement, but still not right.

For POP3, we're not even analyzing the filter move target folders until the user opens the folder. That's easy enough to change, but delaying the biff notification until that process is complete for all folders is going to be a challenge because there is no notification of that event.

Comment 5

12 years ago
Your comment was possibly the most helpful thing I've read about this so far! Thank you. I would like to draw your attention, to what was otherwise something of an afterthought though:

"For POP3, we're not even analyzing the filter move target folders until the
user opens the folder."
This _IS_ the bug we are talking about. :) There are obviously two things going on here :
1 - an issue about WHEN junk mail gets analyzed on POP3 filter move target folder (easy fix)
2 - if biff notifies of new mail. (difficult due to lack of 'filter moved' event)

While yes, I did include emphasis on #2 in my comment, fixing #1 "last year" is 90% of what I want.  I don't pay that much attention to biff notices, but when opening the client, I shouldn't see (n) next to my folders indicating new messages, then find out they were all junk.  Can this be fixed tonight?  What can I do to help?  I'll order you pizza, or whatever you need to do so that junk is filtered intuitively after mail is recieved.  We can worry about the notification (biff) later, but that event does need to get created at some point. Thanks so much! Your response really puts me at ease that exact issues of causality are well understood and can therefore be addressed, at least in the case of #1 (the biggest priority in my book).
(Assignee)

Comment 6

12 years ago
I can code something up tonight, but it'll need some testing...
(Assignee)

Comment 7

12 years ago
Created attachment 214440 [details] [diff] [review]
apply junk mail controls to folders that have pop3 mail filtered into them

this is the patch I'm testing right now - I had to tweak my original stab at this because m_filterTargetFolders was getting multiple entries per folder, if a folder had multiple messages filtered into it. I'll run with this for a bit and see if it behaves well.

One regression this will cause is that we'll leave the db's open for folders that have messages filtered into them, if we analzye any of the new messages for junk status. Not sure how I'll deal with that...
(Assignee)

Comment 8

12 years ago
Comment on attachment 214440 [details] [diff] [review]
apply junk mail controls to folders that have pop3 mail filtered into them

this is working well enough for me and is probably worth getting some testing on the trunk.
Attachment #214440 - Flags: superreview?(mscott)

Updated

12 years ago
Attachment #214440 - Flags: superreview?(mscott) → superreview+
(Assignee)

Comment 9

12 years ago
OK, the fix to make spam detection run on local folders that have messages filtered into them has been checked into the trunk and the 1.8 branch. The new mail alert will still fire even if all messages are marked as junk, but spam will be detected and dealt with immediately after the new mail is all finished downloading.

Comment 10

11 years ago
Thanks for finally fixing this, it really rendered the junk filter useless.

Just one thing, if the "new mail"-notification still is shown, then I still get annoyed by spam and the filter still isn't 100% working.

Say I get a spam every 30mins, then I'd have to check my mail every time only to see there are no new messages. IMO still a very unsatisfying situation. Are there any plans to change that?

Comment 11

11 years ago
I think the recent fix would be an improvement but it's _STILL_ not in the production build where it should have been part of an update months ago.  At this point enough time as elapsed to expect the overall design flaw to have been fixed.  The design flaw being the order and way spam filtering is performed.  By now it is reasonable to expect that code to have been changed.  Either move the spam filtering ahead of user filters, or create the event for filter completed and implement it.  Finally, just change how biff is called.  make an event for spam filter-completed.  Okay, that'll take a couple hours of coding, but it's been months.  Call biff from there.  The current bug fix should have distributed to 1.5x updates long ago.  Thanks guys.  Also, I'm talking about the Windows build, but I'm sure the problem is the same across platforms.
(Assignee)

Comment 12

11 years ago
1.5.0.x builds are mainly for security fixes and regressions.

Comment 13

11 years ago
David, you said this patch was checked in -- should this bug be marked fixed, or are you waiting for something else?
I'm not going to be able to verify the fix as I get very little junk making it thru to Thunderbird.

Updated

11 years ago
Blocks: 314124

Comment 14

11 years ago
Pinging David Bienvenu: is this bug FIXED or not?  (comment 13)
(Assignee)

Comment 15

11 years ago
yes, it's fixed, except that we will still fire a new mail notification even if all filtered mail turns out to be junk. I suppose we can open a new bug for that.
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED

Comment 16

11 years ago
(In reply to comment #15)
> yes, it's fixed, except that we will still fire a new mail notification even
> if all filtered mail turns out to be junk. I suppose we can open a new bug
> for that.

Bug 233930 already exists for that problem.  (See also bug 233929.)

Comment 17

11 years ago
*** Bug 200788 has been marked as a duplicate of this bug. ***

Comment 18

11 years ago
Moving this where it can be found.
Component: General → MailNews: Filters
OS: Linux → All
Product: Thunderbird → Core
Hardware: PC → All
Summary: Automatic spam detection on messages moved by a filter. → Automatic junk (spam) detection on messages moved by a filter.
Version: unspecified → Trunk
fixed1.8.1 per comment 9 and bonsai
Keywords: fixed1.8.1

Comment 20

11 years ago
Sorry to drag some comments into an old bug but this feature is really hurting me!  I subscribe to several mailing lists which are moderated and generally contain no spam but the spam filter still checks them. THis means when I am checking my mail remotely a folder with several thousand mails in it have to all be checked for spam which slows the whole thing down needlessly. Also recently my spam filter started being a bit overzealous and trashed loads of messages from legitimate mailing lists.

I really would like a way to disable Junk checks of IMAP folders. I'd say turn them all off by default as 99% of the time the reason for server side filters is to filter specific contacts and notifications etc. The PENDING case listed in this bug is one folder where you DO want checks, but how many others really need it??

Perhaps my case isn't the norm, but I really think this is a lacking feature right now.

A simple tick box in the folder properties would suffice to turn this on or off, and perhaps a global setting for "Check all folders for SPAM" could be available for IMAP accounts?

What do you think? Perhaps there is a new bug for this already? If not please reply here and I can open one.
(Assignee)

Comment 21

11 years ago
Colin, if your filters are moving messages to these folders, they can also mark the messages as nonjunk, and the spam filter won't run on them.

Comment 22

11 years ago
(In reply to comment #21)
> Colin, if your filters are moving messages to these folders, they can also mark
> the messages as nonjunk, and the spam filter won't run on them.

A cunning plan, unfortunately thwarted by the fact that I am using /server side/ filters, not client side.  Admittedly this was a fact I neglected to mention above.

Basically I have several server side sieve scripts that direct various mailing list mails into their own folders and it is these folder I want Thunderbird's junk filter to ignore.

If the server side could add a header in that says "I am not junk" then a lot of spammers would use this ;) I notice there is the option to "trust SpamAssassin" headers, but I'd imagine this is only in one direction (e.g. it believes it when it says it IS junk). Cheers for the idea tho'.

(Assignee)

Comment 23

11 years ago
Yes, that's why I said "If" :-(

The option to trust SpamAssassin headers can work both ways (good and bad), but it only works on the Inbox, because it runs with the normal incoming mail filters.

So, having a per-folder setting to disable junk processing is probably the easiest way to go, though one could argue that trusting spam assassin headers should work on all folders, not just the Inbox. There is an open bug about the per-folder setting...

Comment 24

11 years ago
(In reply to comment #23)
> There is an open bug about the
> per-folder setting...

I thought there might be but couldn't find it in my search - that's why I found this one ;)

A little more digging and I found this:
https://bugzilla.mozilla.org/show_bug.cgi?id=189970

/me is off to use some votes....

Comment 25

11 years ago
(In reply to comment #21)
> Colin, if your filters are moving messages to these folders, they can also mark
> the messages as nonjunk, and the spam filter won't run on them.

David, if we add "Set Junk Status to NOT JUNK", won't that constantly "train" the junk filter? I would prefer that the junk filter completely ignore these messages... I don't want them used to train the junk filter. Is that possible? Or, does the "mark as NOT JUNK" really just mean "tell the junk filter to skip this message"?

Thanks for continuing to follow up on this bug.
(Assignee)

Comment 26

11 years ago
No, if the junk status is set on a message by a filter, the junk filter does not get involved at all - the "good" message is NOT run through the junk filter. It's NOT the same as pressing the "this is junk" button.

Comment 27

11 years ago
David,

I had always assumed that setting the junk status would alter the training data. Thanks for the clarification.
(In reply to comment #7)
> One regression this will cause is that we'll leave the db's open for folders
> that have messages filtered into them, if we analyze any of the new messages
> for junk status. Not sure how I'll deal with that...

David, has the regression resolved?
If not, "leave db's open for folders" possibly one of reasons of other issues such as "junk not moved", claim of too many file handle for a ".msf" file.
Should we test and provide trace data for the "leave db's open for folders"? 
Note: I recently got "Process Monitor" in additon to "DebugView". And I have an mail account to which only spam's arrive(100 per a day). So I now can test with real junks and get mail folder file/directory related activity of Tb on MS Win.
See http://www.microsoft.com/technet/sysinternals/default.mspx for tools on Win.

Comment 29

10 years ago
Hi !

Sorry to comment here again but I still have this problem in my TB 2.0.0.4 :-(

I have an IMAP account and I want to get all messages locally. So I have messages filters defined for that IMAP account to move the messages to an other account (i got messages per POP before with this).

All messages are correctly filtered to the folders in the other account, but no junk processing is done.

Any idea?

Comment 30

10 years ago
I'm also still having this issue in TB 2.0.0.9. I'd love to see this resolved, as I use filtering extensively and running junk mail controls on each folder individually gets very tiring!

Comment 31

10 years ago
Excuse the double-post. I upgraded to 2.0.0.12 today and the issue is still present.

Comment 32

10 years ago
Erin and Ingo, this WFM on both current trunk builds, as well as 2.0.0.9  I have enabled adaptive filters on all accounts - including both servers and Local Folders (haven't tested to see if that matters though.)

So either you are seeing something different, or there is some issue with your local setup. The patch for this bug definitely went into the branch that TB 2 is based on, and it fixed a problem. I suggest that you file a new bug giving a more clear description of what you are seeing if it is still an issue.

Comment 33

10 years ago
Hi Kent,

for my case there already is a new bug 389098 with this issue for IMAP. It would be great if you could check this. It is confirmed.

And maybe bug 389096 has to do with this directly ...

Comment 34

10 years ago
I have figured out what my problem is, and it seems to be another bug.  I have tested this and confirmed that this is the case in my 2.0.0.12 TB: Even if the "Place a copy in" checkbox under Preferences -> Account -> Copies & Folders is unchecked (ie, no sent mail saved), junk mail controls do not run on the selected folder, even though it is greyed out, indicating that it is not a sent mail folder.  

Steps to reproduce:
1) With the "Place a copy in" checkbox checked, select the "Other" radio button and choose a folder which accumulates lots of spam.
2) Uncheck the "Place a copy in" checkbox.  The "Other" radio button will still be selected, but it and the folder drop down box will be greyed out.
3) Even though no sent mail is being saved in the folder, junk mail controls will not run on that folder.

Comment 35

10 years ago
I can confirm this behavior both on TB 2.0.0.9, as well as a trunk build based on 2008-03-27 source.

Good catch, Erin. Can you please file another bug for this, cc me, and then I'll confirm it? Thanks.

Comment 36

10 years ago
New bug is bug 426950.
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.