Automatic junk (spam) detection on messages moved by a filter from IMAP - should move to junk folder directly after move and detecting junk

REOPENED
Assigned to

Status

MailNews Core
Filters
--
minor
REOPENED
11 years ago
6 years ago

People

(Reporter: Ingo Fischer, Assigned: rkent)

Tracking

(Blocks: 2 bugs)

unspecified
Thunderbird 3.0b4
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [rkent cannot reproduce])

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

11 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 Firefox/2.0.0.5
Build Identifier: TB 2.0.0.5 and before

This is a followup to Bug 329569 because the same issue that was fixed for messages filtered from POP3 to a local folder exists for IMAP too.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.



Right now if you get a spam message and the message gets filtered into a folder
we won't run spam detection on the messages until the user selects the target
folder. This means we show in the UI that there's new messages in the target
folder (even if it's spam and we'd know it's spam if we'd check) and we fire
off notifications about the new message(s).

Seems like the user experience would be much better if we'd simply run a spam
detection pass on the target folder after a filter has moved messages into the
target folder, and then only once that's done, we'd notify the user about the
new messages if there really are new non-spam messages in the folder.

Comment 1

11 years ago
Are you sure this is still happening? Though bug 329569 comment 4 indicate it did, when I tried this the junk detection run before the message was moved (and so it never was.) 

I tested this on Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3pre) Gecko/2008012403 Lightning/0.6a1 Thunderbird/3.0a1pre ID:2008012403 with the trust spamassassin option.
OS: Windows XP → All
Hardware: PC → All
(Reporter)

Comment 2

11 years ago
I currently use TB 2.0.0.9 (20071031) [most current released version] and I have this problem there.
I tried some TB 3.0pre versions in 2007 and had the same problem. Maybe someone has fixed this in the meantime without knowing this bug? Do the changelog tell anything of this?

How can I help to test this?

Comment 3

11 years ago
You could test it with a recent nightly: 
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/

My test assumed the "trust spamassassin" and normal junk detection happens after each other.

And we are talking about thunderbird's filters, not server side ones, right?
(Reporter)

Comment 4

11 years ago
Yes we talk about the tb filters.

The problem is that emails are filtered and moved to the correct folders. In my case they will be filtered not to folders on the server, but to local folders. But junk detection is not run after all filters have been applied.
For POP-messages this was changed in the bug referenced, so that junk filters will run for all folders where messages have been moved too directly after the filter have been applied.

For those IMAP cases the junk processing is done when i enter the folder ... then maybe all new messages are detected as junk and so there are no real new messages here.

I will see if I can test the latest trunk without damaging my normal emails ... I have no real ways to only "test" a little bit so I have to use my live installation for the tests ... ;-(

Comment 5

11 years ago
Make sure to make backups of the filters if you are testing current trunk - see bug 413680.
(Reporter)

Comment 6

10 years ago
Ok, Magnus ...

looks good after a first view. The problem is that I was not really able to test it with real junk mails :-( I needed to use a special mail-account.

And it seems that bug 389096 does not happen anymore too :-)

So this looks great for latest trunk (3.0a1pre (2008021003)), but it would be interesting to have this fixed in the latest v2.x too. Do you know what patch fixed this?

Or other idea for me? I would like to have this, but I don't know if it is a good idea to use the nightlies in "real-life", or ?!

Comment 7

10 years ago
Don't know what would have fixed it. 

It's possible to use nightlies in real-life, as long as you are aware of the risks... not that I have ever had much real problems with them. 
(Reporter)

Comment 8

10 years ago
I have found a spam mail and managed it to resend it to my test-account.

The following happened:
1.) Messages were filtered to local folder and the folder-icon stated that there are "new" message
2.) Directly after this the "new message" flag of the folder got removed, but the number of unread messages is the number of messages in the folder.

This is completely correct because the new messages filted to that folder were junk, BUT: The junk messages were not moved to the junk folder as defined in the settings. When I run the "junk filters" on that folder then they are moved.

So there is something not correct ... :-(

Could you retest this? When you have a junk message simply mark it as "unread" and copy it back to the incoming IMAP-folder. When you now get the messages again they will be filtered again.
My setup is really simple: I use an IMAP mail Account and I use one filter-rule "move all messages to local folder X". That's all to test :-)

Comment 9

10 years ago
Re-tested with adaptive filters. After the mail is moved from IMAP to a local folder, the detected junk mails are not moved to the junk folder.
Status: UNCONFIRMED → NEW
Component: General → MailNews: Filters
Ever confirmed: true
Product: Thunderbird → Core
QA Contact: general → filters
Summary: Automatic junk (spam) detection on messages moved by a filter from IMAP → Automatic junk (spam) detection on messages moved by a filter from IMAP - should move to junk folder directly after move and detecting junk
(Reporter)

Comment 10

10 years ago
Thanx for re-testing and confirmation!

Maybe bug 389096 has to do with this directly ...
Product: Core → MailNews Core
(Assignee)

Comment 11

9 years ago
I have a specific patch for which I would like a home, and I'll try this bug for now. Ingo, feel free to object and I will move it, as this is not only your bug but your issue as reported in other places. I originally reported this work in bug 429950, but Ingo suggested rightly that this might be a better place.

My symptoms are bug 329569 comment 29, which per bug 329569 comment 31 is being tracked here. Yet the definition of this bug has drifted toward the issue of junk move not happening. My issue is junk processing not happening on the local folder after a move from IMAP. The issue that I am dealing with may be a race condition, so it is possible that its coming and goings have been somewhat random.

Here is what is happening. When IMAP does its move, it fires off the copy service, then when it thinks the move is complete, sends the IMAP server a request to set SEEN and DELETED flags on the old message. This happens before the copy service has finalized the copy to the local folder. When the IMAP server gets the SEEN and DELETED flags, it responds back with those flags. TB sees the response with those flags, and then corrects the message header to have the correct values, that is READ, and therefore not new. Meanwhile, the copy service finishes, and tries to create the new message header, copying properties from the old one. But the old header is now READ, and therefore does not get added as a new message. CallFilterPlugins on the local folder will only run spam processing on new messages, so it does not run processing on the new message.

The fix is fairly simple, just persist the message flags in the copy state (see attachment 393308 [details] [diff] [review]). I need to check it some more though.
Assignee: nobody → kent
Status: NEW → ASSIGNED
(Assignee)

Comment 12

9 years ago
Oops, original report was in bug 426950, not 429950.
(Reporter)

Comment 13

9 years ago
Hi,

with "junk move" I mean that the messages are not checked for junk and not moved to junk-folder... So i mean "junk processing" generally ... so I think it should exactly be the same :-)

And I think when we really get this fixed here, then bug 389096 should be solved too because bug 389096 only exists because the junk processing does not appear directly with message-arrival. When the messages are processed for junk directly with arrival then no "old" junk messages should exists when new arrive, or ?!
(Assignee)

Comment 14

9 years ago
Created attachment 394566 [details] [diff] [review]
Fix flag->flags, don't need old IMAP New stuff

I've expanded the original fix in a few ways. This is still a WIP, but with no known issues.

I dealt with:

1) Upstream message parsing has already gone to a great deal of work to set the message flags. I don't want to override that, so I only used the READ and NEW flags from the saved message header.

2) I discovered an error in bug 459680 where I put "flag" when I meant "flags" in the dontPreserve lists.

3) With changes to NEW flag handling, so that the NEW flag from the header can now be trusted at the time of filter application, I don't think we need all of that IMAP new flag correction stuff.

This passes existing unit tests. I suppose I should write a unit test for this issue.

Updated

9 years ago
Blocks: 509423
(Assignee)

Updated

9 years ago
Whiteboard: [has patch]
(Assignee)

Comment 15

9 years ago
The patch for this bug is fixing an issue that is broader than just junk processing. Here are the STR:

1) On an IMAP account, create a filter (Subject Contains Test) -> (Moveto LocalFolder/Inbox)

2) Send a test message to the IMAP account that matches the filter.

Expected results: Message arrives in local inbox, with Unread and New status
Actual results: Message arrives in local inbox, with Unread (but not New) status.

The cause of this is described in comment 11. It may be timing dependent, and/or dependent on the exact IMAP configuration. But it is 100% reproducible for me, and as this bug describes one of the side effects is that junk processing will not run on the moved message.
(Assignee)

Comment 16

9 years ago
Created attachment 398192 [details] [diff] [review]
Removed flag->flags fix, which I moved to another bug
Attachment #394566 - Attachment is obsolete: true
Attachment #398192 - Flags: superreview?(bienvenu)
Attachment #398192 - Flags: review?(bienvenu)
(Assignee)

Updated

9 years ago
Whiteboard: [has patch] → [needs r/sr bienvenu]
(Assignee)

Comment 17

9 years ago
If there is no controversy here, it would be good to get this into TB beta 4.
Target Milestone: --- → Thunderbird 3.0b4

Comment 18

9 years ago
Comment on attachment 398192 [details] [diff] [review]
Removed flag->flags fix, which I moved to another bug

this looks ok, but can you add some parens here so that it's clear what the code is trying to do?

Comment 19

9 years ago
Comment on attachment 398192 [details] [diff] [review]
Removed flag->flags fix, which I moved to another bug

this is the code I was referring to:
+          newHdr->SetFlags(newFlags & ~readAndNew | mCopyState->m_flags & readAndNew);
(Assignee)

Comment 20

9 years ago
Created attachment 399480 [details] [diff] [review]
Add parentheses
Attachment #398192 - Attachment is obsolete: true
Attachment #399480 - Flags: superreview?(bienvenu)
Attachment #399480 - Flags: review?(bienvenu)
Attachment #398192 - Flags: superreview?(bienvenu)
Attachment #398192 - Flags: review?(bienvenu)

Comment 21

9 years ago
Comment on attachment 399480 [details] [diff] [review]
Add parentheses

r/sr=me, and a=me if you want to land this for b4.
Attachment #399480 - Flags: superreview?(bienvenu)
Attachment #399480 - Flags: superreview+
Attachment #399480 - Flags: review?(bienvenu)
Attachment #399480 - Flags: review+
(Assignee)

Updated

9 years ago
Keywords: checkin-needed
Whiteboard: [needs r/sr bienvenu] → [needs checkin]
Checked in: http://hg.mozilla.org/comm-central/rev/8cc77ffdd9a5
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Whiteboard: [needs checkin]
(Reporter)

Comment 23

9 years ago
Hi Kent,

I now use TB3 release version regulary and it works - with one exception: All messages that are moved to the current active/selected folder are not processed for junk. All other folders work great.
I see this using the JunQuilla extension and these messages have no Junk-status.

May your fix have problems with handling this for the current selected folder?

Ways to repeat:
1.) Prepare an email to IMAP and have a filter to move this message into a local folder.
2.) Select the folder the message will be filtered to
3.) send the mail
4.) mail will be received, filtered to local folder, but without any junk status.

Even when leaving and re-selection the folder no junk progressing is done ...

Or are we now in a different behaviour?

Ingo F
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

9 years ago
Duplicate of this bug: 389096
(Assignee)

Comment 25

9 years ago
(In reply to comment #23)
> 
> I now use TB3 release version regulary and it works - with one exception: All
> messages that are moved to the current active/selected folder are not processed
> for junk. All other folders work great.

I was not able to reproduce this behavior. In my tests, emails that were sent to the local folder from a filter on incoming IMAP did receive junk processing.

In this case, the governing Junk Settings is for the "Local Folders" account. Youmust have junk processing enabled there for this to work.
(Assignee)

Updated

9 years ago
Whiteboard: [rkent cannot reproduce]
(Reporter)

Comment 26

9 years ago
The problem is that the same folder works some time with junk-processing after-filtering and some times not. The only thing I was able to find our was that it mostly not works when the folder is currently selected and a mesage is filtered to ...

I checked the settings and junk processing is enabled for the local account and the local folder.

Any idea how to figure out the reason?
(Reporter)

Comment 27

9 years ago
More infos: In TB3 the folder-name is set to "black-bold" when unread messages are in and it will get a "dark-blue" color additional to the bold I assume when new messages have arrived. Correct?

What I noticed is: Any time the junk-processing does not took place, the folder where messages were filtered to is inly black-bold and in all cases where it worked the color is dark-blue-and-bold.

May there be race-conditions where the persistance of the message flags does not work or can not be stored completely and therefor it is the same efect then before?

Can I use any logs to find out more?
(Reporter)

Comment 28

9 years ago
Kent, do you really were not able to reproduce?
Do you have checked that a message was moved to the folder that was currently selected and so "opened" ... because here I can always reproduce it ...
(Reporter)

Comment 29

9 years ago
Is it possinle that the messages are all marked as "not new" as soon as you leave the folder and that this is the reason the junk processing is not done ... because for the other folders the junk processing is done as soon as i open/enter the folder ...
(Assignee)

Comment 30

9 years ago
(In reply to comment #29)
> Is it possinle that the messages are all marked as "not new" as soon as you
> leave the folder and that this is the reason the junk processing is not done

Junk processing uses "new" status to determine which messages needed junk processing (with a kludge to preserve the list when "new" is cleared for a folder). So issues with "new" are likely to be behind whatever you are seeing.

> ... because for the other folders the junk processing is done as soon as i
> open/enter the folder ...

As I understand it this was the behavior of earlier versions of Thunderbird, but not of the current version, at least with the configuration that I currently use. Are you sure that you see this behavior? I can't see how that is possible on local folders at all.
(Reporter)

Comment 31

9 years ago
Is it possible that a "POP3"-Account from Thunderbird acts different then real "Local Folders"?

I have several "real" local forders and I have one folder-tree too that is in an POP3-Account. I used POP3 earlier and that account is still there even if no new messages arrive their.
I have my new IMAP-mail Account where i use filters to filter the messages to the Local folders as well as to the "old" POP3-Account (where all folders are local too) ... Can this be the reason?

Updated

6 years ago
Blocks: 426950
You need to log in before you can comment on or make changes to this bug.