Closed Bug 389098 Opened 17 years ago Closed 1 year ago

Automatic junk (spam) detection on messages moved by a filter from IMAP - should move to junk folder directly after move and detecting junk

Categories

(MailNews Core :: Filters, defect)

defect

Tracking

(thunderbird_esr102 unaffected)

RESOLVED FIXED
Thunderbird 3.0b4
Tracking Status
thunderbird_esr102 --- unaffected

People

(Reporter: ingo, Assigned: rkent)

References

Details

(Whiteboard: [rkent cannot reproduce][patchlove])

Attachments

(1 file, 2 obsolete files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 Firefox/2.0.0.5
Build Identifier: TB 2.0.0.5 and before

This is a followup to Bug 329569 because the same issue that was fixed for messages filtered from POP3 to a local folder exists for IMAP too.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.



Right now if you get a spam message and the message gets filtered into a folder
we won't run spam detection on the messages until the user selects the target
folder. This means we show in the UI that there's new messages in the target
folder (even if it's spam and we'd know it's spam if we'd check) and we fire
off notifications about the new message(s).

Seems like the user experience would be much better if we'd simply run a spam
detection pass on the target folder after a filter has moved messages into the
target folder, and then only once that's done, we'd notify the user about the
new messages if there really are new non-spam messages in the folder.
Are you sure this is still happening? Though bug 329569 comment 4 indicate it did, when I tried this the junk detection run before the message was moved (and so it never was.) 

I tested this on Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b3pre) Gecko/2008012403 Lightning/0.6a1 Thunderbird/3.0a1pre ID:2008012403 with the trust spamassassin option.
OS: Windows XP → All
Hardware: PC → All
I currently use TB 2.0.0.9 (20071031) [most current released version] and I have this problem there.
I tried some TB 3.0pre versions in 2007 and had the same problem. Maybe someone has fixed this in the meantime without knowing this bug? Do the changelog tell anything of this?

How can I help to test this?
You could test it with a recent nightly: 
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/

My test assumed the "trust spamassassin" and normal junk detection happens after each other.

And we are talking about thunderbird's filters, not server side ones, right?
Yes we talk about the tb filters.

The problem is that emails are filtered and moved to the correct folders. In my case they will be filtered not to folders on the server, but to local folders. But junk detection is not run after all filters have been applied.
For POP-messages this was changed in the bug referenced, so that junk filters will run for all folders where messages have been moved too directly after the filter have been applied.

For those IMAP cases the junk processing is done when i enter the folder ... then maybe all new messages are detected as junk and so there are no real new messages here.

I will see if I can test the latest trunk without damaging my normal emails ... I have no real ways to only "test" a little bit so I have to use my live installation for the tests ... ;-(
Make sure to make backups of the filters if you are testing current trunk - see bug 413680.
Ok, Magnus ...

looks good after a first view. The problem is that I was not really able to test it with real junk mails :-( I needed to use a special mail-account.

And it seems that bug 389096 does not happen anymore too :-)

So this looks great for latest trunk (3.0a1pre (2008021003)), but it would be interesting to have this fixed in the latest v2.x too. Do you know what patch fixed this?

Or other idea for me? I would like to have this, but I don't know if it is a good idea to use the nightlies in "real-life", or ?!
Don't know what would have fixed it. 

It's possible to use nightlies in real-life, as long as you are aware of the risks... not that I have ever had much real problems with them. 
I have found a spam mail and managed it to resend it to my test-account.

The following happened:
1.) Messages were filtered to local folder and the folder-icon stated that there are "new" message
2.) Directly after this the "new message" flag of the folder got removed, but the number of unread messages is the number of messages in the folder.

This is completely correct because the new messages filted to that folder were junk, BUT: The junk messages were not moved to the junk folder as defined in the settings. When I run the "junk filters" on that folder then they are moved.

So there is something not correct ... :-(

Could you retest this? When you have a junk message simply mark it as "unread" and copy it back to the incoming IMAP-folder. When you now get the messages again they will be filtered again.
My setup is really simple: I use an IMAP mail Account and I use one filter-rule "move all messages to local folder X". That's all to test :-)
Re-tested with adaptive filters. After the mail is moved from IMAP to a local folder, the detected junk mails are not moved to the junk folder.
Status: UNCONFIRMED → NEW
Component: General → MailNews: Filters
Ever confirmed: true
Product: Thunderbird → Core
QA Contact: general → filters
Summary: Automatic junk (spam) detection on messages moved by a filter from IMAP → Automatic junk (spam) detection on messages moved by a filter from IMAP - should move to junk folder directly after move and detecting junk
Thanx for re-testing and confirmation!

Maybe bug 389096 has to do with this directly ...
Product: Core → MailNews Core
I have a specific patch for which I would like a home, and I'll try this bug for now. Ingo, feel free to object and I will move it, as this is not only your bug but your issue as reported in other places. I originally reported this work in bug 429950, but Ingo suggested rightly that this might be a better place.

My symptoms are bug 329569 comment 29, which per bug 329569 comment 31 is being tracked here. Yet the definition of this bug has drifted toward the issue of junk move not happening. My issue is junk processing not happening on the local folder after a move from IMAP. The issue that I am dealing with may be a race condition, so it is possible that its coming and goings have been somewhat random.

Here is what is happening. When IMAP does its move, it fires off the copy service, then when it thinks the move is complete, sends the IMAP server a request to set SEEN and DELETED flags on the old message. This happens before the copy service has finalized the copy to the local folder. When the IMAP server gets the SEEN and DELETED flags, it responds back with those flags. TB sees the response with those flags, and then corrects the message header to have the correct values, that is READ, and therefore not new. Meanwhile, the copy service finishes, and tries to create the new message header, copying properties from the old one. But the old header is now READ, and therefore does not get added as a new message. CallFilterPlugins on the local folder will only run spam processing on new messages, so it does not run processing on the new message.

The fix is fairly simple, just persist the message flags in the copy state (see attachment 393308 [details] [diff] [review]). I need to check it some more though.
Assignee: nobody → kent
Status: NEW → ASSIGNED
Oops, original report was in bug 426950, not 429950.
Hi,

with "junk move" I mean that the messages are not checked for junk and not moved to junk-folder... So i mean "junk processing" generally ... so I think it should exactly be the same :-)

And I think when we really get this fixed here, then bug 389096 should be solved too because bug 389096 only exists because the junk processing does not appear directly with message-arrival. When the messages are processed for junk directly with arrival then no "old" junk messages should exists when new arrive, or ?!
I've expanded the original fix in a few ways. This is still a WIP, but with no known issues.

I dealt with:

1) Upstream message parsing has already gone to a great deal of work to set the message flags. I don't want to override that, so I only used the READ and NEW flags from the saved message header.

2) I discovered an error in bug 459680 where I put "flag" when I meant "flags" in the dontPreserve lists.

3) With changes to NEW flag handling, so that the NEW flag from the header can now be trusted at the time of filter application, I don't think we need all of that IMAP new flag correction stuff.

This passes existing unit tests. I suppose I should write a unit test for this issue.
Blocks: 509423
Whiteboard: [has patch]
The patch for this bug is fixing an issue that is broader than just junk processing. Here are the STR:

1) On an IMAP account, create a filter (Subject Contains Test) -> (Moveto LocalFolder/Inbox)

2) Send a test message to the IMAP account that matches the filter.

Expected results: Message arrives in local inbox, with Unread and New status
Actual results: Message arrives in local inbox, with Unread (but not New) status.

The cause of this is described in comment 11. It may be timing dependent, and/or dependent on the exact IMAP configuration. But it is 100% reproducible for me, and as this bug describes one of the side effects is that junk processing will not run on the moved message.
Attachment #394566 - Attachment is obsolete: true
Attachment #398192 - Flags: superreview?(bienvenu)
Attachment #398192 - Flags: review?(bienvenu)
Whiteboard: [has patch] → [needs r/sr bienvenu]
If there is no controversy here, it would be good to get this into TB beta 4.
Target Milestone: --- → Thunderbird 3.0b4
Comment on attachment 398192 [details] [diff] [review]
Removed flag->flags fix, which I moved to another bug

this looks ok, but can you add some parens here so that it's clear what the code is trying to do?
Comment on attachment 398192 [details] [diff] [review]
Removed flag->flags fix, which I moved to another bug

this is the code I was referring to:
+          newHdr->SetFlags(newFlags & ~readAndNew | mCopyState->m_flags & readAndNew);
Attached patch Add parenthesesSplinter Review
Attachment #398192 - Attachment is obsolete: true
Attachment #399480 - Flags: superreview?(bienvenu)
Attachment #399480 - Flags: review?(bienvenu)
Attachment #398192 - Flags: superreview?(bienvenu)
Attachment #398192 - Flags: review?(bienvenu)
Comment on attachment 399480 [details] [diff] [review]
Add parentheses

r/sr=me, and a=me if you want to land this for b4.
Attachment #399480 - Flags: superreview?(bienvenu)
Attachment #399480 - Flags: superreview+
Attachment #399480 - Flags: review?(bienvenu)
Attachment #399480 - Flags: review+
Keywords: checkin-needed
Whiteboard: [needs r/sr bienvenu] → [needs checkin]
Checked in: http://hg.mozilla.org/comm-central/rev/8cc77ffdd9a5
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Whiteboard: [needs checkin]
Hi Kent,

I now use TB3 release version regulary and it works - with one exception: All messages that are moved to the current active/selected folder are not processed for junk. All other folders work great.
I see this using the JunQuilla extension and these messages have no Junk-status.

May your fix have problems with handling this for the current selected folder?

Ways to repeat:
1.) Prepare an email to IMAP and have a filter to move this message into a local folder.
2.) Select the folder the message will be filtered to
3.) send the mail
4.) mail will be received, filtered to local folder, but without any junk status.

Even when leaving and re-selection the folder no junk progressing is done ...

Or are we now in a different behaviour?

Ingo F
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to comment #23)
> 
> I now use TB3 release version regulary and it works - with one exception: All
> messages that are moved to the current active/selected folder are not processed
> for junk. All other folders work great.

I was not able to reproduce this behavior. In my tests, emails that were sent to the local folder from a filter on incoming IMAP did receive junk processing.

In this case, the governing Junk Settings is for the "Local Folders" account. Youmust have junk processing enabled there for this to work.
Whiteboard: [rkent cannot reproduce]
The problem is that the same folder works some time with junk-processing after-filtering and some times not. The only thing I was able to find our was that it mostly not works when the folder is currently selected and a mesage is filtered to ...

I checked the settings and junk processing is enabled for the local account and the local folder.

Any idea how to figure out the reason?
More infos: In TB3 the folder-name is set to "black-bold" when unread messages are in and it will get a "dark-blue" color additional to the bold I assume when new messages have arrived. Correct?

What I noticed is: Any time the junk-processing does not took place, the folder where messages were filtered to is inly black-bold and in all cases where it worked the color is dark-blue-and-bold.

May there be race-conditions where the persistance of the message flags does not work or can not be stored completely and therefor it is the same efect then before?

Can I use any logs to find out more?
Kent, do you really were not able to reproduce?
Do you have checked that a message was moved to the folder that was currently selected and so "opened" ... because here I can always reproduce it ...
Is it possinle that the messages are all marked as "not new" as soon as you leave the folder and that this is the reason the junk processing is not done ... because for the other folders the junk processing is done as soon as i open/enter the folder ...
(In reply to comment #29)
> Is it possinle that the messages are all marked as "not new" as soon as you
> leave the folder and that this is the reason the junk processing is not done

Junk processing uses "new" status to determine which messages needed junk processing (with a kludge to preserve the list when "new" is cleared for a folder). So issues with "new" are likely to be behind whatever you are seeing.

> ... because for the other folders the junk processing is done as soon as i
> open/enter the folder ...

As I understand it this was the behavior of earlier versions of Thunderbird, but not of the current version, at least with the configuration that I currently use. Are you sure that you see this behavior? I can't see how that is possible on local folders at all.
Is it possible that a "POP3"-Account from Thunderbird acts different then real "Local Folders"?

I have several "real" local forders and I have one folder-tree too that is in an POP3-Account. I used POP3 earlier and that account is still there even if no new messages arrive their.
I have my new IMAP-mail Account where i use filters to filter the messages to the Local folders as well as to the "old" POP3-Account (where all folders are local too) ... Can this be the reason?
Blocks: 426950
Severity: minor → S4
Duplicate of this bug: 426950

It's not clear to me this should stay open. It was reopened in comment 23 by the reporter "All messages that are moved to the current active/selected folder are not processed for junk.", who is now gone. Should we close this fixed?

Note - patch was from duped bug 426950.

Flags: needinfo?(mkmelin+mozilla)
Whiteboard: [rkent cannot reproduce] → [rkent cannot reproduce][patchlove]

Agreed shouldn't have been reopened. New bug for new issues if, if any.

Status: REOPENED → RESOLVED
Closed: 15 years ago1 year ago
Flags: needinfo?(mkmelin+mozilla)
Resolution: --- → FIXED
Duplicate of this bug: 509423
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: