271930 - Filter action Mark as "Not Junk" doesn't necessarily prevent categorization of message as junk

Status: RESOLVED DUPLICATE of bug 285922

(MailNews Core :: Filters, defect)

Description

[Ken Seehof]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1

Any filter that attempts to set the junk status fails to have any effect,
whether or not multiple effects are specified.  The steps below are one example.
 I have tried many other examples and have reduced the issue to be specifically
related to setting the junk status flag.

Having looked at several other bug reports relating to filters, I believe that
many of these reports might be masked symptoms as compared to this bug report,
in that they do not point out the significance of the junk status flag.

I speculate that this is due to some erroneous interaction between the filter
system and the junk mail controls system.



Reproducible: Always
Steps to Reproduce:
1. Create a simple filter that moves a message to junk.
2. Test the filter in a folder containing one matching message.
3. Move the message back to inbox.
4. Add "Set junk status to junk" to the filter.
5. Repeat step 2.

Actual Results:  
1. Create a simple filter that moves a message to junk.
2. Test the filter in a folder containing one matching message.
   -> test message is moved successfully, i.e. the filter condition works fine.
3. Move the message back to inbox.
4. Add "Set junk status to junk" to the filter.
5. Repeat step 2.
   -> test message is not marked as junk, and does not get moved.

Note: I have done some more tests, and in some cases, the message gets moved,
but does not get marked as junk.  I have never successfully created a filter
that marks mail as junk.


Expected Results:  
...
5. Repeat step 2.
   -> test message should be marked as junk and get moved.


If filters and junk mail controls worked together in harmony they would be
really quite powerful.  In my case, I have two email addresses at
neuralintegrator.com.  One of these is guaranteed to receive only spam (I have
occaisionally used it for online transactions that are likely to attract spam in
order to protect my "real" address).  Unfortunately many spammers have both of
my addresses in their list.  I would like to use this fact to my advantage by
making a filter that marks mail directed at my "spam address" as junk mail,
which would provide samples that would train the junk mail bayesian network to
remove the same and similar junk mail from my "real account".  I could take it a
step further by making my "spam account" highly visible to spiders on an unused
page on my website!

Comment 1

[Ken Seehof]

Many bugs relating to filters seem to get marked as unreproducable.  It's easy
for people to make mistakes when creating filters, so it may be tempting for you
to assume any particular bug is actually user error.  However, I think there are
many real bugs in the filter system.  So I strongly encourage you to test the
filter system quite vigorously, covering every special case, and really try to
expose as many problems as possible.  The filter system is one of the most
important features to me, and it is one of the main reasons I've switched from
Outlook.  It would be really great if it actually worked :)  I'm looking forward
to the next release with lots of bug fixes.

Thanks,
- Ken

Comment 2

[Werner Kratochwil]

Hi!
I experienced similar behaviour:
I wanted to use a filter to mark messages as junk, whose subject contains the
word "*****SPAM*****" (a service from my e-mail provider). 
At first I just wanted it to change the junk status - nothing happened - even
the filter protocol did not log anything. 
Then I added the rule to sign all those messages and change the junk status. The
junk status did not change, the signing did work. 
Then I changed it to moving to the Junk folder and change the Junk status - the
junk status did not change but the messages were moved to the Junk folder. 

Please ask, if you want any more cases tested. 
Nice regards
Werner

Comment 3

[Ken Seehof]

Also, filters that mark as "not junk" have no effect.

This is important because I would like to be able to use rules to bypass the
junk filter to prevent false positives in cases where I know certain emails are
not junk based on header information.

Comment 4

[Jeongkyu Kim]

In thunderbird, each folder type(local, imap and news) has different
implmentation for applying fiter.

nsParseNewMailState::ApplyFilterHit()
nsImapMailFolder::ApplyFilterHit()
nsNNTPNewsgroupList::ApplyFilterHit()

These implmentations seem to be only for newly coming message. For existing
messages, there is another implmentation in
nsMsgFilterAfterTheFact::ApplyFilter(). If you see the method, there is no
handling for 'nsMsgFilterAction::JunkScore' action. So, I implemented it and
created the patch.

However, the problem doesn't stop here. When you move or copy messages into
_IMAP folder_, some properties such as junk score or priority is lost. Thus,
filter has move or copy action (into IMAP folder) with 'just status' action, the
status will be lost after moving or copying. This bug is addressed in bug 261297.

Comment 5

[Jeongkyu Kim]

Attachment: patch for handling junkscore action of filter

Comment 6

[Jeongkyu Kim]

Attachment: patch for handling junkscore action of filter - revised

I just found that 'junkscoreorigin' should be 'user' or 'plugin'. In the case
of 'message filter', 'user' seems to be appropriate.

Comment 7

[Jeongkyu Kim]

Attachment: patch for handling junkscore action of filter - revised

I found 'break;' was missing in the previous patch. It was fixed in this patch.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
As you say, there are two problems when POP3. 

Ken Seehof(the bug opener)'s case, which is DUP of Bug 244374.
Junk marked mail by "Mark as Junk" by filter will be moved to "Junk" if filering
is done automaticaly for incoming mail, but it will not be moved to "Junk" when
manual filtering.
Workaround for this problem is ;
 - Add action of "Move to Junk folder" to the filter rule
 
Ken Seehof's second case in Comment #3, "Mark as not Junk" case, is different
problem.
Since message filter is scheduled before Junk mail filtering, and since one time
"Marking as not Junk" is usually not sufficient for preventing "Mariking as
Junk" by Junk mail control, the mail will probably be marked as Junk by "Junk
Mail Control".
To improve this, I think that junk filtering should distingush status of "Junk
score is not set"(not checked yet) and "Junk score is zero"(checked and set as
not Junk),
  
And different problem exist when IMAP in addition to bug 261297 you mentioned in
comment #4.
 - Because Junk mail control is executed all folders under INBOX,
   moving mail to folder under INBOX by filter can't prevent Junk detection
   by Junk mail control.
   (already opened bug, I can't recall the number though).

Jeongkyu Kim, which problem will be resolved by your patch?
If first problem, I think your patch should be proposed to Bug 244374.

Comment 9

[Jeongkyu Kim]

(In reply to comment #8)
> 
> Jeongkyu Kim, which problem will be resolved by your patch?
> If first problem, I think your patch should be proposed to Bug 244374.

WADA, thanks for your explanation. I will propose my patch to the bug.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Ken Seehof, your original problem in comment #0 is DUP of Bug 244374, and
problem in your comment #3 is different problem from comment #0.
There are 2 ways.
(a) Close this bug as DUP of Bug 244374, then open new bug for comment #3 problem.
(b) Change summary to approprite one for comment #3 problem, then keep this bug
open for it.
It's up to you.

Comment 11

[Ken Seehof]

The summary has been modified.  The previous summary was a dup of bug 244374.
Although apparently related, this is a different problem.

Comment 12

[Howard Chu]

(In reply to comment #11)
> The summary has been modified.  The previous summary was a dup of bug 244374.
> Although apparently related, this is a different problem.

My patch in bug #223591 will prevent marked messages from being reclassified by
the junk filter. But that code currently only works for POP3 accounts, it
deadlocks Mozilla if you use IMAP. I'm still playing with the code to try to get
IMAP behaving.

Comment 13

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 14

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 15

[Wayne Mery (:wsmwk)]

fixed in bug 285922?