367920 - Filter action Set Junk Status to Junk does not work correctly

Status: VERIFIED FIXED

(Thunderbird :: General, defect)

Description

[Nick Howitt]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Build Identifier: version 2 beta 2 (20070116)

My set up is with individual POP accounts and not using local filters. Junk controls are enabled and set to move the junk messages to the Junk folder for that account.
My ISP has a spam filter and inserts [SPAM?] in the beginning of the subject line.
I have a rule "When subject contains [SPAM?]" then Set Junk Status to Junk.
If I receive a [SPAM?] prefixed mail into the account, the rule does nothing. If I then run the rule manually, the Junk status is correctly set.
If I run the filter filter manually, although it marks the messages as junk, it does not move them to my Junk folder. I would expect it to do this as if I flag messages as junk manually, they automatically get moved to the Junk folder.
I am not sure if this should be 2 bugs:
1 - the action "set junk status as junk" does nothing with incoming mail but flags mail as junk when run manually.
2 - the action "Set junk status to Junk" does not obey the preference to automatically move messages to the Junk folder

Reproducible: Always

Steps to Reproduce:
1.See details
2.
3.
Actual Results:  
See details

Expected Results:  
the Set junk Status to Junk action should work automatically with incoming mail and it should obey the preference to move the mail to a designated folder if set.

Comment 1

[Nick Howitt]

Also happens in TB2 Beta 2

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.2pre) Gecko/20070116 Thunderbird/2.0b2 ID:2007011615

Comment 2

[Patrick Brunschwig]

I can confirm this. Also the opposite doesn't work: I have the following filter:
"if sender contains [something]" -> set junk status to NOT Junk

But every morning I find the matching message marked as junk in the Junk folder. This worked fine with TB 1.5.

Comment 3

[Wayne Mery (:wsmwk)]

regression?

Comment 4

[Scott MacGregor]

Hmm, using the 07/11 thunderbird 2 nightly, I can't reproduce this. 


I have a simple filter that just does:

If the subject contains test then Mark as Junk

and that works. 

Wayne did you have any luck reproducing this? Patrick/reporter, does the filter have other actions associated with it like moving the message to another folder?

Comment 5

[Patrick Brunschwig]

Setting to "not junk" is the only action I have defined for this rule. I'll retry with a nightly build.

Comment 6

[Nick Howitt]

It is hard to comment objectively as TB's own Spam Filter may be learning at the same time.
I did have 2 actions on the filter - 1) set Junk Status to Junk and 2) Move to junk folder on ..... (not a local folder). I removed 2 and the next seven or so messages worked correctly. I turned the PC on today, picked up e-mails and 7 were marked as Junk went to the Junk folder correctly, but one did not. It did not get marked as Junk and was left in my inbox. Manually running the filters marks this message as junk but does not move it to my junk folder.
I have another account with its own folders where I have the same 2 actions in a rule as above. The rule moved its messages to Junk but did not set the Junk Status to Junk

In response to comment 3, 1.5.x was exactly the same so it is not a regression from then.

Comment 7

[Patrick Brunschwig]

Attachment: screenshot

It's definitely still not working in all cases. See attached screenshot -- the new mail should have gone to junk but stayed in my Inbox

Comment 8

[David :Bienvenu]

I thought that filters that mark mail as junk don't automatically move them to the junk folder - you have to explicitly add that action yourself. Or do they sometimes get moved to the junk folder because the mail filter marked them as junk?

Comment 9

[Nick Howitt]

(In reply to comment #8)
> I thought that filters that mark mail as junk don't automatically move them to
> the junk folder - you have to explicitly add that action yourself. Or do they
> sometimes get moved to the junk folder because the mail filter marked them as
> junk?
> 
I would expect the action to obey the "Move new junk messages to:" preference on the account, however that is my personal opinion and I would accept it if the decision was that you had to move the Junk as a separate action.
This is a secondary issue. I cannot even get the action to set the Junk Status to Junk.

Comment 10

[Wayne Mery (:wsmwk)]

Scott in comment #4
> Wayne did you have any luck reproducing this? 

I did not attempt to reproduce, I was merely reacting to others' comments since they are on the task...

Nick, do you also see comment 5 if you set "not junk"?

Patrick, did you test with clean training.dat ?

Comment 11

[Nick Howitt]

This is my result for tests on comment 5 - with simple rule Mark not as Junk, the mail moved to my Junk folder and stayed marked as Junk. To me this means that TB's own Junk Filter has detected junk and the rule has failed to override it. Running the filter manually from Tools, Run Filters on Folder works correctly.
Incidentally while doing this testing, I cannot get any filters to run by highlighting them then using the Run Now button on the Message Filters dialogue.

Comment 12

[David :Bienvenu]

mark as non-junk should prevent the built-in junk filter from running at all on that message.

Comment 13

[Patrick Brunschwig]

David you're right that's what should happen. But as I wrote in comment 2, this is exactly what is not happening.

Comment 14

[David :Bienvenu]

I've reproduced the problem Patrick described - marking the message as non-junk doesn't prevent the junk filter from running. This was with a POP3 filter - haven't tried IMAP.

Comment 15

[David :Bienvenu]

patrick, are you using pop3, and do you have the option checked to quarantine incoming messages checked?

Comment 16

[David :Bienvenu]

Attachment: proposed fix

The issue here was that we were setting the junk score based on the msg key, and that key was 0, the offset into the quarantine file. Most other ops were using the msg hdr...

So I changed the code to diddle the message offset, not the message key, so that the filter code would work on the right msg key, and the body search code would work, since the body search code was already using the message offset. And I tweaked the msg hdr code to return the message offset even for local folders, if it's set.

Comment 17

[Patrick Brunschwig]

(In reply to comment #15)
> patrick, are you using pop3, and do you have the option checked to quarantine
> incoming messages checked?

Yes, I'm useing POP3, and yes the option to quarantine incoming messages is active.

Comment 18

[Nick Howitt]

Maybe I should not be asking this here, but does the patch in comment 16 resolve the bug in comment 5 only or does it also apply to the original bug?

Comment 19

[David :Bienvenu]

Comment on attachment 255293 [details] [diff] [review]
proposed fix

I slept (poorly) on this and decided it was a really bad idea at this point in the release cycle. I'm going to come up with a safer fix...

Comment 20

[David :Bienvenu]

Attachment: proposed fix

go through the header to set the junk properties, which is always correct, while the msgKey is sometimes wrong.  This means we won't do some notifications that the nsMsgDatabase::SetStringProperty method does, but I think that's what we want in the case of filters setting junk status. The notifications are more for the case where the junk mail plugin has finished classifying a message.

Comment 21

[David :Bienvenu]

this fixes both setting messages as junk and non junk via filters on pop3 accounts, when message quarantining is turned on.

Comment 22

[Patrick Brunschwig]

Retested with TB version 2 beta 2 (20070218) -- works correctly in both cases (setting junk and not junk).

Comment 23

[Wayne Mery (:wsmwk)]

does this patch fix bug 351811 or bug 232104?

Comment 24

[David :Bienvenu]

Wayne, I don't know if either of those are fixed by this change, but I doubt it.

Comment 25

[Scott MacGregor]

flag cleanup