527430 - Thunderbird should honor server marking message as NonJunk

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Alessandro Vesely]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 GTB5 (.NET CLR 3.5.30729)
Build Identifier: version 2.0.0.23 (20090812)

Trusted servers should be able to tag new messages as NonJunk, thereby indicating that a message needs no anti-spam scanning. Certified mail, vouched servers, PEC (Italian legal equivalent of registered letters, via SMTP) are examples where this ability is needed.

Setting IMAP NonJunk keyword on delivery doesn't apparently affect TB's junk filtering.

Reproducible: Always

Steps to Reproduce:
1. Mark as NonJunk some (or all) messages delivered to a TB account with adaptive junk mail controls enabled. On a Courier-IMAP/Maildrop server, marking may be accomplished by

   if ($CONDITION)
   {
      KEYWORDS="NonJunk"
      to "./Maildir/.WhateverFolder"
   }

2. Wait until some junk arrives
Actual Results:  
Message moved to the Junk folder.


Expected Results:  
Message left alone.

Rather than "fixing" that attribute handling, it would be preferable to devise a reliable and documented way to achieve the desired result, independently of any of the IMAP/POP3 type of client, actual IMAP folder, and additional anti-spam filters.

This behavior might be configured by letting users fill an initially empty list of domain(s) whose Authentication-Results header field they trust, as far as the method "x-spam-filter" is concerned. Such method would be an extension of RFC 5451, with possible codes "passed", "failed", and "exempt": the first two values should result in SpamAssassin's header fields (that can already be managed by TB) while the last one should prevent moving the message to the Junk folder. See bug 189970, comment 34.

Comment 1

[Kent James (:rkent)]

TB's junk processing is supposed to already work the way that you have described. What evidence do you have that it is not working that way? If you can demonstrate this, please also attach an IMAP protocol log for us to examine.

Comment 2

[Alessandro Vesely]

Attachment: Extract from log file obtained with NSPR_LOG_MODULES=IMAP:4

Hm... this log snippet does not show NonJunk keyword for the specific message. I'll have to try again by setting an additional keyword as well.

Comment 3

[Kent James (:rkent)]

When I run a protocol log on startup of SM 2, I see lines like this:
======

424[3aa6740]: 4136800:caspia.com:S-INBOX:SendData: 8 UID fetch 1:* (FLAGS)

424[3aa6740]: ReadNextLine [stream=41901a8 nb=44 needmore=0]
424[3aa6740]: 4136800:caspia.com:S-INBOX:CreateNewLineFromSocket: * 1 FETCH (UID 2356 FLAGS (\Seen NonJunk))

424[3aa6740]: ReadNextLine [stream=41901a8 nb=41 needmore=0]
424[3aa6740]: 4136800:caspia.com:S-INBOX:CreateNewLineFromSocket: * 2 FETCH (UID 2357 FLAGS (\Seen Junk))

======
which clearly show the Junk and NonJunk flags on different messages, and TB/SM use to set the junk status in an IMAP message if there is not an existing junk status.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)
> Created an attachment (id=411449) [details]
> Extract from log file obtained with NSPR_LOG_MODULES=IMAP:4
> 
> Hm... this log snippet does not show NonJunk keyword for the specific message.
> I'll have to try again by setting an additional keyword as well.

Make sure your imap server is configure correctly too ....

Comment 5

[Alessandro Vesely]

(In reply to comment #4)
> Make sure your imap server is configure correctly too ....

Yes, I had messed it up: the code I added for building the previous log was not reached, thus NonJunk wasn't being set. My apologies for that.

After fixing it, today I got

2428[29c8638]: ReadNextLine [stream=295f458 nb=48 needmore=0]
2428[29c8638]: 2b8ca00:wmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 1 FETCH (UID 244429 FLAGS (\Answered \Seen))

...

2428[29c8638]: ReadNextLine [stream=295f458 nb=76 needmore=0]
2428[29c8638]: 2b8ca00:wmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 1747 FETCH (UID 275831 FLAGS (\Recent $label2 NonJunk different_domain))

Today, no messages have been moved to Junk. However, I think I had set "NonJunk" correctly when I filed this. Now I'll remove the "$label2" and "different_domain" extra keywords so as to reproduce the same conditions I had at that time, and see what I'll get tomorrow. (I have to run this test on a busier folder to be sure I get some spam every day.)

Comment 6

[Kent James (:rkent)]

If you load my JunQuilla extension, and enable the junk status + column, then you can see when a message has been marked NonJunk (as opposed to unknown), and why. When the NonJunk status comes from the IMAP server, then the Junk Status + column shows a green circle with a little flag in in.

Comment 7

[Alessandro Vesely]

(In reply to comment #6)
> If you load my JunQuilla extension [...]

Yeah, I'll have to. Also for that folder-setting mentioned in bug 189970, comment 35. I still have TB 2, though.

Today again a found no messages added in the junk folder. However, there's something strange in the log, e.g.

2324[285c690]: ReadNextLine [stream=27e03d0 nb=53 needmore=0]
2324[285c690]: 2bbc888:wmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 648 FETCH (UID 255170 FLAGS (\Seen Junk NonJunk))

Next, I opened my account options and found out that the box "Move new junk messages to:" was unchecked. How come?!? I haven't tinkered with TB options for months... At any rate, I'll keep trying to get a clean log for some more days.

One thought: I don't think many servers indiscriminately set "NonJunk" on all messages. (I only do it for this test, and it's quite annoying ;-) However, from a theoretical POV, what if my mailbox provider had unacceptable criteria for setting "NonJunk"? Does JunQuilla provide a way for users to say whether they trust their providers? (It would pair with how Authentication-Results requires an explicit user's consent.)

Comment 8

[Alessandro Vesely]

Attachment: Extract from log file obtained with NSPR_LOG_MODULES=IMAP:4, and trying to get similar results manually

It happened again, now I was logging, thus I can update the recipe for reproducing this bug as follows:

1) I've sent a message to a folder where the server sets "NonJunk". Verified the attribute is there.

2) Start TB and navigate to that folder. TB issues a command "UID fetch 1:* (FLAGS)" and the "NonJunk" keyword is not displayed for "\Recent" messages --see attached log. At this time the message is still flagged "NonJunk" on the server, but it's not "\Recent" any more.

3) Send another message just like the previous one and manually login to make sure that the keyword is reported by the server. Actually, I did this twice, including sending a third message, because I issued "fetch" rather than "UID fetch" on the first time. The server does report "NonJunk" even on "\Recent".

It seems to me that TB doesn't see the keywords correctly. I don't know why I haven't been able to reproduce this behavior on the "INBOX" folder. The folder "INBOX.Abuse" where this happens is where I saw the bug originally.

Comment 9

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 10

[Wayne Mery (:wsmwk)]

Alessandro do you still see this issue?

Comment 11

[Alessandro Vesely]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #10)
> Alessandro do you still see this issue?

I think I don't.  I just sent a spam message (taken from Junk folder) and sent it to my helpdesk folder, where NonJunk is set by the server, and the message was not moved to the Junk folder by TB.

I'm happy the way it works now.

Thank you for taking care.

Ale

Comment 12

[Wayne Mery (:wsmwk)]

Thanks for the update