525022 - Filter based on Sender field fails. Mismatch between expectation by internal filter execution code(\"sender\") and string set by message filter management code(\"Sender\")

Status: RESOLVED WORKSFORME

(MailNews Core :: Filters, defect)

Description

[WBT]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Build Identifier: version 2.0.0.23 (20090812)

I have a mail filter which has several items OR'ed together ("Match any of the following"), and some of those items are "Sender Contains" <email@address>.  That rule often fails to catch messages which clearly match, or at least show "Sender: <email@address>" when I view it in the reader window.  
At first, I thought this only applied to messages with attachments, but I just got another example case with no attachments.  

Reproducible: Sometimes

Steps to Reproduce:
1. Set a mail filter for Sender Contains "[e-mail]"
2. Get an e-mail with that address in the Sender field, but not the From field

Actual Results:  
The mail stays in the inbox.

Expected Results:  
The mail should go to the appropriate folder, as the other parts of the rule do send it there.

Comment 1

[[:Aureliano Buendía]]

(In reply to comment #0)

> 1. Set a mail filter for Sender Contains "[e-mail]"
You fill account without quotes?

Comment 2

[WBT]

Yes, without quotes.  I have many other e-mail filters which seem to work fine. 

I've also just noticed that the "Sender: " displayed when I look at a message differs from the "Sender" in the list of messages (which apparently uses the "From: " field.  However my rule is duplicated to filter based on From or Sender anyway.

The non-filtering might have something to do with the From and Reply-To fields both being set to something (here, the same) which does not match the Sender field.

Comment 3

[Ludovic Hirlimann [:Usul]]

Can you enable the log in the filter dialog and see if it works ?

Comment 4

[Wayne Mery (:wsmwk)]

Today I added sender contains something.com to a filter that has worked for a long time, and the filter appears to be working.  But that's not great proof that WBT doesn't have a problem.

Comment 5

[WBT]

Sorry for the delayed reply - I did turn on the filter log, but the log only shows messages that the filter DID act on, not the ones that it let sit. 
Bug 525023 may lend some insight into this problem - the "Sender" term means different things when used internally vs. the UI for opening a message vs. the UI for looking at a folder or inbox, and maybe the filters don't actually use the mail "sender" field but one of the related fields (e. g. From or Reply-To).

A good test case would be to set a filter based on the Sender field and try a message where the Sender field matches the filter, but the From and/or Reply-To fields do not.  I don't know how to manipulate those headers directly.

Comment 6

[Wayne Mery (:wsmwk)]

WBT can y ou attach a file to this bug which is a testcase message?

Comment 7

[WBT]

I don't know how to set the Sender: field of a message without setting From: or Reply-To:, in order to construct a testcase.  I'm not even completely clear on what "Sender" means.  Any suggestions?

Comment 8

[Kent James (:rkent)]

'I'm not even completely clear on what "Sender" means'

"Sender" internally in the mailnews code is the best determination of who sent the message. It does not mean the "Sender" rs822 header. Code here:

http://mxr.mozilla.org/comm-1.9.2/source/mailnews/local/src/nsParseMailbox.cpp#1286

will set the value of sender using From if it exists, else Sender if it exists, else Envelope-from.

Comment 9

[Wayne Mery (:wsmwk)]

is there an add-on that can help us with the following...

(In reply to comment #5)
> A good test case would be to set a filter based on the Sender field and try a
> message where the Sender field matches the filter, but the From and/or Reply-To
> fields do not.  I don't know how to manipulate those headers directly.

Comment 10

[Alex]

I cannot find 'Sender' condition at all, had it been removed? [SM2.2]

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Kent James (:rkent) from comment #8)
> 'I'm not even completely clear on what "Sender" means'
> 
> "Sender" internally in the mailnews code is the best determination of who
> sent the message. It does not mean the "Sender" rs822 header. Code here:
> 
> http://mxr.mozilla.org/comm-1.9.2/source/mailnews/local/src/nsParseMailbox.
> cpp#1286

WBT,
In that case, we need your testcase message (from view source).
And snippet of the filter rules from msgFilterRules.dat

Comment 12

[WADA:World Anti-bad-Duping Agency]

Issues in filter rule/filter execution around "replacement of label named Sender by From" was resolved by Bug 404489, but problem on customized "Sender" seems to still remain.

Following is filter rules created by Tb 7, for "If ??? contains abc, Add tag of Important".
(1) ??? = From (standard one, defined since initial, for From:, Sender: etc.)
  name="From(standard) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (from,contains,abc)"
(2) ??? = Sender (added by customize, for Sender: header only)
  name="Sender(customized) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (\"Sender\",contains,abc)"
(3) ??? = sender (added by customize, for Sender: header only)
  name="sender(customized) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (\"Sender\",contains,abc)"
(4) ??? = From (added by customize, for From: header only)
  name="From(customized) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (\"from\",contains,abc)"
(5) ??? = from (added by customize, for From: header only)
  name="from(customized) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (\"from\",contains,abc)"
(6) ??? = from (added by customize, for From: header only)
  name="fROM(customized) contains: abc"
  action="AddTag"
  actionValue="$label1"
  condition="AND (\"from\",contains,abc)"

Adding customized header looks case-sensitive, but internal header name looks lower case always, or lowercase with capital of first letter in some cases.
And, if customized one, Tb looks to enclose it by double quotes, with escaping " by \, in msgFilterRules.dat.

Above condition=".. (\"Sender\",contains,abc)" didn't fire on mail with "Sender: <abc@x.y.z>". So, I changed it to \"sender\" because \"from\" is used for customized From which is for From: header only.
(1) Terminate Tb
(2) Edit msgFilterRules.dat, change \"Sender\" to \"sender\", and save.
    Because only next rule is contained in message filter, it's very easy.
      action="Move to folder"
      actionValue="mailbox://nobody@Local%20Folders/Inbox2"
      condition="AND (\"sender\",contains,abc)"
(3) Restart Tb, Never open message filter,
    Run filter on Inbox => the mail was move to Inbox2.
(4) If any filter rule is touched, manual change to \"sender\" looks lost.
    Even if message filter is not touched, \"sender\" looks lost after
    termination of Tb.

Problem is mismatch between expectation by internal filter execution code(\"sender\") and string set by message filter management code(\"Sender\").
Confirming.

FYI.
Date version of this bug is bug 378449, and is already resolved.
Status version of this bug is bug 435694. I don't know whether that bug still occurs in Tb 7 or not.

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI.
Following is filter rules created by Tb 7 for Status.
(7) Status(standard one, defined since initial, for internal status of mail)
  name="Status(standard) is: Forwarded"
  action="Move to folder"
  actionValue="mailbox://nobody@Local%20Folders/Inbox2"
  condition="AND (status,is,forwarded)"
(8) Status(customized, Status: header only)
  name="Status(customized) contains: xyz"
  action="Move to folder"
  actionValue="mailbox://nobody@Local%20Folders/Inbox2"
  condition="AND (\"Status\",contains,xyz)"
And this \"Status\" worked well on Status: header(i.e. bug 435694 is already WORKSFORME in Tb 7.0.)

I believe "all lower case" or "first is captial" is better consistent among all customized headers for message headers defined by RFCes, in order to avoid future confusions on Tb's design/implementation and program code.

Comment 14

[WADA:World Anti-bad-Duping Agency]

As bug 622307 comment #21 is posted, I checked again with Tb 8.0 on MS Win. 
Condition in msgFilterRules.dat was still \"Status\", and it worked.
  condition="AND (\"Status\",contains,xyz)"

Closing as WORKSFORME, per bug 622307 comment #21 and above test result.

Comment 15

[WADA:World Anti-bad-Duping Agency]

Correction.
Condition in msgFilterRules.dat was still \"Sender\", and it worked as expected.