201244 - Certain filters work in manual mode but fail on incoming mail

Status: RESOLVED WORKSFORME

(MailNews Core :: Filters, defect)

Description

[David A. Cobb]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030406
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030406

The attached snippet from my msgFilterRules.dat file identifies four out of some
200 filters that have some traits in common.

1) All rely on the Custom header "List-ID";
2) All originally worked;
3) All FAIL on incoming mail and leave the unfiltered msg in the Inbox; /but/
4) All work just fine if I open the Filters dialog, select them, and do "Run Now".

Regarding point 1, there are another ~100 filters using this header.
One of them began failing after I did some renaming of folders, others seemed to
  start without stimulus from me.
Once a filter is broken it is 'forever' broken.
Even recreating the filter doesn't help -- suggesting the brokeness is a
characteristic of the target mailbox.

Reproducible: Always

Steps to Reproduce:




SEE ATTACHMENT

Comment 1

[David A. Cobb]

Attachment: Snippet from msgFilterRules.dat

These were not consecutive in the original

Comment 2

[George Karabin]

I can confirm seeing this bug on Mozilla 1.3 and various nightlies leading up to
its release on my Windows XP machine. I filter based on a variety of custom
headers, some of which always work, and some of which always fail.

I do see the problem for "List-Id" (note the capitalization difference in "Id").
I also see it for headers "Reply-To" and "Return-Path". I don't see the problem
for the custom headers "X-BeenThere" and "X-MirroredBy".

As the original poster notes, all of these custom header filters have worked for
me on older builds.

Also, for what it's worth, I think that bugs #201244 and #184490 are likely to
be duplicates of each other. #20128 may be as well, but it's not clear if the
filter was a custom header or not.

For a guess, is mozilla possibly ignoring custom headers that don't have "X-" as
a prefix? I know that's a fairly common convention for a lot of custom headers
that I see in free mailing lists. Some of the proprietary, or less mainstream
free list servers I see don't use the "X-" prefix, and those are the ones I'm
having problems with.

Comment 3

[Mike Cowperthwaite]

(In reply to comment #2)
> Also, for what it's worth, I think that bugs #201244 and #184490 are likely to
> be duplicates of each other.

As noted in bug 184490, that problem is IMAP-related.  David and George, please 
do comment here whether your mail accounts are IMAP -- if so, David, please mark 
this bug as a duplicate of that one.

> #20128 may be as well, but it's not clear if the
> filter was a custom header or not.

I don't know which bug you meant, George; that number is clearly a typo.

Comment 4

[David A. Cobb]

NOT IMAP HERE.  All the peculiar filters are on a POP3 Inbox.

Comment 5

[Alex Robinson]

WinXP and Win98 Moz 1.7 latest version on Aug 31, 2005. Filters apply to a POP3
account. 50+ filters specs, all but the last are "move to
localfolders/some_folder" sort of filters. The last is a "contains x", or
"doesn't contain x" filter to move all unfiltered email to localfolders/Inbox.

E.g. filter: 

name="NYTimes"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/NYTimes"
condition="OR (\"X-tagged-for-folder\",contains,NYTimes)"

Today, I modified all of my msg filters from complex, specific logic to simply
checking a "X-tagged-for-folder:" header for a single word. The single word is,
in practice, the name of the folder to which the email is to be moved. The
header has put in the email by a python script run from procmail. The python
script executes the old Mozilla "complex, specific logic" and tags the emails
with the appropriate folder name.

Anyway, the final filter always works. That is, the email goes to the
localfolder/inbox.

If I manually move the mail back to the account's Inbox (from
localfolders/inbox) and run the filters manually, then the mail is moved to the
proper folders. And, if I manually run a specific filter that applies to one or
more emails, then they will be moved properly. And, if I manually run two,
specifically selected filters on emails that should be moved by one or the other
of the filters, the email is moved properly. BUT, if I manually run two selected
filters, the first of which won't do anything to the email, and the SECOND of
which applies to the emails, the emails are unaffected.
So, it's a very weird bug. Wish I had a debug version and knowledge of where to
set breakpoints. It would be an easy bug to find, if so.
I've not tested the same setup under Tbird, yet. But am trying to set up Tbird
in the hopes the bug won't be there.

Comment 6

[Wayne Mery (:wsmwk)]

Alex, any results to report for Thunderbird?

Comment 7

[Alex Robinson]

(In reply to comment #6)
> Alex, any results to report for Thunderbird?
> 

XP, Thunderbird 1.5. Bug still present.

That is, by hand, I moved a single email from a Local Folders folder named "!Spam_Delete" to the filters' account's Inbox.

The first two filters in the rules check whether the header, "X-tagged-for-folder" is "!Spam" or "!Spam_Delete", respectively. The filters move the email to the Local Folders folder of that name. Here is a copy of the top of the msgFilterRules.dat file:

version="8"
logging="no"
name="Spam"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/%21Spam"
condition="OR (\"X-tagged-for-folder\",is,!Spam)"
name="Spam_Delete"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/%21Spam_Delete"
condition="OR (\"X-tagged-for-folder\",is,!Spam_Delete)"

The email had the header, "X-tagged-for-folder: !Spam_Delete" and had been properly routed when it came in.

I selected these first two rules and ran them on the Inbox. Nothing happened. Then I selected the second rule and ran it. The email was moved to the proper folder.

Comment 8

[Mike Cowperthwaite]

(In reply to comment #7)
> The email had the header, "X-tagged-for-folder: !Spam_Delete" and had been
> properly routed when it came in.

"Properly routed" by the filters?  That contradicts this bug's summary, which states that filters fail on incoming mail.


> I selected these first two rules and ran them on the Inbox. Nothing happened.
> Then I selected the second rule and ran it. The email was moved to the proper
> folder.

That's bug 180239.

Comment 9

[Alex Robinson]

Roger that, bug 180239. Arrghh. :)

I'll post it on the other bug.

Comment 10

[Mike Cowperthwaite]

OK, back to square 1.  

I suspect this related to the problem in bug 209488 comment 4.  However, I 
have a filter on Content-Type (which is a custom header, as far as filtering 
is concerned) that works consistently for me, and has for years.

Comment 11

[Wayne Mery (:wsmwk)]

view fails also for me per bug 209488 for custom field 
 Sender

and fails for BOTH incoming manual mode filter attempts.

Comment 12

[Mike Cowperthwaite]

David Cobb: per comment 10, bug 209488 has been fixed (TB 2.0/SM 1.1).  Please test with a current build; if it's behaving as expected, please mark this bug as a dupe of that one.

Comment 13

[David A. Cobb]

With 2007-01-25-04-trunk (Linux): Not working.
My actions are (1) copy the message, (2) Tag the message, and (3) Add Star.

Here's the MsgFilterRules.dat stanza for one of them:
name="[e-mail] *"
enabled="yes"
type="1"
action="Copy to folder"
actionValue="mailbox://nobody@Local%20Folders/Archive/s/Secunia%20Security%20Advisory/sec-adv%40secunia.com"
action="AddTag"
actionValue="$label1"
action="Mark flagged"
action="JunkScore"
actionValue="0"
condition="OR (from,is,[e-mail]) OR (from,ends with,<[e-mail]>)"

Question is, I suppose, has the fix from Bug#209488 landed in the nightly build?  If so, it did not fix this symptom.

Comment 14

[David A. Cobb]

Anyway, Bug#209488 refers to "Custom Headers."  As seen in the above example using the 'from:' header, that is not the issue here.
Furthermore (1) this is a POP account, and (2) I have several hundred "Custom Header (List-Id:)" filters for which the action is "Move to folder" and which consistently work as expected.  I conclude that the problem is not in triggering the filter rule, but in performing the specified action.

Comment 15

[David A. Cobb]

Aw POO! "Belay my last!"

I can no longer comment intellegently on Windows bugs -- I haven't done Windows in years.
My current symptoms are described in Bug#355537.

Comment 16

[Mike Cowperthwaite]

OK -- I'm resolving this bug WFM, per comment 15.  David Cobb, feel free to reopen this if the bug originally reported is still a problem.  btw, filter/search bugs are unlikely to be platform-specific.