525884 - Message Filters run out of order or continue to next filter after matching (IMAP ?)

Status: RESOLVED WORKSFORME

(MailNews Core :: Filters, defect)

Description

[Jack Eidsness]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20091101 Minefield/3.7a1pre (.NET CLR 3.5.30729)
Build Identifier: Many versions, including 2.x, 3.0.x and 3.1x

I have Message Filters on one of my IMAP accounts (an exchange server, tragically) that have overlapping match conditions, so I would expect the filter order to decide what happens.  All affected filters result in a different "Move to folder" action.  It is unpredictable which filter will ultimately prevail, regardless of the order.


Reproducible: Sometimes

Steps to Reproduce:
0. Get an IMAP email account. Join an email list (we'll call it some_list).  have automations that send mail to that list that you can identify for some specific purposes by their subject line.

1. Set up Filters:

Filter 1 (Conditions ANDed) --
 To or Cc: Some_list
  Subject starts with: foo
Move message to folder "foo" 

Filter 2 (Conditions ANDed) --
 To or Cc: Some_list
  Subject starts with: bar
Move message to folder "bar" 

Filter 3) To or Cc: Some_list
Move message to folder "some_list" 

Filters 1 and 2 are intended to represent something more specific than just the list membership, and filter 3 is intended to catch anything sent to the list for any reason.

2. Receive mail

Actual Results:  
Sometimes specific "foo" and "bar" messages to some_list go in the "foo" or "bar" folders respectively.  Usually, though, they go in "some_list" folder.  Sadly I have not been able to identify a more specific condition for "usually" but I'm sure there's some kind of reason.  I just have no idea what it is.

Expected Results:  
At this point I don't even care what the filters do as long as it did it the same way every time.

I had used filters like these for several years on releases and nightlies (I have switched back & forth quite a few times) without incident.  It's been probably between 1 and 2 years that this has not been working right for me.  I figured it was just a problem with the nightlies at first, but when I switched back to 2.x, it continued.  I even deleted my msgFilterRules.dat, recreated filters, and tested, and created a new profile and recreated the filters, all without solving this problem.

I tried changing the order.  The problem has been that they don't run in a consistent order, or the filtering doesn't stop after the first match. I can tell it's  have filters where one matches a subset of what the other would match, e.g.

Sometimes I'll see both folder names in the folder list (one foo or bar, and some_list) get the bold text as if they have new messages, but then only one or the other of them gets to keep the message.

One of the times I was pleasantly surprised to see mail go into the correct folder was during the first email checking after starting thunderbird.  I thought someone might have fixed the bug, but then later that day during a normal periodic mail checking, it put more of the foo/bar mail in the some_list folder.

I doubt I would have changed my filters at all in about 5 years if they hadn't stopped working consistently, which seemed to me to be out of the blue at the time (How I wish I'd marked the calendar and filed a bug, then). 

I would have filed a bug a long time ago, but usually when something is this bad, I search and I don't find a match, but then I always get marked a duplicate anyway.  Also, I was hoping I'd have a better way to reproduce the bug for you, but this is as good as I expect to get any time soon.

Comment 1

[Kent James (:rkent)]

There is certainly logic in the filter list processing that is designed to keep this from happening (the "applyMore" return from ApplyFiltersToHdr). It would be great if someone from QC could confirm conditions that can reproduce this.

I'm wondering if somehow you aren't getting multiple connections to the Exchange server. I could imagine this happening if, for example, you had two separate accounts set up, each of which was independently checking the server. One way that you might test for that is to try this with a brand new profile and see if that solves the problem.

Comment 2

[David :Bienvenu]

Attachment: simple fix

Not sure how the saved search got the offline flag set (the front end might have set it), but in any case, we should ignore saved searches here.

Comment 3

[David :Bienvenu]

ugh, sorry, right patch, wrong bug - this was supposed to be bug 525842 - I'll move it there, and land it on the trunk...

Comment 4

[Kent James (:rkent)]

(In reply to comment #3)
> ugh, sorry, right patch, wrong bug - this was supposed to be bug 525842 - I'll
> move it there, and land it on the trunk...

... and yet your patch made me think that if we tried to apply filters to some sort of virtual folder, we would also see this current bug.

Comment 5

[David :Bienvenu]

I'm not sure how that would happen - virtual folders don't have any message headers in them.

Comment 6

[Jack Eidsness]

Well, I'm sure it has multiple *TCP connections* because that setting ("Maximum number of server connections to cache") is set to 3.  I don't think that's what you had in mind, but just for grins, I'm going to set it to 1 for a week and see how that goes.

Comment 7

[David :Bienvenu]

We should only ever have one connection open to any given imap folder at a time. If that breaks, then, yes you could get multiple filters running at the same time.

Comment 8

[Jack Eidsness]

I should give a more direct answer to your question -  Short version: I'm sure I'm running only one thunderbird and no other programs are accessing this exchange IMAP account, and that on the thunderbird profile, I have only the one account configured for this server.  I do have another IMAP account on some other server, though.

Long version:  I have three machines I have ever configured to connect to this exchange server via IMAP (all using thunderbird) - one of them is a laptop that's off 99% of the time, one of them is a PC I rarely see physically (usually RDC), and I have confirmed that thunderbird has not been running there for weeks.  All of them were in use before my problem ever manifested. I 
don't like to have thunderbird accessing this account from two places simultaneously, and to the extent that it ever may have happened, well, I'm sure it was not happening at least 98% of the times that I have had this "wrong folder" issue.  I am positive, as a frequent user of process monitor, that I don't have multiple programs (or copies/versions of thunderbird) on my primary PC that would be accessing the account simultaneously.

If it matters, well, sometimes I do move mail from the exchange server to my other IMAP account (because the other one is much much faster), and I have an identity configured there so I can do replies, but I don't see any reason that would affect the loading and filtering of messages.

I have Google Desktop search indexing mail on this account.  If you're not familiar with it, it's works with thunderbird as a plugin, but with TCPview I can see there are no additional connections to the server.  I guess it has some potential to be a source of the problem, anyway, so I'll turn the plugin off for a while and see.

Comment 9

[Jack Eidsness]

I just got some of my "foo" and "bar" mails for the first time since going down to 1 cached connection, and uninstalling the google desktop search thunderbird plugin.
I doublechecked my filters as well, that they are in the original, desired configuration.
The mail still went to the wrong folder ("some_list").

I hope someone else out there can reproduce the problem.

Comment 10

[Jack Eidsness]

I re-did a bunch of my filters a few months ago, without any expectation of relevance to this bug.  They're hardly any different from what they were before, other than that there are slightly fewer of them.  I also started syncing mail locally.  I didn't even notice until today, but I stopped having this problem somewhere along the line.  I've been running shredder all along, so it could have been anything.  Maybe it's fixed by someone, maybe I just stopped doing whatever it is that caused the breakdown.  Maybe I just made a mistake when I wrote the filters, but I thought I looked at them dozens of times.

Anyway I'm glad to be rid of it.  If I can't even reproduce the bug myself, anymore, I don't see how I could ever expect someone to work on it.

If it ever happens again, i'll install an add-on to "bounce" mail to myself over and over and give it a more thorough examination.