457168 - These emails repeatedly match the wrong filter

Status: RESOLVED FIXED

(MailNews Core :: Filters, defect)

Description

[Nelson Bolyard (seldom reads bugmail)]

Attachment: zip file described in comment 0

I run SM trunk.  My nightly build is less than a week old.

For my main POP mail account, I have 57 folders, including 6 search folders,
and 133 filter rules.  One of my filter rules is intended to find email
addressed to a certain mailing list of which I am a subscriber, and to file 
the emails for that list into a folder designated for it.  That rule is:

name="To [e-mail]"
enabled="yes"
type="17"
action="Move to folder"
actionValue="mailbox://[e-mail]/Professional/nssdev"
condition="OR (to or cc,contains,nssdev)"


NONE of the emails I receive daily from bugzilla-daemon can ever match the 
matching rule for that mailing list.  Yet, daily, numerous emails from 
bugzilla-daemon DO match it.  

In the past, when there were misfiltered email messages, I found that by 
moving them back to the Inbox, and then applying the filters to the inbox
again, they would often get filtered correctly.  

But now I have a set of messages, all from the bugzilla-daemon that 
repeatedly 100% every time match this filter that they should never match.

This is so exasperating.  I have put those 3 messages into a folder. 
In the attached zip file, I have
- the folder with those repeatedly mismatched messages
- my file of filter rules
- lines from the filter log, showing the bad match.

It may or may not be relevant that I have my folders nested in subfolders,
and the very last top level folder contains subfolders named
Drafts, Inbox, Sent, Templates
Maybe it blows mailnews's little mind that there are multiple folders 
with the same names.  However, none of these folders is either 
- the intended folder for the message, or
- the folder into which the messages are misfiled.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

In the mail folder (named "misfiltered") in the attached zip file,  
are three messages.  None of them contains the string "nssdev" anywhere
in them.  That string does not appear in the file.  Yet the filter
shown above matches on these messages, EVERY TIME.  If I move those
message back to the inbox, and rerun the filters on the inbox, they
match the nssdev filter again, and get put into the nssdev folder again,
every time, over and over.  

If I could download the mailnews source files I need to debug this, 
without having to download all of comm-central (as is presently necessary), 
I would do it in a heartbeat.

Comment 2

[Kent James (:rkent)]

I downloaded this file, and tried the filters. These filters hit the three files:

From [e-mail] -> bug reports
To/From/Re mozilla -> mozilla
Bug w/CC: [e-mail] -> bug reports

which I believe is correct. This is on a TB debug trunk build running Windows Server 2003 64 bit, approximately one week old. The folder copies fail of course, because I did not recreate your folder structure.

So I am unable to reproduce this. If I could, I would debug it. Sorry.

Comment 3

[Mark Banner (:standard8)]

(In reply to comment #1)
> If I could download the mailnews source files I need to debug this, 
> without having to download all of comm-central (as is presently necessary), 
> I would do it in a heartbeat.

Nelson, within the next few days we will be doing the beta 1 builds that will be used for testing - as part of that we'll be generating a source bundle of the code. Would you be willing to use that?

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

In reply to comment 2, In my case, the false matches occur on a rule 
that is much earlier in the list of rules, so it never gets to the 
correct rule, which is the one you were hitting, which comes much later
is the list of rules.

In reply to comment 3, Mark, realistically, is there any alternative?

Comment 5

[Mark Banner (:standard8)]

(In reply to comment #4)
> In reply to comment 3, Mark, realistically, is there any alternative?

hg has been investigating being able not to download all history, though I don't know how soon they are likely to get that built in. The only other possibility is nightly source bundles, but from your question I guess that's not what you want.

Comment 6

[David :Bienvenu]

Why not debug the CVS version of the source, which I believe you already have? It's unlikely that the relevant code has changed.  There's something about your profile that's causing this, I believe, but I have no idea what it is.

Comment 7

[David :Bienvenu]

Can you try this test? Move one or more of the problem messages back to your Inbox, and then do an advanced search, (in TB, it's edit | find | search messages, cntrl Shift F), and search for to or cc contains nssdev and see if those messages appear in the search hits?

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

David,
I did the test you requested in comment 7.
Yes. all 3 messages show up in that search.

I'm very surprised to read (in comment 6) that you think it is unlikely 
that the relevant code has changed, but if that is correct, then I 
certainly can pull the sources with CVS, with pleasure! 

You once suggested the names of some functions in which to start debugging,
and the names of some directories whose source files are likely to be most
relevant.  But I cannot find that message any more.  Would you provide that
info again, please?

Comment 9

[David :Bienvenu]

The messages are showing up correctly in the thread pane, right? This is very strange. One thing to try is to shutdown, move away/save your Inbox.msf file, restart, wait for the index to rebuild, and then do the same search.

Since search is showing the same issue, it's probably easiest just to debug the search code, after I hear the results of the previous test.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Yes, I believe the messages are showing up correctly in the thread pane.

I made backup copies of Inbox and Inbox.msf, and the removed Inbox.msf 
and restarted.  The next attempt to apply filters to folder appears to 
have behaved correctly.  

I wouldn't call this a cure, but rather symptomatic relief.  
Do you want the .msf file?  Would it help?

Comment 11

[David :Bienvenu]

Yes, I'll look at the .msf file, if you e-mail it to me.

Comment 12

[David :Bienvenu]

In the .msf file, the msg hdrs in question think that the cc header is set to an address containing nssdev, even though there is no cc header in the mail message. We accumulate the cc header in the new mail message parser, as we parse the headers, and we clear it between every message, so I'm confused as to how that value could have been set on the header. In this particular case, we think the cc list is nssdev, and dveditz. Presumably, we inherited it from some previous message that arrived in the same session. Or perhaps the issue is that the we created a db hdr, populated it, and then discovered that we needed to move the message to an other folder, because of a filter, and then truncated the inbox, and created a new msg hdr, with the same offset into the inbox, because the previous header is still somehow in memory...

Comment 13

[David :Bienvenu]

One possibility is that this is an interaction with the duplicate handling pref you've set. Are you moving dups to trash, or just deleting them outright? I believe 2 is the value for moving to trash, and 1 is the value for deleting them outright. Ignoring the fact that that pref doesn't work the way you'd like it to work, what do you have it set to? From looking at the code, I think it could be an issue if you have it set to move to the trash. The next arriving message might inherit the cc list of the duplicate message.

Comment 14

[David :Bienvenu]

Attachment: proposed fix

the problem was that the remove dup code move to trash code wasn't clearing the msg hdr for the key/offset into the mailbox, and since we truncate the inbox on filter moves, the next message had the same key/offset, and inherited the same properties. So we have to do what we do for normal filter moves, and remove the mdb row.

It's ok to just do this for the non quarantine case, !m_downloadingToTempFile, because the remove dup code doesn't work for the quarantine case :-( If there's no bug, I'll file one for it.

I simulated getting a dup with a cc list, followed by a message w/o a cc list, in the debugger, and recreated the bug, and verified that this fixes it.

Comment 15

[Nelson Bolyard (seldom reads bugmail)]

mail.server.default.dup_action is 2

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

I hope that the fix for this bug will also fix bug 389006.

Comment 17

[David :Bienvenu]

so that would be consistent with my theory...

Comment 18

[David :Bienvenu]

yes, they're one and the same, I believe.

Comment 19

[Mark Banner (:standard8)]

Attachment: Testcase

I got bored one evening at the weekend and decided to write a testcase for this (it expands on our filtering cases a little at least).

Comment 20

[Mark Banner (:standard8)]

a=me for landing this for TB 3 beta 1. As this is a hidden pref, the risk should be low, and with the testcase as well we can be confident its a good fix.

Comment 21

[David :Bienvenu]

Comment on attachment 340911 [details] [diff] [review]
Testcase

+    incomingServer.closeCachedConnections();

pop3 doesn't cache connections, so I think this is a noop.

Comment 22

[David :Bienvenu]

fix checked in - changeset:   460:4e502622d8b1

Comment 24

[Mark Banner (:standard8)]

(In reply to comment #21)
> (From update of attachment 340911 [details] [diff] [review])
> +    incomingServer.closeCachedConnections();
> 
> pop3 doesn't cache connections, so I think this is a noop.

Although its a noop for pop3, I'd like to keep it in for consistency, then if anyone copies and pastes for imap/news, they won't miss it.

Comment 25

[Mark Banner (:standard8)]

I've checked in the test case as well, changeset id 461:f95b15892209