719037 - Most mail filtered into wrong folder (into the folder of the filter that matched the first message)

Status: RESOLVED FIXED

(MailNews Core :: Filters, defect)

Description

[Sune Mølgaard]

Attachment: msgFilterRules.dat

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120118 Firefox/12.0a1 SeaMonkey/2.9a1
Build ID: 20120118003001

Steps to reproduce:

Set up a number of filters to move mail to different filters.


Actual results:

Mail that shouldn't get filtered (i.e. that should appear in the Inbox) ends up in the folder that my last filter is set to move mails to.


Expected results:

Filters should apply only to the mail criteria that the filters are actually set to hit...

Comment 1

[:aceman]

Can you also attach a sample message (export as EML and you can also remove the body text) that gets filtered wrongly?

Comment 2

[Sune Mølgaard]

Attachment: Email that got filtered to the linux-kernel folder

Comment 3

[David :Bienvenu]

Is it the case that when a message triggers a filter, all the other messages trigger that same filter? What kind of filter is it, e.g., Subject contains?

Comment 4

[David :Bienvenu]

and are you using pop3 or imap?

Comment 5

[Sune Mølgaard]

That's very possible, and since the linux-kernel list is high volume, it would explain that most are just filtered to there.

Filter is Sender Contains or From, To, CC or Bcc contains.

Also, protocol is pop3

Comment 6

[:aceman]

Did this start to happen at any particular date?

Comment 7

[Sune Mølgaard]

Hard to say, since pop3 and filters has presented a slew of problems as of late (most of January). Thus, some mails may have have ended up in the linux-kernel folder due to other problems...

It would seem to have happened on a large scale since at least the 12th, but possibly before that.

I don't recall any problems (of this nature) when trunk was on 2.8a1, though...

Comment 8

[David :Bienvenu]

I couldn't reproduce this with a "From" filter, but I think I reproduced it with a Cc filter, in that the cc filter fired on messages it shouldn't have.

Comment 9

[Sune Mølgaard]

IIRC, the "Sender" field is one that one needs to define oneself as a custom field. I don't know why that is, and neither do I know if it's still the case, but I do seem to remember defining it as a possible filter trigger a long time ago, but I have no idea where it is stored, as it doesn't seem to be in msgFilterRules.dat...

It's good to know, though, that you see it in a CC filter, as it should be a reasonable guess that it could be the same problem. It's also possible, though, that it's the CC part of "From, To, CC or Bcc" part that kicks in, and in my case, almost all filters employ that trigger...

Comment 10

[David :Bienvenu]

OK, I believe the issue is that earlier messages pollute the state of later messages, if messages are moved by the folder. The first message has a msgHdr object created for it, storing the headers of the message. If that message gets moved by a filter, the next message gets created with the same key, and doesn't get its headers cleared out, so if the first message had a cc header, and the second one did not, the hdr for the second message will have the first message's cc header, and trigger the same filter. If the second message had a different cc header, it would overwrite the earlier cc header, and the filter wouldn't fire.

I have a potential fix, but my pop3 server is really flaky right now, which makes verifying the fix tricky.

Comment 11

[Sune Mølgaard]

Due to just having moved, I don't have my stationary machine handy, but I should be able, although at a much slower rate, compile and test a Seamonkey build with your patch. I follow a number of MLs, so the bug should present itself rather quickly, if it hasn't been fixed by your patch.

Starting DL'ing source now...

Comment 12

[Sune Mølgaard]

I have to head for the bed soon, as I'm on UTC+1, but do attach any patches that you come up with, and I'll build and test tomorrow.

Comment 13

[:aceman]

David, is this some known bug? I have never heard about this while weeding through bugzilla.

Comment 14

[David :Bienvenu]

(In reply to :aceman from comment #13)
> David, is this some known bug? I have never heard about this while weeding
> through bugzilla.

I first heard of it this morning, and it's a regression from the pluggable store landing (to be precise, this regression was hidden by the truncation issue, but fixing that exposed this issue).

Comment 15

[David :Bienvenu]

Attachment: proposed fix with unit test

this should fix it.

Comment 16

[David :Bienvenu]

Attachment: fix line ending in data file

imap fake server is picky about line endings

Comment 17

[Sune Mølgaard]

Compilation fails with:

/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o): relocation R_X86_64_32S against `vtable for std::bad_function_call' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[7]: *** [libnptest.so] Error 1

Dunno if this is because of my setup or because of the patch...

Comment 18

[David :Bienvenu]

(In reply to Sune Mølgaard from comment #17)
> Compilation fails with:
> 
> /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o):
> relocation R_X86_64_32S against `vtable for std::bad_function_call' can not
> be used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad
> value
> collect2: ld returned 1 exit status
> make[7]: *** [libnptest.so] Error 1
> 
> Dunno if this is because of my setup or because of the patch...
It's not the patch.

Comment 19

[:aceman]

(In reply to Sune Mølgaard from comment #17)
> /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o):
> relocation R_X86_64_32S against `vtable for std::bad_function_call' can not
> be used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad
> value
> collect2: ld returned 1 exit status
> make[7]: *** [libnptest.so] Error 1

You are breaking in the linker (ld) which is strange. Have you compiled other projects successfully recently with this toolchain?
I have seen similar breakage intermittently in my setup, but it always shown some of the TB .o file that was broken. So I deleted it and usually a rebuild of that file and link of libxul succeeded (with no change in code!). These problems stopped happening when I disabled cleancache + zcache in kernel (compressing of disk cache - experimental option). Can you look for similar problematic options in your setup? Also in your HW (memory, CPU overheating).

Comment 20

[David :Bienvenu]

fixed on trunk w/ unit test - http://hg.mozilla.org/comm-central/rev/d1a2fb2627fa

Comment 21

[Sune Mølgaard]

(In reply to comment 19)
Problem may be that I miss something for gcc-4.5 to work, since I normally use 4.6, but since the patch seems to have landed on trunk, I guess there is no need for me to compile since I can just wait for the next nightly :-)

Comment 22

[:aceman]

So why was gcc-4.5 invoked here?
Anyway, have you reopened the bug by mistake?

Comment 23

[Sune Mølgaard]

As per REOPENED, I don't know why SM sets it to that, but it did so again when writing this comment. Manually set it to RESOLVED, FIXED just now.

As per gcc-4.5, I grabbed a mozconfig from the tree and noticed that it called 4.5 and I didn't know if that was because of issues with later versions.

Comment 24

[:aceman]

It is interesting that gcc-4.5 is even found on your system when you have 4.6. I think you can try modifying the mozconfig so that it uses your normal gcc version.
But version 4.5 works for me fine.

Comment 25

[Sune Mølgaard]

Can't remember why I kept it, but I do remember that it was a conscious choice. Not much point to it, if I can't link, though, so I might just go ahead and uninstall it...

Anyway, as I said in comment 21, I'll just grab the latest nightly that just finished compiling on the Tinderbox :-)