Last Comment Bug 719037 - Most mail filtered into wrong folder (into the folder of the filter that matched the first message)
: Most mail filtered into wrong folder (into the folder of the filter that matc...
Status: RESOLVED FIXED
: regression
Product: MailNews Core
Classification: Components
Component: Filters (show other bugs)
: Trunk
: All All
: -- normal (vote)
: Thunderbird 12.0
Assigned To: David :Bienvenu
:
:
Mentors:
Depends on:
Blocks: 402392
  Show dependency treegraph
 
Reported: 2012-01-18 07:38 PST by Sune Mølgaard
Modified: 2012-01-20 02:43 PST (History)
6 users (show)
mozilla: in‑testsuite+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
msgFilterRules.dat (4.72 KB, text/plain)
2012-01-18 07:38 PST, Sune Mølgaard
no flags Details
Email that got filtered to the linux-kernel folder (40.54 KB, text/plain)
2012-01-18 07:55 PST, Sune Mølgaard
no flags Details
proposed fix with unit test (8.02 KB, patch)
2012-01-18 16:34 PST, David :Bienvenu
no flags Details | Diff | Splinter Review
fix line ending in data file (8.06 KB, patch)
2012-01-18 17:04 PST, David :Bienvenu
standard8: review+
Details | Diff | Splinter Review

Description Sune Mølgaard 2012-01-18 07:38:32 PST
Created attachment 589500 [details]
msgFilterRules.dat

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0a1) Gecko/20120118 Firefox/12.0a1 SeaMonkey/2.9a1
Build ID: 20120118003001

Steps to reproduce:

Set up a number of filters to move mail to different filters.


Actual results:

Mail that shouldn't get filtered (i.e. that should appear in the Inbox) ends up in the folder that my last filter is set to move mails to.


Expected results:

Filters should apply only to the mail criteria that the filters are actually set to hit...
Comment 1 :aceman 2012-01-18 07:44:22 PST
Can you also attach a sample message (export as EML and you can also remove the body text) that gets filtered wrongly?
Comment 2 Sune Mølgaard 2012-01-18 07:55:23 PST
Created attachment 589508 [details]
Email that got filtered to the linux-kernel folder
Comment 3 David :Bienvenu 2012-01-18 07:56:29 PST
Is it the case that when a message triggers a filter, all the other messages trigger that same filter? What kind of filter is it, e.g., Subject contains?
Comment 4 David :Bienvenu 2012-01-18 07:57:30 PST
and are you using pop3 or imap?
Comment 5 Sune Mølgaard 2012-01-18 08:01:34 PST
That's very possible, and since the linux-kernel list is high volume, it would explain that most are just filtered to there.

Filter is Sender Contains or From, To, CC or Bcc contains.

Also, protocol is pop3
Comment 6 :aceman 2012-01-18 08:04:17 PST
Did this start to happen at any particular date?
Comment 7 Sune Mølgaard 2012-01-18 08:12:31 PST
Hard to say, since pop3 and filters has presented a slew of problems as of late (most of January). Thus, some mails may have have ended up in the linux-kernel folder due to other problems...

It would seem to have happened on a large scale since at least the 12th, but possibly before that.

I don't recall any problems (of this nature) when trunk was on 2.8a1, though...
Comment 8 David :Bienvenu 2012-01-18 11:11:32 PST
I couldn't reproduce this with a "From" filter, but I think I reproduced it with a Cc filter, in that the cc filter fired on messages it shouldn't have.
Comment 9 Sune Mølgaard 2012-01-18 13:41:01 PST
IIRC, the "Sender" field is one that one needs to define oneself as a custom field. I don't know why that is, and neither do I know if it's still the case, but I do seem to remember defining it as a possible filter trigger a long time ago, but I have no idea where it is stored, as it doesn't seem to be in msgFilterRules.dat...

It's good to know, though, that you see it in a CC filter, as it should be a reasonable guess that it could be the same problem. It's also possible, though, that it's the CC part of "From, To, CC or Bcc" part that kicks in, and in my case, almost all filters employ that trigger...
Comment 10 David :Bienvenu 2012-01-18 13:56:42 PST
OK, I believe the issue is that earlier messages pollute the state of later messages, if messages are moved by the folder. The first message has a msgHdr object created for it, storing the headers of the message. If that message gets moved by a filter, the next message gets created with the same key, and doesn't get its headers cleared out, so if the first message had a cc header, and the second one did not, the hdr for the second message will have the first message's cc header, and trigger the same filter. If the second message had a different cc header, it would overwrite the earlier cc header, and the filter wouldn't fire.

I have a potential fix, but my pop3 server is really flaky right now, which makes verifying the fix tricky.
Comment 11 Sune Mølgaard 2012-01-18 14:07:23 PST
Due to just having moved, I don't have my stationary machine handy, but I should be able, although at a much slower rate, compile and test a Seamonkey build with your patch. I follow a number of MLs, so the bug should present itself rather quickly, if it hasn't been fixed by your patch.

Starting DL'ing source now...
Comment 12 Sune Mølgaard 2012-01-18 15:04:18 PST
I have to head for the bed soon, as I'm on UTC+1, but do attach any patches that you come up with, and I'll build and test tomorrow.
Comment 13 :aceman 2012-01-18 15:17:57 PST
David, is this some known bug? I have never heard about this while weeding through bugzilla.
Comment 14 David :Bienvenu 2012-01-18 15:27:31 PST
(In reply to :aceman from comment #13)
> David, is this some known bug? I have never heard about this while weeding
> through bugzilla.

I first heard of it this morning, and it's a regression from the pluggable store landing (to be precise, this regression was hidden by the truncation issue, but fixing that exposed this issue).
Comment 15 David :Bienvenu 2012-01-18 16:34:17 PST
Created attachment 589703 [details] [diff] [review]
proposed fix with unit test

this should fix it.
Comment 16 David :Bienvenu 2012-01-18 17:04:53 PST
Created attachment 589708 [details] [diff] [review]
fix line ending in data file

imap fake server is picky about line endings
Comment 17 Sune Mølgaard 2012-01-19 06:42:21 PST
Compilation fails with:

/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o): relocation R_X86_64_32S against `vtable for std::bad_function_call' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[7]: *** [libnptest.so] Error 1

Dunno if this is because of my setup or because of the patch...
Comment 18 David :Bienvenu 2012-01-19 07:22:23 PST
(In reply to Sune Mølgaard from comment #17)
> Compilation fails with:
> 
> /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o):
> relocation R_X86_64_32S against `vtable for std::bad_function_call' can not
> be used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad
> value
> collect2: ld returned 1 exit status
> make[7]: *** [libnptest.so] Error 1
> 
> Dunno if this is because of my setup or because of the patch...
It's not the patch.
Comment 19 :aceman 2012-01-19 07:29:34 PST
(In reply to Sune Mølgaard from comment #17)
> /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a(functexcept.o):
> relocation R_X86_64_32S against `vtable for std::bad_function_call' can not
> be used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-linux-gnu/4.5.4/libstdc++.a: could not read symbols: Bad
> value
> collect2: ld returned 1 exit status
> make[7]: *** [libnptest.so] Error 1

You are breaking in the linker (ld) which is strange. Have you compiled other projects successfully recently with this toolchain?
I have seen similar breakage intermittently in my setup, but it always shown some of the TB .o file that was broken. So I deleted it and usually a rebuild of that file and link of libxul succeeded (with no change in code!). These problems stopped happening when I disabled cleancache + zcache in kernel (compressing of disk cache - experimental option). Can you look for similar problematic options in your setup? Also in your HW (memory, CPU overheating).
Comment 20 David :Bienvenu 2012-01-19 07:39:52 PST
fixed on trunk w/ unit test - http://hg.mozilla.org/comm-central/rev/d1a2fb2627fa
Comment 21 Sune Mølgaard 2012-01-19 10:18:23 PST
(In reply to comment 19)
Problem may be that I miss something for gcc-4.5 to work, since I normally use 4.6, but since the patch seems to have landed on trunk, I guess there is no need for me to compile since I can just wait for the next nightly :-)
Comment 22 :aceman 2012-01-19 11:13:39 PST
So why was gcc-4.5 invoked here?
Anyway, have you reopened the bug by mistake?
Comment 23 Sune Mølgaard 2012-01-20 02:19:52 PST
As per REOPENED, I don't know why SM sets it to that, but it did so again when writing this comment. Manually set it to RESOLVED, FIXED just now.

As per gcc-4.5, I grabbed a mozconfig from the tree and noticed that it called 4.5 and I didn't know if that was because of issues with later versions.
Comment 24 :aceman 2012-01-20 02:33:49 PST
It is interesting that gcc-4.5 is even found on your system when you have 4.6. I think you can try modifying the mozconfig so that it uses your normal gcc version.
But version 4.5 works for me fine.
Comment 25 Sune Mølgaard 2012-01-20 02:43:12 PST
Can't remember why I kept it, but I do remember that it was a conscious choice. Not much point to it, if I can't link, though, so I might just go ahead and uninstall it...

Anyway, as I said in comment 21, I'll just grab the latest nightly that just finished compiling on the Tinderbox :-)

Note You need to log in before you can comment on or make changes to this bug.