Closed Bug 324467 Opened 18 years ago Closed 10 years ago

TBird 1.5 fails to automatically filter on message body, if Anti-Virus quarentining is ON(mailnews.downloadToTempFile=true). POP3 not IMAP.

Categories

(MailNews Core :: Filters, defect)

x86
All
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla1.9.1

People

(Reporter: makaiguy, Assigned: Bienvenu)

Details

(Keywords: fixed1.8.1)

Attachments

(3 files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5
Build Identifier: version 1.5 (20051201)

I have a couple of filters that filter based upon text in the message body. These no longer seem to work automatically on incoming messages since upgrading from 1.0.7 to 1.5.   MANUAL running of the filters works, just not automatic execution when message received. 

All other filters based on TO, FROM, etc, continue to work as intended. 

Not solved by running in safe mode.  Not using global inbox for any account.

Have experimented with "Move message to" and "Set priority to" actions, but neither works.  Have not experimented with other actions.

Confirmed by another user in this thread in mozillazine:
http://forums.mozillazine.org/viewtopic.php?t=371622

Reproducible: Always

Steps to Reproduce:
1.Create filter based upon matching text in message body
2.Receive message which should trigger filter

Actual Results:  
No filter action taken.  No entry in Filter Log.


Expected Results:  
Filter action should be carried out.

Default theme.  Same performance in TBird safe mode.  Same filters worked flawlessly in 1.0.7.
> MANUAL running of the filters works, just not automatic execution when message received. 
Umh... oposite to Bug 124641. 

(Q1) POP3? IMAP?
     If POP3, please describe "POP3, not IMAP" clearly in summary, in order to
     distinguish this bug with Bug 233902 which is DUP of Bug 127250.
(Q2) Does problem occur on any search word, any body text, any mail, any account?
     Is there any specific condition?
     (e.g. when the search word is splitted to multi-line with specifiyng)
     ( format=flowed,delsp=yes.)
(In reply to comment #1)

> POP3? IMAP?

POP3 for me.

> Does problem occur on any search word, any body text, any mail, any account?

Yes, yes, yes, and yes.

> Is there any specific condition?

Not that I can tell.
(Q3) And any target folder of "move to" action of filter rule?
     (Specific folder which is busy when downloading, and so on, or not.)
(Q4) What filter log says.
     Filter condition is matched but not moved?
     Or filter condition isn't match only when downloading?
Sorry, I missed your "No entry in Filter Log".
Replaced (Q4) :
  'No entry" is only for the filter for body text when download?
  Does it mean filter log for "To:" or "Cc:" is properly written when download? 
 
Can you attach msgFilterRules.dat content of a non-working filter?
 (1) Edit msgFilterRules.dat in mail directry by text editor, then save a set of
     "name=" line to "condition=" line of non-working filter to a text file.
 (2) Attach the file to this bug(mime-type=text/plain)
(In reply to comment #1)
> (Q1) POP3? IMAP?
>      If POP3, please describe "POP3, not IMAP" clearly in summary, in order to
>      distinguish this bug with Bug 233902 which is DUP of Bug 127250.

POP3. I have no experience with IMAP at all.  Have modified summary line.

> (Q2) Does problem occur on any search word, any body text, any mail, any
> account?
>      Is there any specific condition?
>      (e.g. when the search word is splitted to multi-line with specifiyng)
>      ( format=flowed,delsp=yes.)

Far as I can tell it just isn't doing the search in the message body.  

The vast majority of my mail is downloaded from one account and all my incoming filters are set up in that one account.

Example:  Before submitting bug, I set up a simple filter as follows, in the account in which I download my mail:
Body || Contains || FILTER THIS --- move to [test folder I set up for this in the same account]
... and sent myself a message whose entire body was the two words FILTER THIS (plus my normal signature as usual).

Filter was not activated upon message arrival.  No entry in Filter Log.  No error message about folder being busy.  Not compacting folders at the time.  

Interestingly, I have another filter, VERY FAR DOWN THE PRIORITY LIST, which checks for my address in the FROM field and moves to my SENT folder.  This picked the message up and moved it into my SENT folder, and placed appropriate note in filter log.  I disabled this second filter and re-sent myself the message.  Upon arrival it was no different than before, except, of course, the disabled filter didn't move it to SENT folder.
Summary: TBird 1.5 fails to automatically filter on message body → TBird 1.5 fails to automatically filter on message body, POP3 not IMAP
(In reply to comment #3)
> (Q3) And any target folder of "move to" action of filter rule?

Have tried several different folders as "move to" destination, both in same downloading account and in another account.  None work. 

>      (Specific folder which is busy when downloading, and so on, or not.)

As said in previous reply: No error message about folder being busy.  Not compacting folders at the time. 

> (Q4) What filter log says.
>      Filter condition is matched but not moved?
>      Or filter condition isn't match only when downloading?

Filter log has no entry at all for this filter upon downloading.  Proper entries for all the other filters which have worked as intended, but none of them are matching text in message body.

Tools | Run Filter on Folders while viewing Inbox causes the same filter to operate and place appropriate entry in Filter Log.  Ditto for manually running filter from filter list.

Edited msgFilterRules.dat attached as requested.
(In reply to comment #8)
> condition="OR (body,contains,FILTER THIS)"

Problem when space in condition string?
What will happen when the condition is changed to logic like next.
 IF body contains "FILTER", AND IF body contains "THIS", THEN move to a folder.
 msgFilterrules.dat probably becomes;
   condition="AND (body,contains,FILTER) AND (body,contains,THIS)"
Will it work when download with Tb 1.5?
> Problem when space in condition string?
> What will happen when the condition is changed to logic like next.
>  IF body contains "FILTER", AND IF body contains "THIS", THEN move to a folder.
>  msgFilterrules.dat probably becomes;
>    condition="AND (body,contains,FILTER) AND (body,contains,THIS)"

OK, tried that.  No difference.

> Will it work when download with Tb 1.5?

If I'm understanding the question, this _IS_ TB 1.5.   Such filters always worked in 1.0.7. (_with_ spaces in the condition string).  My original filter, which brought this to my attention before screwing around with the simplified test case above, checked for database error messages sent by the vBulletin bbs I administer.  It checks for text string "mysql error number: 0" and always worked fine under 1.0.x.

And remember, these all work fine when run manually but not automatically upon message download.

(In reply to comment #10)
> > Problem when space in condition string?
> > What will happen when the condition is changed to logic like next.
> >  IF body contains "FILTER", AND IF body contains "THIS", THEN move to a folder.
> >  msgFilterrules.dat probably becomes;
> >    condition="AND (body,contains,FILTER) AND (body,contains,THIS)"
> 
> OK, tried that.  No difference.

Further comment.  But even if it worked, it would not be what I was desired because it would catch any message body that happened to contain both FILTER and THIS, not just the specific phrase FILTER THIS.
(In reply to comment #10)
> It checks for text string "mysql error number: 0" and always
> worked fine under 1.0.x.

> And remember, these all work fine when run manually but not automatically upon
> message download.

Which do you say on automatic filtering(during download) when Tb 1.0.x?
 (A) The filter of "mysql error number: 0" DID    work well when Tb 1.0.x.   
 (B) The filter of "mysql error number: 0" DIDN'T work well when Tb 1.0.x.
("And remember, ..." is for Thunderbird 1.5?)

(In reply to comment #11)
> But even if it worked, it would not be what I was desired
That is only test for problem determination.
Other concerns:
 (1) The message filter rule is processed or not when download.
     Special character in name has relation or not when download.
     (1-1) What happens when condition of the filter is changed to
           "subject,contains,FILTER THIS", keeping name as "FILTER THIS!!!".
     (1-2) What happens when name is changed to "FILTER THIS",
           keeping condition as "body,contains,FILTER THIS".
 (2) "Where the text is placed in mail source" has relation or not.
     (2-1) Difference between;
           - "FILTER THIS" is in the first line of mail body
             (in mail source, not mail display)
           - "FILTER THIS" is not in the first line of mail body
I recomend you to see what happens when next simplest case.
  (1) No special character in filter name, such as "AAA".
  (2) Condition of "body,contains,AAA" as the first(or only) rule in filter.
  (3) Text of "AAA" in the first line(or only one line) of mail body
      of text/plain mail.
(In reply to comment #12)
 
> Which do you say on automatic filtering(during download) when Tb 1.0.x?
>  (A) The filter of "mysql error number: 0" DID    work well when Tb 1.0.x.   
>  (B) The filter of "mysql error number: 0" DIDN'T work well when Tb 1.0.x.

 (A) The filter of "mysql error number: 0" DID    work well when Tb 1.0.x.

There has been NO problem under 1.0.x.  Failure of automatic execution problem began with installation of 1.5.   

By the way 1.5 installation was NOT on top of previous version.  I renamed 'Mozilla Thunderbird' program file directory to 'Mozilla Thunderbird1.0.7' and allowed installation program to create a new 'Mozilla Thunderbird' directory in default location.

> ("And remember, these all work fine when run manually but not automatically upon
message download." is for Thunderbird 1.5?)

Yes, manual application works under TBird 1.5.  Automatic execution does not work under 1.5.  I doubt that I ever had occasion to try manually under 1.0.x since it worked automatically as desired.
> 
> (In reply to comment #11)
> > But even if it worked, it would not be what I was desired
> That is only test for problem determination.

I hoped that was the case, but since I've apparently not made mysclf clear on several items that you've asked about more than once, I wanted to be sure I was being understood.

(In reply to comment #14)
> I recomend you to see what happens when next simplest case.
>   (1) No special character in filter name, such as "AAA".
>   (2) Condition of "body,contains,AAA" as the first(or only) rule in filter.
>   (3) Text of "AAA" in the first line(or only one line) of mail body
>       of text/plain mail.

New filter, at top of priority list:
Filter name - "AAA" (without the quotes)
Single filter condition - Body || Contains || AAA
Action move to folder

Sent email as follows:
Subject - AAA-1
Entire body - AAA (no signature)

Filter did not activate upon reception.  
> actionValue="mailbox://fdw@mail.chartermi.net/Inbox/NotToMe"

Is "Inbox/NotToMe" is usual sub-folder named "NotToMe" in parent folder named "Inbox"?
Or virtual folder(saved search folder) named "Inbox/NotToMe"?
If latter, anything bad can occur due to Bug 286523 ...
(In reply to comment #17)
> > actionValue="mailbox://fdw@mail.chartermi.net/Inbox/NotToMe"
> 
> Is "Inbox/NotToMe" is usual sub-folder named "NotToMe" in parent folder named
> "Inbox"?

Yes, exactly.  Just using that one for testing purposes since nothing else goes in there.  However changing the destination folder doesn't make any difference.  Have tried it with destination folder in same account being downloaded into, and as folder in different account than being downloaded into, with same inability to auto-run upon download.

> Or virtual folder(saved search folder) named "Inbox/NotToMe"?
> If latter, anything bad can occur due to Bug 286523 ...

Not a virtual folder.  I've never even experimented with this feature. 
(In reply to comment #16)
> Sent email as follows:
> Subject - AAA-1
> Entire body - AAA (no signature)
> 
> Filter did not activate upon reception.  

I can't still re-create your problem("body,contains,WORD1 WORD2" works well even when download in my test) with Thunderbird 1.5.0, Japanese release version, on Japanese MS Win-2K, with new profile, new account definition.
And I'm still unable to find the reason why even "body,contains,AAA" doesn't work when download in your environment.
So finding "what is the difference between your test and my test" is very important. 

Can you attach this un-filtered mail?
 - Save the mail as ".eml" file, and attach the file with MimeType=text/plain.

I think opening your message filter to public is not so good. 
Can you send me your current msgFilterRules.dat file?
Please put 'Bug 324467' in subject of mail if you can send me the file.

(In reply to comment #18))
> Not a virtual folder.  I've never even experimented with this feature.
Oh, you are a unlucky guy, because you can not enjoy Bug 286523 ;-)
are you quarantining incoming messages? tools | options | privacy | anti-virus?
(In reply to comment #19)
> Can you attach this un-filtered mail?
>  - Save the mail as ".eml" file, and attach the file with MimeType=text/plain.

Done.  Since it reveals an email address that I never use publicly, I've masked it with XXXes, otherwise it's the email as received.

> I think opening your message filter to public is not so good. 
> Can you send me your current msgFilterRules.dat file?
> Please put 'Bug 324467' in subject of mail if you can send me the file.

Done
> 
> (In reply to comment #18))
> > Not a virtual folder.  I've never even experimented with this feature.
> Oh, you are a unlucky guy, because you can not enjoy Bug 286523 ;-)
> 

One aggravation at a time, please.  ;-)
tools | options | advanced | config editor - what's the value of mailnews.downloadToTempFile ? If it's true, I suspect that's the problem.
(In reply to comment #20)
> are you quarantining incoming messages? tools | options | privacy | anti-virus?

Yes.  Never thought of that as a possibility, but definitely is a difference with 1.5 over earlier versions.

Turned antivirus quarentining OFF, sent the mail again, and the filter WORKED, automatically moving message to desired folder upon message delivery!!!
ok, good. I'll try to fix it.
(In reply to comment #23)
> tools | options | advanced | config editor - what's the value of
> mailnews.downloadToTempFile ? If it's true, I suspect that's the problem.

Now FALSE, but enabling quarentining of course changes it back to TRUE.

Good catch, David.  Bet I'd never have thought to try that on my own.
(In reply to comment #20)
> are you quarantining incoming messages?

I was quarantining messages. Now I'm not, and my filters work. Thanks, David!
Changing summary according to comment #23 to 26.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: TBird 1.5 fails to automatically filter on message body, POP3 not IMAP → TBird 1.5 fails to automatically filter on message body, if Anti-Virus quarentining is ON(mailnews.downloadToTempFile=true). POP3 not IMAP.
this is really difficult to fix at the moment - the code that evaluates the filter criteria vs. the msg header has no idea about the temporary quarantine file where the msg body is, and educating that code could require changing several interfaces in the filter code, which I'm a bit reluctant to do. It's nice to have the filters run on the temporary file, because then messages that end up getting moved by filters never get put in the inbox at all. Either I have to set up out of band that the filter code needs to get the msg body from a different input stream, or I have to pass the input stream all the way down. I could put the input stream in the nsIMsgFilter or nsIMsgFilterList, but those are both hacky. I could add it to nsIMsgFilterHitNotify.idl, and pass that in to MatchHdr (or add an nsIMsgFilterHitNotify member to the filter/filter list object) - that's slightly less hacky, since we put the polymorphism in nsIMsgFilterHitNotify...

While I'm at it, I should fix the incoming duplicate detection code to work when msg quarantining is turned on, which I notice it doesn't.
Assignee: mscott → bienvenu
Component: General → MailNews: Filters
Product: Thunderbird → Core
Version: unspecified → Trunk
(In reply to comment #23)
> mailnews.downloadToTempFile

David, is this option applicated to Global Inbox(usually 'Local Folders') too?
If it's applicable to defering account(who uses Global Inbox) only, just after RETR only, "moving rules from an account to Local Folders" can be a workaround for Global Inbox users, since message filter of Local Folders is kicked when download by Tb 1.5, although this workaround makes fallback to 1.0.x difficult.    
this bug affects the users of global inbox as well.
Attached patch proposed fixSplinter Review
This patch moves the creation of the scope term to the outer level, ApplyFiltersToHdr, so that we only create the scope one time instead of once for each filter. We pass in the nsILocalFile that will get to where the message is stored, so for the case of the temp file, we can pass in the temp file instead of the mail folder. I couldn't find a way of doing this w/o changing nsIMsgFilterList.idl, but it is a noscript interface...
Attachment #209966 - Flags: superreview?(mscott)
pinging for review - I'd like to get this out of my tree and baked on the trunk.
pinging for review probably works better when Scott is cc'ed :-)
Attachment #209966 - Flags: superreview?(mscott) → superreview+
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Keywords: fixed1.8.1
This seems to be broken in Linux as well, still, in 2.0

In addition, it impacts Virtual Folders (aka "Saved Searches") in TB2.0.  It's very irritating; the folder count updates correctly, but you cannot find the new messages in the Virtual Folder.

If you uncheck the Anti-virus box, the folder count doesn't update when new messages come in or you read a message.  Only if you click on a folder.  However, now you can find the new messages. :-/

Mozillazine threads:
http://forums.mozillazine.org/viewtopic.php?t=542817
http://forums.mozillazine.org/viewtopic.php?t=538342
http://forums.mozillazine.org/viewtopic.php?t=541310

ciao!
Status: RESOLVED → REOPENED
OS: Windows XP → All
Resolution: FIXED → ---
I forgot to mention.  This is with IMAP.

Ciao!
A restart of Thunderbird seems to have fixed the problem with the Virtual folders message count not updating when the anti-virus box is unchecked.

I'll add another comment if I see the bad behaviour come back.

Ciao!
(In reply to comment #33)
> pinging for review - I'd like to get this out of my tree and baked on the
> trunk.
> 

Does this imply that it might come to branch soon?

looks like it's already on the 2.0 branch.

This is pop only - it has nothing to do with imap.
Would you like me to open a new bug, then, for IMAP?

Ciao!
IMAP filtering on the body of incoming messages is an other long standing, open bug, so please don't open a new one for it...
David B:
 Two things:
  1) This virtual folder stuff *works* in 1.5, but not 2.0. 
  2) I'm not filtering on the body, but the headers.

Ciao!
Christian, IMAP is simply not affected by the anti-virus quarantining - I'm not sure what bug you're seeing, but it's not related to this one, or the anti-virus quarantining - it might be an other existing bug, but I can't tell from what you've said in this bug...
QA Contact: general → filters
Product: Core → MailNews Core
I am experience similar problems, but it is more universal. My message filters work fine and as intended when the "anti-virus quarantining" is turned off. Filters also run normally and as expected if being run manually.

When anti-virus quarantining is turned ON, my filters will not work. It is not limited to the body as this bug states. It also doesn't work for "from" and for the "subject".  The filter log says the filter is being applied, however, it doesn't happen.

If I choose to "move" instead of "copy", I will get an error that says "The message could not be filters to folder 'xxx' because another operation is in progress."

Anti-virus: Zone Alarm 7.0.483.000
POP3
Thunderbird 2.0.0.16
re-closing
Status: REOPENED → RESOLVED
Closed: 18 years ago10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla1.9.1
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: