Closed Bug 275190 Opened 20 years ago Closed 20 years ago

Mozilla no longer filters downloaded POP3 mail after upgrade to Moz 1.7.5 (global inbox is not being used ! )

Categories

(MailNews Core :: Filters, defect)

Product:
MailNews Core
Component:
Filters
Version:
1.7 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED DUPLICATE of bug 275558

People

(Reporter: xenite, Assigned: Bienvenu)

References

Details

(Keywords: regression)

Attachments

(1 file)

IMAP log for filter not working automatically.
184.48 KB, text/plain
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217

I have set up  messagefilters to move YahooGroup messages to their folders. This
was working fine for years but right after I upgraded to 1.7.5 Mozilla just
simply leaves them in the inbox. Manually running the filter on the inbox ("run
filters on folder") does move the messages to their folders. I am not using a
global inbox.

This filter no longer gets used automatically:

name="some_filter"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/some_folder"
condition="OR (\"Mailing-List\",contains,some_filter@yahoogroups.com)"

This filter still seems to work:
name="testfilter"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/testfolder"
condition="OR (from,is,Steffen.Motzer@gmx.de)"


Reproducible: Always
Pointer to similar Bug 275268, according to your comment in the bug.

Steffen Motzer, what does filter log say about the unmoved incoming mail?
Log says "condition met and moved", but the mail is not moved?
Or no "condition met" log?

Are there any filter rules of other accounts which will move incoming mail to
the "some_folder"?

*** Bug 275268 has been marked as a duplicate of this bug. ***
I thought, looking at the example filter that's not working, that perhaps it was 
a problem only for filters based on a custom header; but I tested one of those 
(on a POP mailbox) and did not encounter a problem.

Is there more than one mail account actively retrieving mail when the filter 
fails?  Is the account with the failing filter POP or IMAP mail?
Component: General → MailNews: Filters
Product: Mozilla Application Suite → Core
Version: unspecified → 1.7 Branch
Similar question to Mike's comment #3.

Similar problem to Bug 275132?
Acording to Bug 275132, "target folder busy" may occur, if filter of multiple
accounts try to move mail to same folder, then mail is not moved to target folder.
This is possibly a result of some design changes, if design change was done ;
 - Old Mozilla downloaded mails of multiple accounts serialy.
 - But new Mozilla downloads mails of multiple accounts concurrently.

Note: Bug 275132 is a spin-off from Bug 168648, and both bugs are report on
Thunderbird 1.0.
There is a bug relates to parallel POP3 connection.
> Bug 267361 : Parallel connections to different pop servers?
In Bug 267361 Comment #1, David says (on 2004/11/02) ;
> I believe we do that, unless you're filtering the different servers
> into the same inbox, in which case we serialize the get new mails
> so that we won't have contention over the local store.
This sounds for me that parellel POP3 connection is already implemented and
serialization is done only when Global Inbox.

The filter log is completely empty after receiving mail that should have been
filtered. After manually applying the filter there are entries in the filter
log. I have two POP3 accounts and one NNTP account.Only the first POP3 account
and the NNTP account have filter rules. There is only one rule for each
folder/mailinglist and I manually download e-mail from both POP3 servers but not
at the same time.
(In reply to comment #6)
> Only the first POP3 account and the NNTP account have filter rules.
> There is only one rule for each folder/mailinglist
> I manually download e-mail from both POP3 servers but not at the same time.
What is your setting of "Check for new message every NN minutes" for 2 POP3 / 1
NNTP?
Does filter rule for News account have action of "move to folder" to the
"some_folder" of "Local Folders" account?

"Check for new message every NN minutes" and "Check for messages at startup" are
both disabled for all accounts. The filter rule for my NNTP account is a delete
rule.
(In reply to comment #6)
> I manually download e-mail from both POP3 servers but not at the same time.
By "Get Msgs" button?
Or by "Get All New Messages" of expanded menu?
Or specific accout name of expanded menu?

If "by Get Msgs button", which account is selected when the button is clicked?   

> The filter log is completely empty after receiving mail that should have been
> filtered.

You said in commnet #0 ; 
> This filter still seems to work:
>  name="testfilter"

Always no log for all of filter rule of any account?
Or always no log for all of filter rule of specific account?
Or logs are written sometimes except for the problematic filter rule of
name="some_filter"?
(My question is "Message filter did run or not" when mails are downloaded.) 
Another question.
Are there any duplicate filter rule name?
See msgFilterRules.dat contents.
(If renamed, duplicate check is(or was) not done, then causes(or caused) some
problems. I don't know whether this is fixed or not.)  
Bug 275558 reports similar filtering problem after migration to Mozilla 1.7.5
and only when automatic mail filtering for incoming mail.
Bug 275558 case is filter for "List-Id" header case, and "List-Id" header causes
problem of Bug 124641 many time, filter/seacrh problem when multiple-line header.
> Bug 124641 : Filter or Search: do not handle multi-line (wrapped, folded)
headers correctly.

Is "Mailing-List:" header in your unmoved mail multi-line header?
If yes, is the mail address you want to filter placed in first line? or second
or later line?

But if your problem and Bug 275588 is multiple line header problem,
I cannot imagine reasons why the problem occurs only when 1.7.5 and "automatic
filtering for incoming mail"...
I download e-mail from my POP3 accounts by selecting an account and then
clicking the "Get Msgs" button. After I finish downloading the e-mail from the
first account I select the other POP3 account and click the "Get Msgs" button again.

After downloading mail there are only entries in the filterlog for those filters
that did actually work ("testfilter"). There are no entries at all for
"some_filter" (my "Mailing-List:" custom-header testfilter):

Applied filter "testfilter" to message from Steffen Motzer
<blanked_to_keep_spambots_from_harvesting_address> - testfilter at 2004-12-22
09:35:59 Action = Move to folder mailbox://nobody@Local%20Folders/testfolder id
= 41C931EF.3000701@blanked_to_keep_spambots_from_harvesting_address

After manually applying the filters to the inbox this entry is appended to the
filterlog:

Applied filter "some_filter" to message from Steffen Motzer
<blanked_to_keep_spambots_from_harvesting_address> - custom-header test at
2004-12-22 09:36:21 Action = Move to folder
mailbox://nobody@Local%20Folders/some_folder id =
41C93205.9030704@blanked_to_keep_spambots_from_harvesting_address

There are no duplicate filter rules in msgFilterRules.dat.

The "Mailing-List:" header doesn't use more than one line. Here is a (modified)
example:

Mailing-List: list MerwolfPack%obfuscated_to_prevent_spam%@yahoogroups.com;
contact MerwolfPack-owner%obfuscated_to_prevent_spam%@yahoogroups.com
(In reply to comment #12)
> After downloading mail there are only entries in the filterlog
> for those filters that did actually work ("testfilter").
Message filter is aparently executed when mail is downloaded.

> The "Mailing-List:" header doesn't use more than one line.
>  Here is a (modified)
> example:
> Mailing-List: list MerwolfPack%obfuscated_to_prevent_spam%@yahoogroups.com;
> contact MerwolfPack-owner%obfuscated_to_prevent_spam%@yahoogroups.com

Looks multiple-line(folded header).
Is this header displayed as single line when "View Message Source"?

Another possibility.
As far as I remebmer, I saw a bug that said result becomes "False" if condition
is sutisfied more than once, though I can not recalll bug number...
But even if this or Bug 124641, why problem did not occur before 1.7.5?
Why problem does not occur when "Run Filters on folder"?
(In reply to comment #13)
> (In reply to comment #12)
> > The "Mailing-List:" header doesn't use more than one line.
> >  Here is a (modified)
> > example:
> > Mailing-List: list MerwolfPack%obfuscated_to_prevent_spam%@yahoogroups.com;
> > contact MerwolfPack-owner%obfuscated_to_prevent_spam%@yahoogroups.com
> 
> Looks multiple-line(folded header).
> Is this header displayed as single line when "View Message Source"?

Yes, the header is displayed as a single line that doesn't wrap. In Mozilla
1.8a5 and 1.7.3 all filters work just fine.
I just installed the latest nightly build 2004122106 to see if it is also broken
in the latest 1.8a6 build and all filters worked without any problems.
Add Moz 1.7.5 in summary for ease of search.
Summary: Mozilla no longer filters downloaded POP3 mail (global inbox is not being used ! ) → Mozilla no longer filters downloaded POP3 mail after upgrade to Moz 1.7.5 (global inbox is not being used ! )
Bug 275558 comment #0 and Bug 275558 comment #5 also reported completely same
symptom as yours, except used "cutomized" message header in filter.
 - Your case             : "Mailing-List:" & "contains"
 - Bug 275558 comment #0 : "List-Id:" & "contains"
 - Bug 275558 comment #5 : "X-Spam-Level" & "contains"
And nexts are found.
 - Message filter is executed on downloading mail.
 - There is no relation to problem when "multi-line header"
 - There is no relation to problem when "searching of decoded header data".

(Current status summary)
Symptom:
  When customized header is used in filter,
       the filter fails to detect,
       when automatic filtering for incoming message.
  This problem occurs only on Mozilla 1.7.5.
  No problem when "Run Filter on folder", manual filtering.
Related builds:
  Problem occurs only 1.7.5.
  No problem on 1.7.3, 1.8a5 and latest-trunk builds after 1.8.a5.
  It is still unclear whether Thuderbird 1.0 has this problem or not.
  It is still unclear too on Thuderbird 1.0 branch latest nightlies.

Additional founds.
 - Posiibilty of problem when target folder busy or contention is very very low,
   because of your comment #8 and comment #12.

I encountered this after installing 1.7.5 two days ago.  I use K9 spam detector
and have it flag spam messages with header line X-Text-Classification: spam. 
Each of 11 accounts has a filter I call K9Spam as first or only filter; each
moves the message to the K9Spam folder of the main account.  This is not working
with 1.7.5.  Resumed working after I fell back to 1.7.3.
I have [unfortunately] upgraded today to 1.7.5, and now I have the same problem.
My  old message filter:
action="Label"
actionValue="5"
condition="AND (\"RT-Ticket\",contains,PortaOne)"
which used to work perfectly in 1.7.3 does not work anymore. Filter log is
empty. If I re-run filters on the folder, then the messages are labeled as they
should have been in the first place.
Change "Assigned to:" to David.
David, help us please.
Protocol log and filter log can do nothing for problem analysis.
Are there any good way to see what is happening in filter?
Assignee: general → bienvenu
(Q1) Can someone get protocol log and attach log file to this bug?
See http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#pop for
protocol log.
Do not forget to change "set NSPR_LOG_MODULES=protocol:5" to POP3:5 or IMAP:5
Please note that log file is overlayed on restart of Mozilla.
(Please don't paste long log to this bug.)

To all of POP3 users.
(Q2) What is your "Leave messages on sever" setting and sub-settings of it?
 - "Leave messages on sever" : checked or unchekd
 - "Fetch header only"       : grayed-out or not, checked or unchekd
 
If "Leave messages on sever" is unchecked and "Fetch header only" is grayed out
but "Fetch header only" is checked, 
(Q3) Will unchecking "Fetch header only" resolve problem?
     - Check "Leave messages on sever" and uncheck "Fetch header only",
     - Then uncheck "Leave messages on sever" again.
See Bug 248742 Comment #6 for problem when grayed out but checked sub-options
under "Leave messages on sever". 


In this attachment there is IMAP log (not POP3, but the problem is the same).
I run Mozilla, press Get New Messages, enter password. New message appears in
my INBOX. Then I press "Run filters on folder" and the message is moved to
another folder.
Filter is "X-ResentFrom contains cstef@mail.ru".
As an addition to attachment, see comments on bug 275558 (the same problem, but
with IMAP account).
There seem to be a few bugs open on this, so I hope this is the "current" one ...

Same probelm with 1.7.5 and IMAP.

My spam moving rule uses custom header:
 "X-Bogosity" contains "yes"
as well as a number of other rules using:
  "From" contains <various strings>.
I am using mail fiters to filter messages from various mailing lists using
Reply-To header. This stopped working when I upgraded from 1.7.3 to 1.7.5. When
I run them manually most did work and messages were filled into proper folders,
but one of them didn't work even this way, I observed that in this case searched
string was on second line of header. When I tried to use some other string from
header from first line which unfortunately was MIME encoded it didn't work
either. When I completely deleted file with filter definitions and recreated
rules from scratch it didn't help. After gone back to 1.7.3 things started to
work as before.
Blocks: 276944
Keywords: regression
(In reply to comment #17)

> Symptom:
>   When customized header is used in filter,
>        the filter fails to detect,
>        when automatic filtering for incoming message.

I think I can confirm that. All my filter using customized headers stopped
working on both IMAP and POP3 accounts when I upgraded to 1.7.5, while those
that filter on sender only still work properly. I don't think it is a "target
folder busy" problem, because even filters that only label the message and do
not move it at all do not work anymore. It also does not seem to be related to
folded header lines, since none of those customized header lines are folded.

Shouldn't this be a blocker? Filters are a crucial part of Mailnews, and
breaking them is by far the most annoying bug I have run into for a long time.

If you want me to run some tests or need more information to identify the
problem please let me know. Otherwise I'll revert to 1.7.3 soon.

Developer seems to be trying to resolve the problem in newer Bug 275558 instead
of this bug.
Watch Bug 275558, please.
> Watch Bug 275558, please.

Ok, thanks.

*** This bug has been marked as a duplicate of 275558 ***
Status: NEW → RESOLVED
Closed: 20 years ago
Resolution: --- → DUPLICATE
Status: RESOLVED → VERIFIED
*** Bug 276056 has been marked as a duplicate of this bug. ***
The only filters that seem to be affected by this bug on my system are those
defined on the "Reply-to" line, and failure is intermittent. Most of the time
the filter works, and it always works manually. This suggests a timing problem,
and the lack of code to enforce action even if timeout reached. The reports that
it has been solved in 1.8 are reassuring, but we have to ask whether the fix is
reliable, if not based on understanding of why it occurred in 1.7.5.
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: