200496 - Spam/junk filters move messages unexpectedly with IMAP (the filter runs at unexpected times)

Status: RESOLVED DUPLICATE of bug 198961

(MailNews Core :: Filters, defect)

Description

[Joseph Delaney]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030328
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030328

I have my spam filters set to move spam to "Junk" folder. Sometimes it misses
messages (Bug 200087) so to track that I started moving those messages to
another folder and  manually marking them as junk. Now when I click on that
folder to view them, the filter runs again and moves those messages to the junk
folder, even though they are not new, have already been marked as junk, and were
deliberately moved to a separate folder. 

Without following the steps to reproduce, you will sometimes see that messages
that have been correctly moved to the junk folder will get unmarked when you
view the folder (after the filters get messed up as in Bug 200087), because the
filter runs again on clicking the folder. 

This could potentially lead to data loss (I have not seen it happen yet) if the
option to delete messages in the junk folder is turned on, and the filters start
misidentifying older messages as junk and moving them to the junk folder. I am
filing this as "critical" for that reason, but someone might want to downgrade
it if my assessment is incorrect.

Reproducible: Sometimes

Steps to Reproduce:
1. Turn on Junk mail filters, with the option to move junk messages
2. Wait for some junk messages to be missed (it might be possible to fake it,
because this step could take weeks)
3. Mark the missed junk messages as junk manually, and  move them to a new folder.
4. click on another folder (it might have to be in a different e-mail account),
then click back on the new folder.

Actual Results:  
The spam filter runs again, and will either move the messages to the junk
filter, or "unmark" them so they are no longer tagged as junk. 


Expected Results:  
The junk filters should only run on new mail, or only on the Inbox. Messages
that are moved for other reasons should stay where  they are. Messages that are
manually marked as junk should stay marked as junk. 

I have not tried to reproduce this with POP, so I do not know if it affects POP
the same way.

I'm not sure why, but my steps to reproduce work often but not always.

Comment 1

[David Favor]

This problem also relates to 195737, 207650 and many other defects.

The broader view of this problem is each message should have some type of
filtered indicator inserted into the physical message.

Ideas:

1) message start out 'Raw', unfiltered with no indicator set.

2) mail filters run for account and if message matched by a filter the
   filter indicator is set to 'Filtered'.

3) after mail filters run, then junk mail controls should run, if turned
   on and messages should be marked as 'Junk' if matched and 'Not Junk'
   if no match occurred.

4) if messages are manually marked as 'Not Junk' their indicator should
   also be marked 'Not Junk'.

5) when entering a new folder, junk mail controls should only operate
   on 'Raw' messages.

This indicator should be set in the msf also and will probably improve
performance by avoiding running junk mail controls on duplicate messages.

Comment 2

[Jason M. Abels]

I can confirm that this happens

Comment 3

[Jason M. Abels]

Forgot to update Hardware and OS. 

Comment 4

[Craig Ringer]

Unfortunately, storing this 'filtered' state in the local mozilla data doesn't
handle the case of multiple mozilla instances accessing one mail account, as I
use (this is one of the advantages of IMAP after all). Think mozilla at work,
when mozilla is also used at home over an SSH tunnel to access the same mail
account (though not generally at the same time).

The filtered state would only be known to a given mozilla profile, so another
profile living on another machine might still re-filter everything.

Additionally, I think it is important to retain the ability to run junk filters
on mailboxes other than the INBOX, given the ability of IMAP to run server-side
filters to deliver into specific mailboxes. Perhaps mailboxes could be marked
"filter for junk" or "don't filter for junk" by the user? This would be stored
on the local machine also, but would only have to be configured once per
mailbox. The INBOX would default to "yes" (if junk filtering was enabled) and
all other mailboxes to "no". The Junk box should probably be locked at "no".

Does this seem like a sensible solution?

Ideally of course data like the junk/not junk flag would be stored on the IMAP
server its self (along with address book, bookmarks, etc in an ideal world)...
but I am lead to understand that this is a rather complex task that exposes even
more mail server quirks than normal IMAP access.

Comment 5

[David Favor]

Having the ability to filter specific mailboxes for junk seems good,
with the default as only filter the INBOX.

This defect is becoming a show stopper for people subscribing to high
volume lists like MTA and CMS systems. I typically filter these messages
to a mailbox. Then when I select that mailbox junk filtering runs. Some
lists have hundreds of messages a day, so filtering takes forever.

Also somewhere along the line the stop button got broken for junk mail
filtering, so I can no longer terminate junk mail processing.

I recommend this bug's criticalness be upgraded to the next level and
the temporary fix be to only run junk filtering on INBOX, with the
permanent solution as the ability to turn on junk filtering per mailbox.

Comment 6

[Joseph Delaney]

Setting "blocking1.5a ?"

Since I filed this bug, I experienced the data loss problem that I suspected
could happen. I've now turned off the Mozilla spam filters entirely, and
installed SpamAssassin on our server with much better results. 

Comment 7

[Asa Dotzler [:asa]]

mozilla1.5a released. unsetting blocking request. nominating for 1.5b.

Comment 8

[Jamie Flournoy]

How about only running filters on unread messages, unless the user selects "Run Filters on Folder"?

Would you expect Mozilla to refilter your whole INBOX every time you start it? (Why would you want 
it to do that?) Or would you only expect it to filter new messages?

Comment 9

[Daniel M. St.André]

I confirm that this happens.  It happens to me often enough that I consider it
100% reproducible.  I HAVE LOST MESSAGES DUE TO THIS BEHAVIOR.  [see note below]

I agree with the comment to leave some bread crumbs in the message as you do the
filtering.

SUGGESTION:  Consider borrowing from SpamAssassin or similar and adding records
to the header to track what is going on as you filter and decide and move and
mark messages.  Perhaps the amount and detail level of these added records might
become a configurable preference.

NOTE:  Lost Messages
    I created a new account and turned on filters and junk mail controls. 
Messages arrived to this account while I was in the setup process.  I had not
yet configured the move vs. delete junk option and the empty the trash option. 
The messages were gone before I could discover what they were, if they were
really junk or what.

Comment 10

[Joseph Delaney]

*** This bug has been marked as a duplicate of 198961 ***