517849 - Only scan NEW emails for junk

Status: RESOLVED DUPLICATE of bug 211826

(MailNews Core :: Filters, defect)

Description

[Karanbir Singh]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-GB; rv:1.9.0.11) Gecko/2009061613 CentOS/3.0.11-2.el5.centos Firefox/3.0.11
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-GB; rv:1.9.1.4pre) Gecko/20090915 Shredder/3.0b4pre

When thunderbird is setup with an IMAP account, it looks to be scanning all unread emails in all folders for junk mail. Given that at install time the junk training data is quite poor, all sorts of emails get marked as junk while they are not, they were just left as unread on the mail server.

Reproducible: Always

Steps to Reproduce:
1.install thunderbird
2.setup a mailbox to look at remote imap-account ( something that already has emails )
3.ensure junk controls are turned on for the account
3.click on 'Get New Messages' 
Actual Results:  
Older emails ( some even 3 years old ) from various folders are marked as junk during the first sync that thunderbird does against the remote imap server, and they are moved into the Junk folder for the account.

Expected Results:  
Emails with a timestamp older than the time when thunderbird was installed, should not be checked for junk status.

This is specially a problem for people like me who have large mailbox's and are working on testing thunderbird under load's. doing a fresh install this morning and importing my work mailbox resulted in 6765 emails from various folders being moved into the Junk mailbox. None of them are junk emails. Some of the emails moved were from over 2 years back!

Comment 2

[Ben Bucksch (:BenB)]

See bug 189970, a request to exclude specific, user-specified folder from Junk
filtering, which is a very often filed dupe.
Most of the people who ask for that have, however, actually a different
problem: They open a folder which has never been spam-filtered by Thunderbird,
e.g. because it's coming from another mailer. Suddenly, Thunderbird starts
Junk-filtering the *old* mail.

Proposal:
Let Thunderbird only check "new" mail. Where "new" is defined as:
- Downloaded via POP3
- IMAP "unseen" flag (or however it's called)

Currently, "new" is defined as "not seen by Thunderbird yet", which leads to
the bug described above.

Comment 3

[Kent James (:rkent)]

I'm not convinced that comment 2's dupe to this email is moving this bug in the correct direction. Comment 0 describes a real issue with new accounts, that they will process all of the existing mail through the junk filter at a time when junk processing is not well trained, and the junk processing will tend to take actions on large numbers of emails that are incorrect. The solution to that could take a variety of directions, including for example adjusting the junk decision algorithm to take into account the amount of training in making its decisions.

The issue in bug 526268 concerns the definition of "new" mail as it affects junk processing, which is an issue in not only junk but other kinds of processing as well (such as filters). The algorithm described is roughly the current mailnews algorithm, so the bug as written should be resolved as WFM. But "UNSEEN" is the same as "unread", so I think the reporter intends "RECENT", and that is worth discussing. My vague recollection from Bienvenu has been that "RECENT" is not a very reliable flag, and so is not used frequently in mailnews, but the details are worth understanding and discussing as a possible fix, in IMAP only, for the issues of bug 189970.

Comment 4

[Ben Bucksch (:BenB)]

I had the same issued with accounts in longer use, but old folders that I had not opened or even subscribed to in Thunderbird. So, I don't think the differentiation between completely new accounts and not is useful. The problem is generally with old mail, not just new accounts.

As for the IMAP flag, I may have gotten it wrong, that's why I explicitly wrote "(or however it's called)". Thunderbird can differentiate between unread and new mail, it does so even in the UI, in folders and thread panes.