249643 - Junk Mail default settings cause excessive downloads and junk marking of unread messages

Status: RESOLVED DUPLICATE of bug 517849

(MailNews Core :: Filters, defect)

Description

[Sam Johnston]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040626 Firefox/0.9.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040626 Firefox/0.9.1

Junk Mail Control dialog has 'Enable Adaptive Junk Mail detection' set by
default, which would be fine were it not for this resulting in the downloading
of every message in the Inbox, and the marking of every unread mail as junk.
This may be acceptable for small boxes, but with 5500 messages in my Inbox (of
which 1100 are unread), this is a significant usability problem, and one which
probably results in the skewing of the bayesian filters. In my case, I leave
important/todo messages unread, which means my most important mail is likely to
be considered junk in future. Incidentally, I use IMAP and Thunderbird on
multiple platforms/machines.



Reproducible: Always
Steps to Reproduce:
1. Download and Install Thunderbird
2. Set up (IMAP) account
3. Observe your mail being downloaded and unread messages being marked as junk

Actual Results:  
All items downloaded and unread marked as junk.

Expected Results:  
Downloaded only headers (+ max 1 email - the one currently being previewed) and
marked none as junk.

Comment 1

[James Lockie]

What version of thunderbird?
I don't see this problem.

Comment 2

[Robert Lange]

I can at least partially confirm this. I run Thunderbird 1.0 (20041206) and use
IMAP.

I can confirm the excessive downloading part. I cannot directly confirm the junk
filter skewing part.

What I can confirm:
I have an inbox with close to 4000 total messages. I get about 50-70 new
messages per day. At my company it is common to copy entire groups of users when
you only intend one recipient. It is also common to send multi-megabyte files so
on an average day I get about 5 or 6 messages of size 1.5 to 5 MB. These are
rarely addressed to me, so I prefer to ignore them. (ergo: I have set
Thunderbird to not d/l and cache messages over 300KB in size.) However, once I
enable adaptive junk mail detection, the next time I click "Get Mail" I can
detect that Thunderbird is downloading every single unread message, even those
with an excessive attachment size.

To make matters worse, after the attachments are downloaded and scanned by the
spam filter, it appears that they are discarded from memory/cache. For example,
if I ease the restriction on downloading messages over 300KB, the messages are
downloaded once, scanned, then appear to be downloaded again! (eg: new headers
arrive, downloading for scanning begins, downloading for scanning ends, messages
marked as junk or not junk, then downloading begins again)

My speculation:
I have not noticed any skewing of junk mail filters. However, I have gotten into
the habit of marking messages with large attachments as read, to avoid future
downloading, so my conitions are not exactly the same as those of the original
poster. I would not doubt that leaving large quantities of messages unread could
skew the results, although mathematically I am not sure if there is anything
that can be done about this.

My recommendation:
If possible, I would recommend that that junk mail filters not download
attachments as part of the scanning procedure, or that they obey the limits a
user sets in "Offline & Disk Space" section of Account Settings. I do not know
enough about the IMAP protocol to know whether this is possible. Any ideas, anyone?

I really love the junk mail filtering, but it is starting to kill the usability
of the app for me. I hope you are able to straighten this out.

Comment 3

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 4

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 5

[Sam Johnston]

> Statistically, we have found that bug reports that have not been confirmed by
a second user after three months are highly unlikely to be the source of a fix
to the code.

A second user has responded in February, having observed the same problem (and
worse - that messages and attachments are downloaded and the attachments
discarded). I'd like to see someone who knows comment on this before it's closed.

Comment 6

[Robert Lange]

Unfortunately, I have nothing really new to offer over my previous comment. I would just like to point out that I have disabled junk mail filtering, specifically because it is such a burden on performance.

Comment 7

[Moreia]

I am using Thunderbird 2.0 on Ubuntu (Feisty) and on Windows, both with IMAP accounts. The default junk mail settings caused all unread mail to be marked as junk on several occasions (set up my IMAP accts on Ubuntu, set up matching IMAP accts on Windows XP, set up husband with TBird for the first time). 

As soon as I clicked "Enable adaptive junk mail controls for this account" along with my preferred settings (move junk mail to junk folder) all of my unread mail was moved into the spam folder and marked as spam.

Comment 8

[James Lockie]

Unmark all that mail as junk and move it back to the inbox.
I think the problem is with the mail and I think that will help retrain the filters.
OR add the from addresses to your address book and then thunderbird will ignore them.
I filter most of my legit mail and all those addresses are in my address book.
I get a ton of junk and thunderbird is pretty good at catching it since leget mail is in my address book and usually filtered.

Comment 9

[Moreia]

Bob Lockie,

That is a fine workaround, but it isn't a solution. The default settings should not move mail en masse to the junk folder -- that is simply too jarring for new users.

-Amanda

Comment 10

[James Lockie]

Ah, you want junk mail filtering to be off by default.
I can live with that. :-)

Comment 11

[Moreia]

Nope.

I want junk mail filtering on by default, but either ...

* the initial settings should not move mail marked as "junk" out of the inbox, or
* the initial settings should not mark every message as junk in one fell swoop.

The fact that people are reporting that every single message in their inbox was marked as junk suggests that there is maybe something off about the filter. 

Comment 12

[James Lockie]

That's the problem, the majority of people don't have a problem with the junk mail settings.

Comment 13

[Moreia]

Right, Bob, but some people do have a problem and I think it is worth getting to the bottom of what the problem is instead of dismissing it out of hand. The existence of a reasonable workaround doesn't make it not a bug.

Comment 14

[James Lockie]

This bug was opened in 2004, I don't think anyone really cares to investigate it.

Comment 15

[JoSH Lehan]

I had this happen to me as well.

I think that the true cause of this bug is that *pre-existing messages* in the Inbox are scanned when junk mail filtering is first activated by the user.

IMO, it would be safer to simply apply junk mail filtering to *newly arriving* messages that arrive in the Inbox in the future.

Notice the difference between *newly arriving* messages and *pre-existing* messages.

Don't surprise the user by going back and retroactively filtering their old messages also!

If the user actually wishes to apply junk mail filtering to their pre-existing messages, then they can select the "Run junk Mail Controls on Folder" menu item, which is designed for doing just that.

Comment 16

[Kent James (:rkent)]

I just noticed this behavior a few weeks ago, and wsmwk pointed me to this bug. There are some real issues here, some of which have easy fixes, but not all. I'll investigate.

Comment 17

[Wayne Mery (:wsmwk)]

sorry - reload page messed it up

Comment 18

[Wayne Mery (:wsmwk)]

rkent in comment #16:
> I just noticed this behavior a few weeks ago, and wsmwk pointed me to this bug.
> There are some real issues here, some of which have easy fixes, but not all.
> I'll investigate.

kent, what was your thinking for this bug?

Comment 19

[Kent James (:rkent)]

The issue that I was considering is that, for someone who is installing Thunderbird for the first time, all existing messages will be downloaded, scanned for junk, and possibly then marked as junk. Then there is the related issue that when you first manually mark a message junk, it will then mark everything as junk until you train a few messages as good. That is not very friendly behaviour to a new user.

The solution should be some method of recognizing that these existing messages are presumed valid, and also a more friendly startup of junk processing. That would probably involve only allowing junk training at first (not automatic processing), adding automatic training of good messages based on whitelisting, and then turning on automatic junk processing only when a minimum number of junk and good messages had been trained.

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Kent James (:rkent) from comment #19)
> The issue that I was considering is that, for someone who is installing
> Thunderbird for the first time, all existing messages will be downloaded,
> scanned for junk, and possibly then marked as junk. Then there is the
> related issue that when you first manually mark a message junk, it will then
> mark everything as junk until you train a few messages as good. That is not
> very friendly behaviour to a new user.

dup to bug 517849?

Comment 21

[Magnus Melin [:mkmelin]]

Yes