224318 - Bayes filtering should learn through use of external/serverside filters

Status: RESOLVED FIXED

(MailNews Core :: Filters, enhancement)

Description

[Robert Accettura [:raccettura]]

Bayes filtering should be aware of the increasingly popular X-Spam headers. 
Products such as SpamAssassin use them to mark suspected spam emails.

Ideally, Bayes should ignore the top message that SpamAssassin attaches to
suspected spam.

Invite discussion on how to deal with Bayes and other spam filtering software.

I'm attaching 2 emails, 1 spam, 1 ham, both filtered through SpamAssassin.  The
spam is very distinctive, as it's always attached, to a spam notice email.  All
have X-Spam.


There is also the possibility to have an option and utalize external spam
filters with/or bayes.  For example, training on SpamAssassin's results of
spam/ham (such as SA's Bayes filtering does).  Or allowing Mozilla to simply
recognize SA's decision as what the email is.  Rather than SA does the scan,
then Mozilla does it again.  This would in essence allow third party filters the
ability to use Mozilla's Spam UI, or work in conjunction with Mozilla.

Comment 1

[Robert Accettura [:raccettura]]

Attachment: Spam

Spam Sample (note attached original email, and headers).

Comment 2

[Robert Accettura [:raccettura]]

Attachment: Ham

Ham.  Note difference from Spam.

Comment 3

[Robert Accettura [:raccettura]]

Note messages *can* be inline, though no longer the default format in SA later
than 2.50, now attachments are default behavior.

Comment 4

[David :Bienvenu]

Scott and I had some ideas about what else we could do with this. We're thinking
a new tab on the spam settings window (which we're proposing to have tabs when
we add more options like this) with the following choices about what to do with
the x-spam-status header:

1. Ignore
2. Trust positives
3. Trust negatives (can trust both pos and neg)
4. Give Weight to x-spam-status (somehow combine x-spam-status result with
bayesian score)

If the user choses to trust both positives and negatives, then we don't need to
run the bayesian filter.

Comment 5

[Robert Accettura [:raccettura]]

I think an option to "use X-Spam" would be good to.  Rather than use Mozilla's
bayes filtering.  Honor the spam filter on the server.  Use Mozilla's UI and
spam handling with the server's decision.  Would cut down on CPU for those users.

I like "give weight"

Would be nice if we can have a checkbox to "feed" the bayes filter and train
with the results from X-Spam.  

Since SpamAssassin as well as other products are relatively accurate, without
user interaction, Bayes could be trained quite quickly in those situations, as
noted here (among other places):
http://www.eweek.com/article2/0,4149,1366242,00.asp

There's a ton of potential.  That little tag can really do a lot to enhance Mozilla.

Comment 6

[Matthias Versen [:Matti]]

Attachment: Header examples

There is one problem: There are different headers with different products.
I use spampal (Mail-Proxy freeware for win32) and it adds different headers.

Comment 7

[Robert Accettura [:raccettura]]

*Detection Checklist*

SpamAssassin:
Ham    X-Spam-Status: No,
Spam   X-Spam-Status: Yes,
Spam   X-Spam-Flag: YES
(x-Spam-Status can have data after Yes, No)
I believe X-Spam-Flag was added in later versions.

SpamPal:
Ham    X-SpamPal: PASS
Spam   X-SpamPal: SPAM 
(spam can have data after the word spam)

SpamCatcher:
Ham   X-SpamCatcher-Flag: No
Spam  X-SpamCatcher-Flag: Yes

Comment 8

[David :Bienvenu]

It occurs to me that we need this to be extensible, and that we need pattern
matching of some sort. So, I'm thinking I should add a filter action to set the
junk score. Then, users can write their own filters to handle some of the
server-side spam products.  Then, integrating with these products becomes a
matter of defining some filters. We could do like the MDN code does and define
the filters internally on the fly, invisible to the user. So I think I'll add a
filter action that sets a junk score.

 

Comment 9

[Robert Accettura [:raccettura]]

I agree, there should be a way through filters.

But I'm thinking in Junk Mail Controls, there should be a new tab, with
checkboxes for:

[ ] Enable Support for External Mail Filters (Select..)
[ ] Enable Habeas Support
[ ] More soon...
and a mention they can define their own filters to customize this further.

In the first one, have a button, that brings a popup asking "Spam Assassin",
"SpamPal", "SpamCatcher".  Able to check multiples.

Turn them on by default (if the user doesn't have spamassassin, it just will
never fire, no real harm done).  If it does, it automatically kicks in.  The
other option is to have Thunderbird detect the first instance.

By having the filter, and the tab in junk mail controls, the user can not only
define their own rules with a filter (power user), but the basics are within
easy reach for the general user.

As more products emerge, we could easily add UI options for a few of the most
popular options.  I think the 3 mentioned are the most popular right now.

Only doing filter rules, would make this feature beyond the casual user, who may
want filtering, but is not geeky enough to make a filter.  Besides.  I think the
above 3 will apply to most of those who want the feature anyway, so it would
work out of the box for most people, and the rest can adapt it to their needs.


As a sidenote bug 11040 is indirectly related to this bug.  I wouldn't say
blocking, but definate influcence.

Comment 10

[David :Bienvenu]

sorry if I wasn't clear - that's what I meant by "Then, integrating with these
products becomes a matter of defining some filters. We could do like the MDN
code does and define the filters internally on the fly, invisible to the user."
So the implementation of that UI would internally be some filters.

One issue I need to deal with is to propagate the junk status set by filters to
the imap server so that if the message gets moved to another folder, the junk
status is also moved.

Comment 11

[Robert Accettura [:raccettura]]

David:  Sounds good to me.

Changing summary of this bug a little, since it's more more than X-Spam Headers
now.  

Comment 12

[David :Bienvenu]

One drawback of this filter approach, as opposed to putting some code in the
code that parses mail headers, is that filters only run on new mail downloaded
in the inbox. If there are server-side filters that classify messages *and* move
them to other imap folders on the server, the client-side spam header detection
filters won't detect them. Not sure if this is an important issue...if it turns
out to be, we could run the internal filters on folders other than the inbox, I
guess. The advantage of using filters is that they're extensible.

Comment 13

[David :Bienvenu]

Attachment: support for filters setting junk score

this patch makes it so the user can set a message as junk or not junk through a
mail filter.

I think I'm going to make this three separate bugs.

1. this one - UI and backend for filters to set junk score.
2. adding hidden custom filters for various well-known server-side plugins.
3. Adding ability to train bayesian filter on server-data

Comment 14

[Robert Accettura [:raccettura]]

(In reply to comment #13)
> 2. adding hidden custom filters for various well-known server-side plugins.
> 

I think I might take that one when I get a few free cycles.

Comment 15

[David :Bienvenu]

Robert, I'll get you started by doing one of them, when I get a few cycles :-)

Comment 16

[Robert Accettura [:raccettura]]

(In reply to comment #15)
> Robert, I'll get you started by doing one of them, when I get a few cycles :-)
If your doing these in seaparate bugs, CC me on them, so I can keep track.  Thanks.

Comment 17

[Scott MacGregor]

Comment on attachment 144072 [details] [diff] [review]
support for filters setting junk score

awesome!

Comment 18

[Gervase Markham [:gerv]]

I think Habeas should write their own as an XPI, personally. Lazy so-and-so's ;-)

Gerv

Comment 19

[David :Bienvenu]

Habeas headers - they suggest filtering on #3...
http://www.habeas.com/configurationPages/headers.htm

I'm thinking the way this will work is that we'll add the ability to load this
kind of spam filter from disk, so we'll store the individual filters on disk.
That way dropping in new kinds of filters won't involve changing the code so much.

Comment 20

[David :Bienvenu]

Attachment: fix for custom headers

turns out custom headers were somewhat broken, in terms of what the UI allowed
you to set.

Comment 21

[David :Bienvenu]

Attachment: filters for spamassassin

Comment 22

[David :Bienvenu]

Attachment: filters for SpamCatcher

Comment 23

[David :Bienvenu]

Attachment: filters for Habeas

Comment 24

[David :Bienvenu]

Attachment: filters for SpamPal

Comment 25

[David :Bienvenu]

I'm thinking the way this might work is we add some attributes to
nsISpamSettings for handling server-side spam filters:

1. ServerSpamFilterName
2. ServerSpamAction - trust yes, trust no, trust both

Then, when we're starting up a server, if the spam filter name is set, we load
the correspondingly named filter file, and enable the Yes and/or No filters,
according to what the user has specified.

As far as the UI for picking the server side spam filter to incorporate is
concerned, I imagine it'll just be a drop down where you can pick from the list
of server-side spam filters we know about (maybe with a default choice of None,
or a checkbox to turn off this behaviour). It would be cool to populate this
list from the .dat files on disk, so that dropping in a new one adds it
automatically to the list, but we might not get there...

Comment 26

[David :Bienvenu]

Comment on attachment 144072 [details] [diff] [review]
support for filters setting junk score

this would involve an exception for the localization freeze (it adds a few
strings) but we'd really like to get this into tbird .6 and Moz 1.7 - the fix
is fairly safe, and allows you to make filters set a junk score.

Comment 27

[David :Bienvenu]

Comment on attachment 144171 [details] [diff] [review]
fix for custom headers

this is needed because the custom headers stuff was always slightly broken...

Comment 28

[chris hofmann]

Comment on attachment 144171 [details] [diff] [review]
fix for custom headers

a=chofmann for 1.7

Comment 29

[David :Bienvenu]

http://bugzilla.mozilla.org/attachment.cgi?id=144171&action=view checked in.

Comment 30

[Robert Accettura [:raccettura]]

David:

If we are learning from positive marks from external spam filters, isn't it
necessary to learn from negatives as well?  Otherwise we are essentially
tainting the built in bayesian filters with one sided results.

Just thinking outloud really.

Comment 31

[David :Bienvenu]

Not sure what you mean - I've added settings to trust both positive and negative
results in my patch, and in the filters (except for Habeas). But I haven't done
anything about actually feeding the data into the spam filter to train it...I'm
probably going to leave that to you or someone else.

Comment 32

[Robert Accettura [:raccettura]]

Hmm.. I retract my last comment.

I apparantly have some networking issues, when I was looking at the filter for
spamAssassin I saw:
>name="SpamAssasinYes"
>enabled="yes"
>type="1"
>action="JunkScore"
>actionValue="100"
>condition="OR (\"X-Spam-Status\",begins with,Yes) OR (\"x-Spam-Flag\",begins
with,YES)"

and that was it... hence my question.

But now I see the rest.  I've also been double posting on at least one forum,
and having connections time out.  So I think I have some networking problem here
at the minute, though my MRTG graph barely shows a change in ping time.

Anyway.  Disregard my last comment.  

Comment 33

[David :Bienvenu]

Attachment: backend support for automatic server spam filter filters

This handles automatically creating hidden filters for a given server-side
filter, if the per-server pref serverFilterName and serverFilterTrustFlags are
set appropriately.

Comment 34

[David :Bienvenu]

Attachment: .sfd (server filter description) files

diff for filter description files (includes a typo fix in SpamAssassin.sfd)

Comment 35

[David :Bienvenu]

Attachment: packager diffs for new .sfd files

Comment 36

[Scott MacGregor]

Comment on attachment 144590 [details] [diff] [review]
backend support for automatic server spam filter  filters

looks great.

Comment 37

[Asa Dotzler [:asa]]

Comment on attachment 144072 [details] [diff] [review]
support for filters setting junk score

a=asa (on behalf of drivers) for checkin to 1.7

Comment 38

[David :Bienvenu]

front and backend support for filters setting junk score checked in.

Comment 39

[Mike Cowperthwaite]

Bug 181631 was already about having Mark as Junk/Not Junk in the message filter 
actions; I've marked it Fixed with an xref to this bug.

I've opened bug 238816 about adding those enhancements for custom-header 
matching to MailViews and Search.

Comment 40

[Stefan Borggraefe]

I think there are some small issues with the strings that were checked in:

> +<!ENTITY setJunkScore.label "Set Junk Status">

The other filter actions that use a combobox all end with a colon. Also I think
this filter action should end with a "to" to be consistend with "Change message
priority to:".

I'm not sure whether "Junk Status" should be upper case or not.

> +<!ENTITY notJunk.label "NotJunk">

There should be a blank between Not and Junk.

Comment 41

[Scott MacGregor]

Attachment: following up on Stefan's suggestions to the filter UI

I agree with Stefan about the language changes he suggested. This patch does
just that. 

1) It adds a space beteen Not and Junk
2) It adds a colon to the phrase: Set Junk Status to to be consisent with
setting the priority
3) I also moved the junk status action in the dialog so it was grouped with the
rest of the combo box driven actions such as setting priority, label the
message, etc. Don't let the wierd way cvs diff generated the patch for that
change fool you. It was just moving a few lines of xul higher up in the file. 

Still have one remaining problem...whenever we read the filter in from disk,
this action always resets to Not Junk even if you had it set to Junk.

Comment 42

[Stefan Borggraefe]

(In reply to comment #41)
> Created an attachment (id=145191)
>
> 1) It adds a space beteen Not and Junk

This is not included in the patch. :-(

Comment 43

[Scott MacGregor]

actually it is. But cvs diff -uw ignores white space and it views that change as
white space so it didn't show up. Weird

:)

Comment 44

[David :Bienvenu]

I'm not able to reproduce the filter returning to non-junk problem, even with a
fresh tree from CVS. Maybe it's a release build only issue...

Comment 45

[Scott MacGregor]

Comment on attachment 145191 [details] [diff] [review]
following up on Stefan's suggestions to the filter UI

asking for 1.7 status for this polish

Comment 46

[chris hofmann]

Comment on attachment 145191 [details] [diff] [review]
following up on Stefan's suggestions to the filter UI

a=chofmann for 1.7

Comment 47

[Scott MacGregor]

Comment on attachment 145191 [details] [diff] [review]
following up on Stefan's suggestions to the filter UI

this patch has been checked in for 1.7 final

Comment 48

[Stefan Borggraefe]

(In reply to comment #41)
> Still have one remaining problem...whenever we read the filter in from disk,
> this action always resets to Not Junk even if you had it set to Junk.

I see this too. The actionValue contains a random number instead 0 or 100 when
the FilterListDialog is opened for the first time after mozilla is started. When
you just open the FilterListDialog and close it immediatly without opening the
FilterEditor this value is written to msgFilterRules.dat.

Also in FilterEditor.js sometimes gJunkScoreCheckbox and sometimes 
gChangeJunkScoreCheckbox is used for something that looks like it should be just
one variable instead. But this is unrelated to the random number problem.

Comment 49

[Scott MacGregor]

Attachment: fixes a bug where the junk action value never gets initialized

this patch fixes one of the issues with this bug fix:

"whenever we read the filter in from disk,
this action always resets to Not Junk even if you had it set to Junk."

We were never reading in the junk mail action value when reading the filter
from disk. Hence, the action value was garbage, causing it to sometimes get set
to mark as junk and sometimes as not junk

However there is still another really nasty issue out there. Any filter that
fires has the random potential of marking mail as junk. Even if the filter does
not have the junk status action checked. See Bug #239349 for information about
that issue.

Comment 50

[Scott MacGregor]

Comment on attachment 145363 [details] [diff] [review]
fixes a bug where the junk action value never gets initialized

david see my comment above that explains this patch.

Comment 51

[Scott MacGregor]

Comment on attachment 145363 [details] [diff] [review]
fixes a bug where the junk action value never gets initialized

uninitialized variables leading to random behavior == good candidate for 1.7
final :)

Comment 52

[Scott MacGregor]

I just found the cause of Bug #239349 which caused messages to get randomly
marked as junk or not junk if you had a filter rule that set a label action.
That fix should also go into 1.7

Comment 53

[chris hofmann]

Comment on attachment 145363 [details] [diff] [review]
fixes a bug where the junk action value never gets initialized

a=chofmann for 1.7

Comment 54

[David :Bienvenu]

Comment on attachment 145363 [details] [diff] [review]
fixes a bug where the junk action value never gets initialized

I swear I wrote that code...

Comment 55

[Scott MacGregor]

Comment on attachment 145363 [details] [diff] [review]
fixes a bug where the junk action value never gets initialized

this patch has been checked in for 1.7

Comment 56

[David :Bienvenu]

backend support is checked in. I still need to write some front end code to
allow the user to set this up (though for now you can just set a hidden pref on
the server, serverFilterName, to the appropriate server side filter name
(Habeas, SpamAssassin, SpamCatcher, or SpamPal).

Comment 57

[Andrew Schultz]

this busted balsa tinderbox (gcc3.4):
/builds/tinderbox/SeaMonkey-gcc3.4/Linux_2.4.7-10_Depend/mozilla/mailnews/base/src/nsSpamSettings.cpp:458:
error: extra `;'

Comment 58

[Robert Accettura [:raccettura]]

Attachment: Fix Bustage

Comment 59

[David :Bienvenu]

Comment on attachment 146072 [details] [diff] [review]
Fix Bustage

I actually already checked in the same fix

Comment 60

[:aceman]

What about forget headers? Is this code immune to that? E.g. takes only the
later added headers? The spammers could insert their own headers saying
spam-status: 0.

And how are new spam filters added to this? My server has some new stuff, it
inserts a score into the header and even the cause for this score - what was
suspicious in the mail. Something like this:
X-Spam-Status: No, hits=0.1 required=5.0
X-Spam-Level: HTML_MAIL, NO_SENDER

Comment 61

[Martin Bagge / brother]

(In reply to comment #60)
> What about forget headers? Is this code immune to that? E.g. takes only the
> later added headers? The spammers could insert their own headers saying
> spam-status: 0.

Good point.
Some of my mails gets filtered two or more times and get different X-Spam
headers if not all marks it as spam then it might be a problem. 

Comment 62

[:aceman]

I meant forged, sorry for the typo.

Comment 63

[David :Bienvenu]

Attachment: fix for pop3 filter junk score action

Comment 64

[David :Bienvenu]

Comment on attachment 147413 [details] [diff] [review]
fix for pop3 filter junk score action

very safe fix, only affecting setting junk score with pop3 filters...

Comment 65

[Asa Dotzler [:asa]]

Comment on attachment 147413 [details] [diff] [review]
fix for pop3 filter junk score action

a=asa (on behalf of drivers) for checkin to 1.7

Comment 66

[Ulrich Grassberger]

*** Bug 243049 has been marked as a duplicate of this bug. ***

Comment 67

[Ulrich Grassberger]

My mail server sends x-junkmail-status headers, an example value is
"score=150/50, host=mx01.versatel.de". There also is a header X-Junkmail-Whitelist.

Comment 68

[Mike Kaply [:mkaply]]

I noticed that the sfd files were added to packages-os2 (and others), but I
build mailnews, these files never get exported into my dist.

It appears that the Makefile never gets hit?

Comment 69

[Giacomo Magnini]

(In reply to comment #68)
> I noticed that the sfd files were added to packages-os2 (and others), but I
> build mailnews, these files never get exported into my dist.
> 
> It appears that the Makefile never gets hit?

Makefile.in (in mailnews/base/search/src/) has only added SpamAssassin and SpamPal, leaving out the others two, even if the packager scripts try to install them (see attachment 144592 [details] [diff] [review]). Legal issues or simply forgot to add them?