Closed Bug 290237 Opened 19 years ago Closed 19 years ago

add UI for honoring ISP spam headers

Categories

(Thunderbird :: Mail Window Front End, defect)

x86
Windows XP
defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED
Thunderbird1.1

People

(Reporter: mscott, Assigned: Bienvenu)

References

Details

Attachments

(7 files, 1 obsolete file)

David's already done most of the back end work. Just need to add pref UI for it.

"all it needs is a UI
it just needs to iterate through the .sfd files and create a menu of them to let
the user pick one...
per server"
Status: NEW → ASSIGNED
Target Milestone: --- → Thunderbird1.1
We also need UI for specifying if we should trust positives, negatives, or both.
Here's some notes I wrote up on this:

For the incoming server, set the following prefs:

/"serverFilterName" and "//serverFilterTrustFlags"//

e.g.,

pref("mail.server.server2.serverFilterName", "habeas");
pref("///mail.server.server2.serverFilterTrustFlags", <trust flags>);

//

//The trust flags are used to specify if you trust positive results from the
server filter, and/or negative results
//

TRUST_POSITIVES = 1;
TRUST_NEGATIVES = 2;


so 1 if you trust only positives, 2 if only negatives, and 3 if you trust both.

Then, you need to create your server filter description file. You'll give it the
name <serverFilterName>.sfd, so in this example, habeas.sfd. Then you need to
define Yes and No filters, with names constructed from the server filter name,
so in this case, HabeasYes and HabeasNo. Here's an example:

version="8"
logging="yes"
name="HabeasNo"
enabled="yes"
type="1"
action="JunkScore"
actionValue="0"
condition="OR (\"x-SpamPal\",begins with,PASS)"
name="HabeasYes"
enabled="yes"
type="1"
action="JunkScore"
actionValue="100"
condition="OR (\"x-SpamPal\",begins with,SPAM)"


This would be the contents of your habeas.sfd file. You could create your own
filter file by using the filter UI to define the filter(s) that you want,
including the criteria, and the actions, and then go copy the filters text from
your normal msgFilterRules.dat into habeas.sfd. Then, if you put habeas.sfd into
the defaults/messenger sub-directory of your install dir, the next time you run,
assuming the serverFilterName pref is set correctly, it will copy the habeas.sfd
file into your profile directory (I think at the top-level, so multiple servers
can share .sfd files)
Here's one thing we could do in the short term:

Have a check box in the junk mail dialog like Mail.app which says:

[X] Trust Junk Mail Headers set by your ISP

And we ship with a handful of the more common spam header files (spam assassin,
razor, etc.). Enabling this check box, makes us look at these header files.

Then later on, we can add an advanced button to the right of this widget which
takes you to a rich user interface for creating and modifying these files?
Attached patch the fix (work in progress) (obsolete) — Splinter Review
This patch adds a checkbox and a menu list to the junk mail dialog for trusing
ISP headers. 

I also added a boolean flag to nsISpamSettings / mailnews.js for using a server
based junk filter. This pref drives the checkbox in the UI.

This patch doesn't include the .sfd files we would ship with by default.
Attached image screen shot
Attached image updated screen shot
I put the two list boxes in a grid so they are aligned together.
Attached patch updated patchSplinter Review
1) We now scan the user's profile directory for .sfd files first. After that,
we scan the defaults\messenger directory and add unique items from that
location to the list. This new wrinkle allows users to dynamically drop in .sfd
files into their profile directory that aren't shipped with thunderbird and
have them show up in the list.

2) I moved the .sfd files from mailnews\base\search\resources\content to
mailnews\base\search\src since they aren't really content files. I then got rid
of all the Makefiles that were hanging around (but not being used) in
mailnews\base\search\resources

3) I wrapped the trusted headers and trusted address book elements into a grid
so the two list boxes are in alignment together.
Attachment #183226 - Attachment is obsolete: true
Comment on attachment 183321 [details] [diff] [review]
updated patch

When I check in, I'll only export the spam assassin sfd file for now until we
figure out which services we wish to support by default.

I'll also add packaging code to package up SpamAssassin.sfd
Attachment #183321 - Flags: superreview?(bienvenu)
David,
I also changed the condition for the spam assasin filter to look for ***SPAM***
in the subject so it now looks like:

"OR (\"X-Spam-Status\",begins with,Yes) OR (\"X-Spam-Flag\",begins with,YES) OR
(subject,begins with,***SPAM***)"

If we are worried about the overhead of downloading extra headers when this
filter is running, I think we can safely get rid of the X-Spam-Status clause in
the Yes search because spam assassin will always set X-Spam-Flag. X-Spam-Status
is typically used by folks that want to write more complicated filters that care
about the actual SA junk score in addition to what parts of the message caused
it to be marked as junk.

Attachment #183321 - Flags: superreview?(bienvenu) → superreview+
Attachment #183321 - Flags: approval-aviary1.1a1?
Comment on attachment 183321 [details] [diff] [review]
updated patch

a=chofmann
Attachment #183321 - Flags: approval-aviary1.1a1? → approval-aviary1.1a1+
Status: ASSIGNED → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
(In reply to comment #9)
> I also changed the condition for the spam assasin filter to look for ***SPAM***
> in the subject

Is it going to be possible for end-users to change these rules, becuase for
example my ISP has the SpamAssassin code written to write [-SPAM-] rather than
***SPAM*** and when I use SpamAssassin localy, I change it to be [SPAM].
Sweet dude.

People hit this blog post of mine all the time for instructions on how to set
this up themselves:
http://robert.accettura.com/archives/2003/12/18/using-spamassassin-and-mozilla/

this is a big enhancement.

I must ask:
Is it possible to install .sfd files with an extension?
(In reply to comment #11)
> (In reply to comment #9)
> > I also changed the condition for the spam assasin filter to look for ***SPAM***
> > in the subject
> 
> Is it going to be possible for end-users to change these rules, becuase for
> example my ISP has the SpamAssassin code written to write [-SPAM-] rather than
> ***SPAM*** and when I use SpamAssassin localy, I change it to be [SPAM].

Ditto here for variable message text.

Also, what versions of Spam Assassin does this work with? We are on a slightly
older version. Thanks.
The other thing is whether or not filtering even works. I was still getting
messages mixed up with subjects, and missing messages, etc. with Thunderbird, so
had to shut off filtering completely. I do still have junk controls running,
though, and they catch a good 40-50% of the junk mail.
And our company prefixes the From: header with  ===SPAM===
But I've verified that there is a X-Spam-Status header starting with Yes.
(Novell Groupwise system 6.5)
FYI, Yahoo! Mail inserts the header X-YahooFilteredBulk: into emails they think
are spam. 
I've created a filter file for honoring SpamPal headers. I think this should be
shipped with Thunderbird as well as the SpamAssassin file becaus AFAIK SpamPal
is a very popular local spam filter.
I'm using Thunderbird 1.1a1 on WinXP, and I've switched on the Trust headers set
by SA option, but I'm seeing messages with X-Spam-Flag: NO get junked and
messages with X-Spam-Flag: YES fail to get junked. Is anyone else seeing this
problem?
Reopening bug per discussion with David Bienvenu. The trust headers option
appears to be ignored entirely. May have to do with temporary filters not being
enabled.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee: mscott → bienvenu
Status: REOPENED → NEW
OK, the problem here is that user_pref("mailnews.customHeaders") isn't getting
the new spam headers, which makes us fail when parsing the new filters. Normally
these get added when you use the UI to define the custom headers in a filter,
but since we're not going through the UI for this, those new headers aren't
getting added to mailnews.customHeaders. I'll have to think about the solution here.
Attached patch proposed fixSplinter Review
we're going to allow arbitrary headers here, as long as the header is OK.
Attachment #188476 - Flags: superreview?(mscott)
Ali, you can either add those two spam filter headers to your
"mailnews.customHeaders" by hand, or add them to the custom headers via the
customize button in the header dropdown. That should enable your filters to
start working.

But, these filters are not different from filters you define by hand, so I don't
think they're going to fix the problem you were trying to deal with in the first
place.
Attachment #188476 - Flags: superreview?(mscott) → superreview+
Attachment #188476 - Flags: approval-aviary1.1a2?
Attachment #188476 - Flags: approval-aviary1.1a2? → approval-aviary1.1a2+
https://bugzilla.mozilla.org/attachment.cgi?id=188476 checked in
Status: NEW → RESOLVED
Closed: 19 years ago19 years ago
Resolution: --- → FIXED
as we talked about, add the appropriate junk actions to the trusted yes junk
filter.
Attachment #188848 - Flags: superreview?(mscott)
Comment on attachment 188848 [details] [diff] [review]
add junk actions to trusted yes filter

this will be a life saver.
Attachment #188848 - Flags: superreview?(mscott) → superreview+
Comment on attachment 188848 [details] [diff] [review]
add junk actions to trusted yes filter

only affects the built-in server-side junk filters, which are off by default
Attachment #188848 - Flags: approval-aviary1.1a2?
Attachment #188848 - Flags: approval-aviary1.1a2? → approval-aviary1.1a2+
last fix checked in.
mail/base/content/junkMail.js

const kDefaultFilterServerName = "SpamAssasin";

should be SpamAssassin...
*** Bug 298798 has been marked as a duplicate of this bug. ***
Blocks: TB2SM
No longer blocks: TB2SM
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.