add UI for honoring ISP spam headers

VERIFIED FIXED in Thunderbird1.1

Status

Thunderbird
Mail Window Front End
VERIFIED FIXED
13 years ago
10 years ago

People

(Reporter: Scott MacGregor, Assigned: Bienvenu)

Tracking

Trunk
Thunderbird1.1
x86
Windows XP

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(7 attachments, 1 obsolete attachment)

(Reporter)

Description

13 years ago
David's already done most of the back end work. Just need to add pref UI for it.

"all it needs is a UI
it just needs to iterate through the .sfd files and create a menu of them to let
the user pick one...
per server"
(Reporter)

Updated

13 years ago
Status: NEW → ASSIGNED
Target Milestone: --- → Thunderbird1.1
(Assignee)

Comment 1

13 years ago
We also need UI for specifying if we should trust positives, negatives, or both.
Here's some notes I wrote up on this:

For the incoming server, set the following prefs:

/"serverFilterName" and "//serverFilterTrustFlags"//

e.g.,

pref("mail.server.server2.serverFilterName", "habeas");
pref("///mail.server.server2.serverFilterTrustFlags", <trust flags>);

//

//The trust flags are used to specify if you trust positive results from the
server filter, and/or negative results
//

TRUST_POSITIVES = 1;
TRUST_NEGATIVES = 2;


so 1 if you trust only positives, 2 if only negatives, and 3 if you trust both.

Then, you need to create your server filter description file. You'll give it the
name <serverFilterName>.sfd, so in this example, habeas.sfd. Then you need to
define Yes and No filters, with names constructed from the server filter name,
so in this case, HabeasYes and HabeasNo. Here's an example:

version="8"
logging="yes"
name="HabeasNo"
enabled="yes"
type="1"
action="JunkScore"
actionValue="0"
condition="OR (\"x-SpamPal\",begins with,PASS)"
name="HabeasYes"
enabled="yes"
type="1"
action="JunkScore"
actionValue="100"
condition="OR (\"x-SpamPal\",begins with,SPAM)"


This would be the contents of your habeas.sfd file. You could create your own
filter file by using the filter UI to define the filter(s) that you want,
including the criteria, and the actions, and then go copy the filters text from
your normal msgFilterRules.dat into habeas.sfd. Then, if you put habeas.sfd into
the defaults/messenger sub-directory of your install dir, the next time you run,
assuming the serverFilterName pref is set correctly, it will copy the habeas.sfd
file into your profile directory (I think at the top-level, so multiple servers
can share .sfd files)
(Reporter)

Comment 2

13 years ago
Here's one thing we could do in the short term:

Have a check box in the junk mail dialog like Mail.app which says:

[X] Trust Junk Mail Headers set by your ISP

And we ship with a handful of the more common spam header files (spam assassin,
razor, etc.). Enabling this check box, makes us look at these header files.

Then later on, we can add an advanced button to the right of this widget which
takes you to a rich user interface for creating and modifying these files?
(Reporter)

Comment 3

13 years ago
Created attachment 183226 [details] [diff] [review]
the fix (work in progress)

This patch adds a checkbox and a menu list to the junk mail dialog for trusing
ISP headers. 

I also added a boolean flag to nsISpamSettings / mailnews.js for using a server
based junk filter. This pref drives the checkbox in the UI.

This patch doesn't include the .sfd files we would ship with by default.
(Reporter)

Comment 4

13 years ago
Created attachment 183227 [details]
screen shot
(Reporter)

Comment 5

13 years ago
Created attachment 183320 [details]
updated screen shot

I put the two list boxes in a grid so they are aligned together.
(Reporter)

Comment 6

13 years ago
Created attachment 183321 [details] [diff] [review]
updated patch

1) We now scan the user's profile directory for .sfd files first. After that,
we scan the defaults\messenger directory and add unique items from that
location to the list. This new wrinkle allows users to dynamically drop in .sfd
files into their profile directory that aren't shipped with thunderbird and
have them show up in the list.

2) I moved the .sfd files from mailnews\base\search\resources\content to
mailnews\base\search\src since they aren't really content files. I then got rid
of all the Makefiles that were hanging around (but not being used) in
mailnews\base\search\resources

3) I wrapped the trusted headers and trusted address book elements into a grid
so the two list boxes are in alignment together.
Attachment #183226 - Attachment is obsolete: true
(Reporter)

Comment 7

13 years ago
Comment on attachment 183321 [details] [diff] [review]
updated patch

When I check in, I'll only export the spam assassin sfd file for now until we
figure out which services we wish to support by default.

I'll also add packaging code to package up SpamAssassin.sfd
Attachment #183321 - Flags: superreview?(bienvenu)
(Reporter)

Comment 8

13 years ago
Created attachment 183323 [details] [diff] [review]
changes to package SpamAssassin.sfd
(Reporter)

Comment 9

13 years ago
David,
I also changed the condition for the spam assasin filter to look for ***SPAM***
in the subject so it now looks like:

"OR (\"X-Spam-Status\",begins with,Yes) OR (\"X-Spam-Flag\",begins with,YES) OR
(subject,begins with,***SPAM***)"

If we are worried about the overhead of downloading extra headers when this
filter is running, I think we can safely get rid of the X-Spam-Status clause in
the Yes search because spam assassin will always set X-Spam-Flag. X-Spam-Status
is typically used by folks that want to write more complicated filters that care
about the actual SA junk score in addition to what parts of the message caused
it to be marked as junk.

(Assignee)

Updated

13 years ago
Attachment #183321 - Flags: superreview?(bienvenu) → superreview+
(Reporter)

Updated

13 years ago
Attachment #183321 - Flags: approval-aviary1.1a1?

Comment 10

13 years ago
Comment on attachment 183321 [details] [diff] [review]
updated patch

a=chofmann
Attachment #183321 - Flags: approval-aviary1.1a1? → approval-aviary1.1a1+
(Reporter)

Updated

13 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 13 years ago
Resolution: --- → FIXED

Comment 11

13 years ago
(In reply to comment #9)
> I also changed the condition for the spam assasin filter to look for ***SPAM***
> in the subject

Is it going to be possible for end-users to change these rules, becuase for
example my ISP has the SpamAssassin code written to write [-SPAM-] rather than
***SPAM*** and when I use SpamAssassin localy, I change it to be [SPAM].
Sweet dude.

People hit this blog post of mine all the time for instructions on how to set
this up themselves:
http://robert.accettura.com/archives/2003/12/18/using-spamassassin-and-mozilla/

this is a big enhancement.

I must ask:
Is it possible to install .sfd files with an extension?

Comment 13

13 years ago
(In reply to comment #11)
> (In reply to comment #9)
> > I also changed the condition for the spam assasin filter to look for ***SPAM***
> > in the subject
> 
> Is it going to be possible for end-users to change these rules, becuase for
> example my ISP has the SpamAssassin code written to write [-SPAM-] rather than
> ***SPAM*** and when I use SpamAssassin localy, I change it to be [SPAM].

Ditto here for variable message text.

Also, what versions of Spam Assassin does this work with? We are on a slightly
older version. Thanks.

Comment 14

13 years ago
The other thing is whether or not filtering even works. I was still getting
messages mixed up with subjects, and missing messages, etc. with Thunderbird, so
had to shut off filtering completely. I do still have junk controls running,
though, and they catch a good 40-50% of the junk mail.

Comment 15

13 years ago
And our company prefixes the From: header with  ===SPAM===
But I've verified that there is a X-Spam-Status header starting with Yes.
(Novell Groupwise system 6.5)

Comment 16

13 years ago
FYI, Yahoo! Mail inserts the header X-YahooFilteredBulk: into emails they think
are spam. 

Comment 17

13 years ago
Created attachment 186314 [details] [diff] [review]
Filter file for honoring SpamPal headers

I've created a filter file for honoring SpamPal headers. I think this should be
shipped with Thunderbird as well as the SpamAssassin file becaus AFAIK SpamPal
is a very popular local spam filter.

Comment 18

13 years ago
I'm using Thunderbird 1.1a1 on WinXP, and I've switched on the Trust headers set
by SA option, but I'm seeing messages with X-Spam-Flag: NO get junked and
messages with X-Spam-Flag: YES fail to get junked. Is anyone else seeing this
problem?

Comment 19

13 years ago
Reopening bug per discussion with David Bienvenu. The trust headers option
appears to be ignored entirely. May have to do with temporary filters not being
enabled.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

13 years ago
Assignee: mscott → bienvenu
Status: REOPENED → NEW
(Assignee)

Comment 20

13 years ago
OK, the problem here is that user_pref("mailnews.customHeaders") isn't getting
the new spam headers, which makes us fail when parsing the new filters. Normally
these get added when you use the UI to define the custom headers in a filter,
but since we're not going through the UI for this, those new headers aren't
getting added to mailnews.customHeaders. I'll have to think about the solution here.
(Assignee)

Comment 21

13 years ago
Created attachment 188476 [details] [diff] [review]
proposed fix

we're going to allow arbitrary headers here, as long as the header is OK.
Attachment #188476 - Flags: superreview?(mscott)
(Assignee)

Comment 22

13 years ago
Ali, you can either add those two spam filter headers to your
"mailnews.customHeaders" by hand, or add them to the custom headers via the
customize button in the header dropdown. That should enable your filters to
start working.

But, these filters are not different from filters you define by hand, so I don't
think they're going to fix the problem you were trying to deal with in the first
place.
(Reporter)

Updated

13 years ago
Attachment #188476 - Flags: superreview?(mscott) → superreview+
(Assignee)

Updated

13 years ago
Attachment #188476 - Flags: approval-aviary1.1a2?

Updated

13 years ago
Attachment #188476 - Flags: approval-aviary1.1a2? → approval-aviary1.1a2+
(Assignee)

Comment 23

13 years ago
https://bugzilla.mozilla.org/attachment.cgi?id=188476 checked in
Status: NEW → RESOLVED
Last Resolved: 13 years ago13 years ago
Resolution: --- → FIXED
(Assignee)

Comment 24

13 years ago
Created attachment 188848 [details] [diff] [review]
add junk actions to trusted yes filter

as we talked about, add the appropriate junk actions to the trusted yes junk
filter.
Attachment #188848 - Flags: superreview?(mscott)
(Reporter)

Comment 25

13 years ago
Comment on attachment 188848 [details] [diff] [review]
add junk actions to trusted yes filter

this will be a life saver.
Attachment #188848 - Flags: superreview?(mscott) → superreview+
(Assignee)

Comment 26

13 years ago
Comment on attachment 188848 [details] [diff] [review]
add junk actions to trusted yes filter

only affects the built-in server-side junk filters, which are off by default
Attachment #188848 - Flags: approval-aviary1.1a2?

Updated

13 years ago
Attachment #188848 - Flags: approval-aviary1.1a2? → approval-aviary1.1a2+
(Assignee)

Comment 27

13 years ago
last fix checked in.

Comment 28

13 years ago
mail/base/content/junkMail.js

const kDefaultFilterServerName = "SpamAssasin";

should be SpamAssassin...
(Assignee)

Comment 29

13 years ago
*** Bug 298798 has been marked as a duplicate of this bug. ***

Updated

12 years ago
Blocks: 360488

Updated

11 years ago
No longer blocks: 360488
Status: RESOLVED → VERIFIED
Duplicate of this bug: 361738
You need to log in before you can comment on or make changes to this bug.