243430 - Improving the chi-square spam detection method

Status: NEW

(MailNews Core :: Filters, enhancement)

Description

[Scott MacGregor]

Bug #181534 improved our core junk mail algorithm by switching to a chi-squared
spam detection method. 

Gary Robinsion made the following comments on how this new improved algorithm
can now be improved on:

"Hello, for anyone interested in the chi-square spam detection method,

1) I've written an <a
href=http://www.garyrobinson.net/2004/04/improved_chi.html>article that 
introduces an improvement to the chi-square algorithm</a> and includes test
results. The 
improvement is statistically significant. 

2) Greg Louis of the Bogofilter project has also <a href=/bogofilter/esf.html
>tested the improvement mentioned above</a> and obtained positive results.

3) I've written <a href=http://www.garyrobinson.net/2004/05/why_chi.html>another
article about 
the theory behind the chi-square technique</a> and reasons for using it. 

4) I tried using the log of the number of messages as part of the chi-square
algorithm. It didn't 
help, at least not significantly. There is some possibility that there  was an
error in the test, but at 
this point I'm not encouraged that it would lead to a big improvement. I mention
it because the 
subject came up earlier in these comments."

I'm cc'ing some of the more active folks from Bug #181534 so we can continue
discussion here since I'm about to resolve 181534 as being fixed.

Comment 1

[Scott MacGregor]

We really haven't made any improvements to our spam filter since we first moved to the chi-square detection method back in May of 2004. I suspect Gary and the spam bayes folks have made more improvements since that time.

I'd like to re-energize some discussion or even better, patches to improve the filter from the technologies we used back in 2004.

Making a Thunderbird 2.0/3.0 blocker so we don't lose track of this bug.

Comment 2

[Scott MacGregor]

punting for tb2, we didn't get any traction on this.

Comment 3

[Tony Mechelynck [:tonymec]]

"mozilla1.8beta1" (i.e., Tb 1.5 beta1) has long gone. I suspect the Milestone of this bug would merit being changed either back to --- or forward to some Mxx on the way to Tb3 (Gecko 1.9).