437098 - Enable junk token limits

Status: RESOLVED FIXED

(MailNews Core :: Filters, enhancement)

Description

[Kent James (:rkent)]

We started a discussion in bug 228675 about whether to enable by default the pruning of junk tokens in Thunderbird, the capability of which is added in that bug. This bug continues that discussion, and proposes that the token limit preference be set by default, but to a relatively large value that will not affect the vast majority of users.

The relevant preference, defined in bug 228675, is mailnews.bayesian_spam_filter.junk_maxtokens

Comment 1

[Kent James (:rkent)]

Here's some rough numbers to use as a basis for setting this limit.

On a Windows 2000 system, I installed TB trunk and then loaded different sizes of training.dat. The training.dat came from different runs that I did using the TREC 2006 corpus. Here is TB memory used versus number of tokens. Note there is some noise in the memory data:

Token Counts   TB memory (MB) training.dat size (MB)
 58110            64.7               1.3
 92114            64.4               2.1
119972            70.2               3.1
390577            90.4               9.6
524622           106.2              16.2

A linear regression of this shows relationship between memory usage and total allowed token counts:

(TB Memory) = 58.2 MB + (8.83E-5)(Total Counts)

With these numbers, bayes filter memory usage is 10% of original TB memory at 66,000 counts, and is 50% of original TB memory at 330,000 counts. By comparison, the current training.dat I am running has 124,000 token counts. Also by comparison, the testing graph shown in http://wiki.mozilla.org/User:Rkentjames:Bug228675 using the TREC 2006 corpus shows an average false negative rate of about 12% using a limit of 77480 tokens.

I propose that the maximum counts be set at 200,000. At that value, the bayes filter should use a maximum of about 30% of the memory of the base TB program, and achieve an error rate of about 10%.

Comment 2

[Kent James (:rkent)]

Attachment: Token limit set to 200000

Just to keep this moving, here's the patch to set the limit at 200,000. I added Neil to the review mix to make sure to hear his opinion.

Comment 3

[Magnus Melin [:mkmelin]]

blocking‑thunderbird3+ as this could give great memory consumption improvements, according to the numbers in comment 1.

Comment 4

[[e-mail]]

Comment on attachment 324738 [details] [diff] [review]
Token limit set to 200000

I'd probably be quite happy with a smaller figure (e.g. 100,000) if you prefer.

Comment 5

[David :Bienvenu]

I'd also be happier with a default limit of 100,000

Comment 6

[Kent James (:rkent)]

Attachment: Token limit set to 100000

OK, here's the patch for 100,000

Comment 7

[David :Bienvenu]

Comment on attachment 324819 [details] [diff] [review]
Token limit set to 100000

thx, Kent

Comment 8

[Kent James (:rkent)]

Let's then talk about risk. When this hits, some people are going to see their training.dat shrunk. As a rough guess, filter error rate will initially change from about 8% to 11% for people affected by this - but with a significant boost in performance and better startup time. But really aggressive trainers who have huge training files might see performance go from, say 6% to 11% and really notice it. That is best case assuming it works as designed.

Do you think that we need to warn anyone about this, and if so how?

Comment 9

[Ronald Killmer]

For the upcoming Shredder 3.0a2 preview release it could be a release note item listed as a performance enhancing feature.  This has been needed for a long time.

Comment 10

[Mark Banner (:standard8)]

Checking in mailnews/mailnews.js;
/cvsroot/mozilla/mailnews/mailnews.js,v  <--  mailnews.js
new revision: 3.316; previous revision: 3.315
done

Comment 11

[Serge Gautherie (:sgautherie)]

This will affect SeaMonkey too, right ?
-> Core / Backend ?

Comment 12

[Kent James (:rkent)]

(In reply to comment #11)
> This will affect SeaMonkey too, right ?
> -> Core / Backend ?
> 
Yes you're right. Changing to Core / Mailnews:Filters

Comment 13

[Kent James (:rkent)]

Mkmelin, why did you reopen this?

Comment 14

[Magnus Melin [:mkmelin]]

Ah sorry, just meant to assign it back to you...

Comment 15

[Sander]

FWIW, I've _really_ noticed the decrease in junk filtering performance from this change; My old 4.1MB training.dat with threshold at 85 had me look at maybe two spam emails a month; now (with 1.4MB) I have three a day. (Out of a total of 10,000/month.)
Could it be worth investigating setting this number dynamically, based on total number of incoming junk mails per certain time period? Those with a lot of spam might be much more willing to accept the higher memory usage.

Comment 16

[Kent James (:rkent)]

Sander, if you want to adjust the limit, it is easily done by setting the integer preference mailnews.bayesian_spam_filter.junk_maxtokens. A UI to set the limit is also available in my JunQuilla extension at https://addons.mozilla.org/en-US/thunderbird/addon/9886. JunQuilla by default increases the limit to 300,000.

But you should also know that your experience is not typical, and by that I mean that the numbers that you are reporting are not realistic for the junk filter in Thunderbird. Typical performance of the internal bayes filter by itself would be a 10% false negative rate, which in your case would equate to around 30 spam emails per day showing. So I doubt if you are going to go from your current impossibly good results to your former really impossibly good results simply by increasing the junk token limit.

Comment 17

[Konstantin Svist]

I just installed JunQuilla and manually increased the limit to 300k -- but after training and then restarting Thunderbird (24.1.0, btw), the reported amount of tokens drops down to about 70k...
Does that mean this setting is ignored now?

Comment 18

[Kent James (:rkent)]

Konstantin: I am unaware of any changes that would disable the token limits. That being said, when you hit the limit and tokens are pruned, there is no simple formula to determine how many tokens would remain, because it depends on how many single-use tokens that you had. But I would expect that a remainder of 70K is more likely from a 300K starting place than from a 100K starting place.

In short, you have not presented any evidence that proves this is not working. Do some more training, monitoring the increase in token count, and see if the count increases beyond 100K.