471833 - Provide hooks to control calculation of bayesian filter traits

Status: RESOLVED FIXED

(MailNews Core :: Filters, enhancement)

Description

[Kent James (:rkent)]

With the recent generalization of the bayesian filters to classify arbitrary message traits, we need mechanisms to decide when to apply these calculations. Currently, we never apply the calculations to folders that make no sense for junk mail calculations (drafts, sent, trash). But if additional non-junk traits are enabled, then the calculations are done for most other folders. This has performance implications, including not only the calculation time, but also the decision to download the message body. Users and extensions need ways to enable and disable these calculations.

Since currently the only way to enable non-junk trait calculations is through an extension, the scope of this current bug will be to provide the backend support to control when non-junk traits are calculated, without any user interface to those methods.

Comment 1

[Kent James (:rkent)]

Attachment: in-progress, posted to demonstrate use of inheritedStringProperty

This patch is fully working (after application of the patch for bug 478360) but has not been fully checked by me. I mainly post it to show the use of inheritedFolderProperties in allowing hooks for precise control of the application of the bayesian filter processing.

Note I've also enabled bayesian filter calculation in rss and news here, controlled by an inheritedStringProperty in callFilterPlugins()

Comment 2

[Kent James (:rkent)]

Attachment: Add hack for RSS iframe tokenizing

This provides precise control of application of the bayes filter, including supporting RSS and nntp. RSS has an issue, if you set the option to link to the web page, then the summary information is stored as alternate content in an iframe, which is not passed through the serializer. I added a hack for that controlled by the preference. For nntp, we get tokens, but the headers are not broken out, so there are no header-specific tokens. It still works, though is less than ideal. That should probably be fixed, if possible, before fully enabling support for junk in nntp (which is not done by this patch.)

You need to apply the current patch for bug 476389 before applying this patch, because they bitrot each other.

Comment 3

[Mark Banner (:standard8)]

Comment on attachment 368556 [details] [diff] [review]
Add hack for RSS iframe tokenizing

>   if (serverType.EqualsLiteral("rss") ||
>+     (mFlags & nsMsgFolderFlags::Newsgroup && !filterNewsForJunk) ||
>      (mFlags & (nsMsgFolderFlags::Junk | nsMsgFolderFlags::Trash |
>              nsMsgFolderFlags::SentMail | nsMsgFolderFlags::Queue |
>              nsMsgFolderFlags::Drafts | nsMsgFolderFlags::Templates |
>              nsMsgFolderFlags::ImapPublic | nsMsgFolderFlags::ImapOtherUser)
>      && !(mFlags & nsMsgFolderFlags::Inbox)))

urgh. This is a nice confusing mix of styles (not caused by you). Can you adjust it please? I think this would be slightly cleaner:

if (serverType.EqualsLiteral("rss") ||
    (mFlags & nsMsgFolderFlags::Newsgroup && !filterNewsForJunk) ||
    (mFlags & (nsMsgFolderFlags::Junk | nsMsgFolderFlags::Trash |
               nsMsgFolderFlags::SentMail | nsMsgFolderFlags::Queue |
               nsMsgFolderFlags::Drafts | nsMsgFolderFlags::Templates |
               nsMsgFolderFlags::ImapPublic |
               nsMsgFolderFlags::ImapOtherUser) &&
     !(mFlags & nsMsgFolderFlags::Inbox)))

>+  /*
>+   * We'll use inherited folder properties for the junk trait to override the
>+   * standard server-based specification of junk processing. This provides a
>+   * hook for extensions to customize the application of junk filtering.
>+   * The general format for the property is "dobayes." + trait ID for the
>+   * trait, with values "true" or "false" to override defaults.
>+   */
>+  nsCAutoString junkEnableOverride;
>+  GetInheritedStringProperty("[e-mail]#junk", junkEnableOverride);

The comment is slightly confusing wrt the code. From what the comment is saying, I'd expect the code here to be checking all dobayes.* properties, but it isn't, that happens later on. Maybe rearrange how the comments work?

Comment 4

[Kent James (:rkent)]

Attachment: Fixed Standard8's issues

Comment 5

[Mark Banner (:standard8)]

Comment on attachment 369445 [details] [diff] [review]
Fixed Standard8's issues

That's better. There was one bit of bitrot where bug 471559 and this one were removing the same block/line, so I ignored that ;-)

I've checked this in: http://hg.mozilla.org/comm-central/rev/b9ddff83f2ed