297852 - Enable Boolean expressions in Filter UI (nested/grouped/mixed/combined/complex filters having AND and OR conditions)

Status: NEW

(MailNews Core :: Filters, enhancement)

Description

[Howard Chu]

Since Bug 171073 has been fixed, the filter engine is now capable of processing
arbitrarily complex Boolean expressions. There are two problems though:
  1) the UIs have not been updated to allow users to take advantage of this
feature. The feature is only used internally, for managing virtual folders.
(There is already Bug 274192 requesting that the Search UI be updated, but
obviously the Filter UI must get a corresponding update.)
  2) the current Filter expression language (i.e., what gets saved in
msgFilterRules.dat) is ill-suited to actually representing complex filters. On
bug 171073 comment #13 I made a suggestion to borrow the operator syntax used
for the string representation of LDAP search filters. We could use anything
really, but standardizing on LDAP format may allow consolidation of this code
with Address Book searching, for example.

Comment 1

[Howard Chu]

(In reply to comment #0)
> Since Bug 171073 has been fixed, the filter engine is now capable

I use the word "now" loosely of course, as bug 171073 was fixed two and a half
years ago.

Anyway, my main concern is - is it acceptable to make a major change to the
syntax of filters, which will necessitate conversion of all existing
msgFilterRules.dat files to a new format?

We can mitigate some of the pain by incrementing the version number,
automatically converting the old format to the new format, and saving the new
format with a different filename. Then if someone needs to run an older version,
it will continue to work with the old filter definitions; they just wouldn't see
any new filters that were added with the new version.

Comment 2

[Matthias Versen [:Matti]]

*** Bug 297851 has been marked as a duplicate of this bug. ***

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

This strikes me as a good idea.  Some of the guts of the search logic in
mailnews/search/base is actually already shared between addressbooks and message
searching.  There are some rather strange artifacts in that code, including
separate but identical "and" operators for LDAP and local addressbooks.

As far as the filter files go, we've done exactly this sort of format upgrade in
the past.

Comment 4

[Wayne Mery (:wsmwk)]

*** Bug 301011 has been marked as a duplicate of this bug. ***

Comment 10

[Veritas]

thanks for marking this as a DUPE, but is anything actually being done? This bug is now 4 years old.  If the functionality exists, is there an explanation somewhere of the syntax?

Comment 11

[Kent James (:rkent)]

The functionality exists in the backend, but there is no user interface that exists that can use it (though the functionality for AND/OR filters is used for a variety of purposes in the backend). I am not aware of any ongoing efforts to fix that, though I'm sure contributions would be welcome.

I am also not aware either of any documentation of the syntax for that, other than the relevant .idl files which are quite sparse. However, I had to figure it out for myself in some work I did recently, and I fixed a serious bug there and also wrote a javascript unit test for it. So you might check out file test_searchBoolean.js as a demonstration of the syntax, or check bug 513277.

One workaround that is currently available is to write some simple javascript to do the actual test that you want, then implement that as a custom search term. I'll be releasing a TB3-compatible extension soon that implements a simple javascript interface for a search term, which also works in filters. Sorry I don't have more of that yet, but you can check http://mesquilla.com/2009/11/20/maybe-i-need-a-search-extension-searchaquilla/ where I mention that functionality briefly.

Comment 12

[:aceman]

What is the difference of this bug and bug 274192? They seem to be both about Search AND Filters now. What about differentiating them?

Comment 13

[Wayne Mery (:wsmwk)]

How do you see this as being filter-mgmt?

Bug 274192 should be about advanced search only.
And this about Filters only.

It seems there might be some rationale for having common code. But I don't think we should let that delay delivering one bug or the other.

Comment 14

[Howard Chu]

Eh? This bug is over 6 years old, I find any concern over "delay" at this point to be laughable.

Comment 15

[:aceman]

Wayne, I think you defined filter-mgmt as being about UI to do with creation and editing of filters. See your email of 22.10.2011.
I think comment 0 and comment 11 actually say this bug is requesting to add the UI for the existing backend features.
Is that not the case?

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to :aceman (away 27.-2.) from comment #15)
> Wayne, I think you defined filter-mgmt as being about UI to do with creation
> and editing of filters. See your email of 22.10.2011.
> I think comment 0 and comment 11 actually say this bug is requesting to add
> the UI for the existing backend features.
> Is that not the case?

I've posted in mozilla.dev.apps.thunderbird so that anyone interested in this subject can participate. http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/233a96f96259c988#

Comment 17

[John David Galt]

The newsgroup lives on mozilla's private server, so going there via google (as above) gives read-only access: anything I try to post disappears.

But adding a newsgroup account for news.mozilla.org, then subscribing to mozilla.dev.apps.thunderbird works.

[Just trying to help anyone else who runs into the same problem.]

Comment 21

[Axel Grude [:realRaven]]

Attachment: Nested Filter UI

Here is a suggestion for a User Interface that is fairly minimal as it is based on the existing Filter COnditions dialog. The idea is to allow nesting filters by allowing a new condition "Nested Filter applies to mail" (or is applicable, wording may vary). And then add a button to edit a nested filter or to select an existing filter.

Comment 22

[:aceman]

Axel, is the attachment meant for this bug? This one is about boolean expressions (AND, OR) between filter rules.

Nested (macro) rules and actions are in different bugs.

Can you explain?

Comment 23

[Axel Grude [:realRaven]]

(In reply to :aceman from comment #22)
> Axel, is the attachment meant for this bug? This one is about boolean
> expressions (AND, OR) between filter rules.
Well actually I thought this one was about a _UI_ for combining different boolean rules.

The paradigm of nested "filters" would be a simple way to display and generate complex conditions.

e.g.
A and (B or C)

here you would choose the model

Condition A and Z
As Z you would build a condition (B or C) - simply by clicking the Edit button. This would lead to a new (if you like anonymous) filter which has no action and will not be shown in the main filters list. It can display exactly the same interface the users are already used to so they can build the condition
"B or C". I can create a mock up for this "nested condition" to illustrate if you like.

Once they name this new condition and click Ok, they are returned to the original "nesting" filter. Of course to bring together the (already existing?) mechanism that stores AND / OR mixed filters and to transpose this into the UI would be the difficult bit.

My thoughts on this were to introduce the concept of building this filter "gently" without the user having to learn a completely new UI.

Alternatively, we could try to "indent" the nested conditions and change the operator from AND to OR and back on each nested level, but I think that is going to work out as a lot more difficult both to design and comprehend.

Comment 24

[:aceman]

But this looks like a new UI and would be probably very hard to add as it probably breaks many paradigms in the current filters.

The current filter editor already had a column to prepend AND or OR before each rule. But that was removed for now as the editing is not implemented. And there is also the question on how to edit "brackets" (grouping of the rules). Yes, your approach solves this part.

Comment 25

[Blake Winton (:bwinton) (:☕️)]

Comment on attachment 720376 [details]
Nested Filter UI

Wow.  I'm really conflicted on this one.

On the one hand, this seems like a fairly elegant hack given the current constraints, on the other hand, I'm not sure whether people will figure out that this is the way for them to do what they're trying to do.  When I think of "A and (B or C)", I don't think of the word "nesting", and I'm not sure whether we should accept the current constraints, or whether we should try to overcome them while we're making this change…

As some nits with the screenshot itself, the first two boxes should be one, and maybe just call it "Filter applies to mail".  (I'ld like something that made the whole thing into a sentence, if possible, like "Subject contains QuickPasswords"…  Hmm, how about "Message matches filter FromMailingListFilter"?)

Having the four buttons there does seem a little much.  Also, no-one will actually be typing in the name of another filter, so I think we could get away with a single button to let you choose a different filter, and have the "create new sub-filter" option on that dialog.

Finally, when we create a new sub-filter, we should set it to never run, and change the default action to either "Stop execution", or better than that, a new action: "return success to parent filter".

So, I guess, based on all of that, I'm going to say ui-r-, but if you fill out the whole process of creating subfilters a little more, and specify how to deal with all the edge cases, and tidy up the dialog a bit, that'll probably change to a ui-r+…  :)

Thanks,
Blake.

Comment 26

[Axel Grude [:realRaven]]

(In reply to Blake Winton (:bwinton) from comment #25)
> Comment on attachment 720376 [details]
> Nested Filter UI
> 
> Wow.  I'm really conflicted on this one.
> 
> 
> As some nits with the screenshot itself, the first two boxes should be one,

except it might be easier if users also had the option of 
[Subcondition][V]  [does not apply][V]

> and maybe just call it "Filter applies to mail".  (I'ld like something that
> made the whole thing into a sentence, if possible, like "Subject contains
> QuickPasswords"…  Hmm, how about "Message matches filter
> FromMailingListFilter"?)

first of all, thanks for reviewing. :)
thing is, whatever is included as nested / subfilter, is in actual fact not a full filter but just a filter condition. The proposal is just to reuse the (already known) filter details dialog to edit the included condition. So I believe the real trick will be to find the right (intuitive) labels for these things. So having a select for a different filter might indeed be superfluous.

> 
> Having the four buttons there does seem a little much.  Also, no-one will
> actually be typing in the name of another filter, so I think we could get
> away with a single button to let you choose a different filter, and have the
> "create new sub-filter" option on that dialog.

I believe when the button for the condition is clicked, the dialog should be shown again, the question is whether we allow giving the condition a name or should it just try and cram all condition elements in there so it becomes explicit? THis could become very long poentially, consider an example:

A and (B or C or (D and E))

that would show up as 2 lines:

     sender is John Williams
AND  subcondition is (subject is music or body contains "film music" or subcondition is (body doesn't contain guitar AND subject contains "Sky"))

so it might indeed be better for the user to be able assign a meaningful name to the subcondition such as 

     sender is John Williams
AND  subcondition is (JW the composer)

subcondition "JW the composer":
      subject is music 
   or body contains "film music" 
   or subcondition is (not related to Sky's guitar player)



> 
> Finally, when we create a new sub-filter, we should set it to never run, and
> change the default action to either "Stop execution", or better than that, a
> new action: "return success to parent filter".

Yes. I also believe that "sub-filter" should not actually show up in the filter list at all (because it is just a condition). Since it is stored in the original filter anyway and is not an entity on its own this shouldn't actually present a problem. The main rationale behind the hack is that we reuse the interface which our users already know how to use, and we do not actually reinvent a back-end for mode complicated filter conditions.

> 
> So, I guess, based on all of that, I'm going to say ui-r-, but if you fill
> out the whole process of creating subfilters a little more, and specify how
> to deal with all the edge cases, and tidy up the dialog a bit, that'll
> probably change to a ui-r+…  :)

yes I will do, once I get a bit of time. Especially, I need to make a screenshot of the "nested" filter / condition editor. I am almost in favor or re-instantiating another modal edit filter as child dialog, as this reflects the hierarchical relationship between the filter and its contained conditions. The hardest thing here is to put this into words without complicating it.

Axel

Comment 29

[Axel Grude [:realRaven]]

Attachment: multiple-Filters.png

Here is another approach on how the nested filters could be displayed. It doesn't address the question on how to add the start / finish of the nesting but shows how it could be displayed - adding 2 resizing columns at the left and right which will show the parenthesis and the operator (any or all). I added some color to make the "nesting" more obvious, but that might be technically to difficult as implementation. This is based on the desriptions in 
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIMsgSearchTerm#beginsGrouping and endsGrouping. I assume these attributes denote open and closed brackets (although I am not sure how you would store double brackets which close 2 groupings as in the example). Advantage of this approach is that it shows all information on one screen and is very close to the implementation of the back end (AFAIK), even though it might be a bit less intuitive. I did not mark my first UI as obsolete as I am still not sure whether it is not better than this one. 

An open bracket would necessarily always denote toggling the operator (and - or - and) and could be showing the term any / all as shown in the screen shot.

Comment 30

[Carl Ponder]

I'd be happy just to have one overriding AND condition with a sequence of OR conditions.
For example

AND: Does not contain "confirmation"
OR: from is Orbitz
    from is Travelocity
    from is La Quinta
    <etc.>
ACTION: Move to "travel" folder

This would move information on travel deals into the "travel" folder, which I would look at when I'm interested in browsing for travel deals.
I want confirmation messages to stay in my inbox, where I look for more urgent notices.
This is my fundamental desire, to separate information I would browse when I'm interested, versus notices that I want to see as soon as they come in. (Like "polling" versus "interrupt-driven").

Comment 33

[rwstahlhut]

For UI ideas, take a look at how OS X implements more complex boolean logic in the Finder window search (used for searches or to save search strategies).  The appearance is just like TB filters, but when you option-click the PLUS for another step, you get an indented sub-step that itself is just a recursive version of the level above.  The option-click is not obvious, which keeps newbies from getting confused (but probably hides it from a lot of non-newbies too). Anyway, the logic is AND/OR as complex as you'd like.  I think it would work well for TB.  It is intuitive to use and would not require the development of a new filter interface concept. The great OS X utility software Hazel does something very similar.  It works well there too.  FYI.

Comment 35

[Axel Grude [:realRaven]]

(In reply to rwstahlhut from comment #33)
> For UI ideas, take a look at how OS X implements more complex boolean logic
> in the Finder window search (used for searches or to save search
> strategies).  The appearance is just like TB filters, but when you
> option-click the PLUS for another step, you get an indented sub-step that
> itself is just a recursive version of the level above.  The option-click is
> not obvious, which keeps newbies from getting confused (but probably hides
> it from a lot of non-newbies too). Anyway, the logic is AND/OR as complex as
> you'd like.  I think it would work well for TB.  It is intuitive to use and
> would not require the development of a new filter interface concept. The
> great OS X utility software Hazel does something very similar.  It works
> well there too.  FYI.

Could you post a screenshot with an example? Not everybody owns a Mac (including me)

Comment 38

[grzegorz.szyszlo]

Wayne: 1175822 isn't duplication. This is approach of other side.
Logically from email message side this is the same. But from user side this is different.
My approach is, filters next to filters are in 'or' logic. But next filter isn't executed when first is completly true and takes any action with message. Let we name it vertical behavior.
Second, horizontal behavior idea, filter has its subfilters. subfilter executes only when master filter is true. This is like 'and' logic. master filter can group more subfilters. subfilters are in 'or' logic ech to other in the same level, like files in one flat directory.
This cause, I can prepare simpler common rules in one master filters, and additional detailed rules in subfilters. Then common rule is executed only once, saves cpu work.
Currently I must duplicate common rules inside subfilters, because subfilters are not implemented.

Do you stil think 1175822 is duplicate ? I think 1175822 is more complex behavior, and on UI side can better use currently implemented booleans in filtering.

Comment 39

[Axel Grude [:realRaven]]

(In reply to grzegorz.szyszlo from comment #38)
> Wayne: 1175822 isn't duplication. This is approach of other side.
> Logically from email message side this is the same. But from user side this
> is different.
> My approach is, filters next to filters are in 'or' logic. But next filter
> isn't executed when first is completly true and takes any action with
> message. 

I think this is only the case if the email is moved to a different folder, but I might be wrong? Outlook has an option "stop processing filters on email" (paraphasing), would that be helpful in Thunderbird? It does make sense not to have additional filter actions on an email that has been moved / archived to a different folder.

> Let we name it vertical behavior.
> Second, horizontal behavior idea, filter has its subfilters. subfilter
> executes only when master filter is true. This is like 'and' logic. master
> filter can group more subfilters. subfilters are in 'or' logic ech to other
> in the same level, like files in one flat directory.

I wonder, would it be helpful if you could add filters to other folders than the INbox? I was always wondering whether there was a practical application for "cascading" filters down by copying mail to some other "parent nodes" and then doing more detailed filtering from there, rather than having all filters in one long hard to manage list.

Comment 40

[:aceman]

(In reply to Axel Grude [:realRaven] from comment #39)
> I think this is only the case if the email is moved to a different folder,
> but I might be wrong? Outlook has an option "stop processing filters on
> email" (paraphasing), would that be helpful in Thunderbird? It does make
> sense not to have additional filter actions on an email that has been moved
> / archived to a different folder.

I think Move implies "stop filter execution" in TB too.

> I wonder, would it be helpful if you could add filters to other folders than
> the INbox? I was always wondering whether there was a practical application
> for "cascading" filters down by copying mail to some other "parent nodes"
> and then doing more detailed filtering from there, rather than having all
> filters in one long hard to manage list.

Yes, there is a bug requesting per folder filters (e.g. bug 294632).

Comment 41

[grzegorz.szyszlo]

(In reply to Axel Grude [:realRaven] from comment #39)
> (In reply to grzegorz.szyszlo from comment #38)
> > Wayne: 1175822 isn't duplication. This is approach of other side.
> > Logically from email message side this is the same. But from user side this
> > is different.
> > My approach is, filters next to filters are in 'or' logic. But next filter
> > isn't executed when first is completly true and takes any action with
> > message. 
> 
> I think this is only the case if the email is moved to a different folder,
> but I might be wrong? Outlook has an option "stop processing filters on
> email" (paraphasing), would that be helpful in Thunderbird? It does make
> sense not to have additional filter actions on an email that has been moved
> / archived to a different folder.

This is not my idea. You think about 'action' stage. I think about 'make decision' state.
It is no difference between moveint email from inbox to other folder, and marking email with star
or doing whatever. Simple example. I want make something with mail incomming from mailing group [e-mail] sent by user [e-mail]. Then I want do somethink different from the same mailing group, bot sent by user [e-mail]. What I do usually? I make two filters. First with rule replyto: [e-mail] and sender: [e-mail] and any action. Second filter rule is replyto: [e-mail] and sender: [e-mail] and any action.
You see, replyto: [e-mail] is common.
I think making parent filter0 with only replyto: [e-mail] without any action, or with any default action. Then making two slave/sub filters with only filter1 [e-mail] and any action and  filter2 [e-mail] and any action.
Then if filter0 doesnt match, then filter1&filter2 are not run.
Consider idea, when filter0, filter1&filter2 are more complicated. My approach cause simplification for thunderbird user. Why? because filters are naturally grouped by this function. it isn't flat list. secondly, user on filters configure only important configuration on its level.

> > Let we name it vertical behavior.
> > Second, horizontal behavior idea, filter has its subfilters. subfilter
> > executes only when master filter is true. This is like 'and' logic. master
> > filter can group more subfilters. subfilters are in 'or' logic ech to other
> > in the same level, like files in one flat directory.
> 
> I wonder, would it be helpful if you could add filters to other folders than
> the INbox? I was always wondering whether there was a practical application
> for "cascading" filters down by copying mail to some other "parent nodes"
> and then doing more detailed filtering from there, rather than having all
> filters in one long hard to manage list.

No. no like that. I investigate only the case when we filter out inbox only.

Comment 42

[Axel Grude [:realRaven]]

I had a closer look at the filter parser, and apparently it doesn't currently support parsing additional ( and ) for setting the beginsGrouping and endsGrouping attributes:

http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgFilterList.cpp#727

Would writing a patch for support of this be in scope for this bug? 

Example / Test Case:
(AND (subject,contains,FS Notification) 
  OR (\"list-id\",contains,test)) 
 AND (from,contains,fastspring.com)

simplified: (A OR B) AND C

expected Parsing result:

A.booleanAnd = true     // is this ignored?
A.beginsGrouping = true
A.endsGrouping = false

B.booleanAnd = false
B.beginsGrouping = false
B.endsGrouping = true

C.booleanAnd = true
C.beginsGrouping = false
C.endsGrouping = false

Rules for writing a condition: 
beginsGrouping => "(" before booleanAnd
endsGrouping => ")" at end of a term - which is after the condition list)

(Parsing might be easier if we use different parentheses, such as { } or [ ] )

Comment 43

[Kent James (:rkent)]

(In reply to Axel Grude [:realRaven] from comment #42)
> I had a closer look at the filter parser, and apparently it doesn't
> currently support parsing additional ( and ) for setting the beginsGrouping
> and endsGrouping attributes:
> 
> http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/
> nsMsgFilterList.cpp#727
> 
> Would writing a patch for support of this be in scope for this bug? 
> 

Yes.

The tricky part would be making it backwards compatible. Perhaps there is some way to add these additional terms that would allow the old format to be used? I have not really thought that through much, not sure if that is a Good Idea or not.

Comment 45

[Joe Pollock]

I've wanted this in Thunderbird for many years. While the base concept is pretty simple and obvious, apparently, the implementation is not.

Now that Thunderbird is striking out on its own, maybe this could be factored in when designing the new improved version.

Comment 48

[Daniel Santos]

I don't know the code for this project, but from a data representation standpoint, any complex algebraic expression can be stored in a simple linked list/array/vector using polish notation consisting of only the conditions themselves, the binary operators OR, AND, XOR and a unary NOT operator.

Example2:
    cond1 AND (cond2 OR cond3)

Becomes:
    AND, cond1, OR, cond2, cond3

Or using the example above:
    ((subject,contains,FS Notification) OR (\"list-id\",contains,test))
     AND (from,contains,fastspring.com)

Becomes:
    AND,
    (from,contains,fastspring.com),
    OR,
    (subject,contains,FS Notification),
    (\"list-id\",contains,test)

The filter conditions become the operands to the operators and these lists are easily processed by each object having an "eval" function that takes a list iterator by reference and returns its result.  The eval is then called recursively until all operators and operands are evaluated.  C++ example:

    /* operator or operand */
    template<typename T, typename O> class Oper {
        typedef std::vector<Oper<T, O> > vector_type;
        typedef vector_type::iterator iterator;
        
        virtual T eval(iterator &i, const iterator &end, const O &obj) = 0;
    };

Every call to eval increments the iterator (throwing or asserting if it reaches one-past-the-end).  Binary operators will do something like a = iter->eval(); b = iter->eval(); and then return their result of a OP b.  Unary operators behave similarly.  At the root of the operation, you just call eval() on the head of the list and verify that the the resulting iterator is the last valid element.

    Oper<bool, Message>::vector_type my_filter = something;
    Oper<bool, Message>::iterator i = my_filter.begin();
    bool result = (*i)->eval(i, my_filter.end());
    assert(i == my_filter.end() - 1);

AND might look like this:
    template<typename T> class OperandAnd : public Oper<T> {
        T eval(iterator &i, const iterator end, const O &obj)
        {
            ++i;
            assert(i != end);
            return (*i)->eval(i, end, obj) && (*i)->eval(i, end, obj);
        }
    };

Of course, you should be aware of C++ operator precedence and evaluation order here.

Comment 49

[Howard Chu]

Daniel - that's all great, but please read the ticket you're responding to.

The filter engine already implements this stuff. LDAP filter syntax is already in Polish notation.
The only thing missing is a UI to allow users to use the full capability of the filter engine.

That's what this ticket is about.

Comment 50

[Daniel Santos]

HAH!  "Since Bug 171073 has been fixed, the filter engine is now capable of processing
arbitrarily complex Boolean expressions"  You mean I should READ? :)

I guess I'll have to live with my 80-odd email filters for now.  Sorry about that.

Comment 51

[Ulf Zibis]

I like to suggest to apply this issue compatible to Sieve: https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)#Use

Comment 52

[Ulf Zibis]

See also bug 79525

Comment 54

[travneff]

Is the current functionality enough to resolve cases like bug 1465076 ?

Comment 55

[Howard Chu]

(In reply to travneff from comment #54)

Is the current functionality enough to resolve cases like bug 1465076 ?

Yes. All the required functionality in the filter engine is present. The only thing lacking is the UI.

Comment 56

[travneff]

(In reply to Howard Chu from comment #55)

Yes. All the required functionality in the filter engine is present. The only thing lacking is the UI.

I'm missing some point. Can I build such a filter with single rule in existing UI?
I mean, seems like everything is present there: I set "sender" -> "contains" -> "[e-mail] OR [e-mail]"
Tried to play with various separators with no success.
Do I just missing the right syntax? Or it shall not work this way at all?

Comment 57

[Howard Chu]

(In reply to travneff from comment #56)

(In reply to Howard Chu from comment #55)

Yes. All the required functionality in the filter engine is present. The only thing lacking is the UI.

Do I just missing the right syntax? Or it shall not work this way at all?

No. Nothing in the current UI will do what you want. That's the entire point of this bug report.

Comment 58

[Axel Grude]

I still think nesting filters (adding filter condition "filter X applies" and allowing "no action" as an action for "sub-conditions") would be the cheapest and most intuitive way of supporting this functionality. And it wouldn't really require any new UI to be built.

Comment 59

[Steve Kelem]

Sigh... this UI bug has been around for 8 years, and we still don't have any functionality. While waiting for perfection, none of the users have access to the elegant filter logic that's in the underlying filter engine. Axel Grude's suggestions look promising. Can we get something usable? This year?

Comment 60

[Ulf Zibis]

I like to suggest to apply this issue compatible to Sieve: https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)#Use
See also bug 79525

Comment 61

[Axel Grude]

(In reply to Ulf Zibis from comment #60)

I like to suggest to apply this issue compatible to Sieve: https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)#Use
See also bug 79525

If there is a public parser for Sieve that might be a nice addition for FiltaQuilla too... if you can find a developer who wants to devote time to this.

Comment 62

[Thomas D. (:thomas8)]

Hi Alex, FYI - 19 duplicates. We have the backend, but missing the UI.
This would be a high-value addition to the filter ecosystem.
Perhaps we could at least try something like bug 1465076 so that users don't have to add 100 filter rows just for filtering "sender contains (s1 or s2 or s3 .... or s100).

Comment 63

[Thilo [:ThiloteE]]

Could we put this at least on the Thunderbird roadmap for filters?

I know it has been 17(!) years, but this enhancement here (bug 297852) is the single most useful addition to the filter eco-system apart from what the addons filtaQuilla and quickfilters can provide.

Current bugs on the roadmap (https://thunderbird.topicbox.com/groups/planning/Tf4da52fcc100a6d2):

1. Rewrite of Filtering into javascript (bug 1574136).
2. Support to filesystem actions (bug 1664171)

Other notable additions to the roadmap would be:

• the adoption of a new transfer protocol that goes beyond what pop3 and imap can provide as those determine many or at least some of the filter bugs (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=770137#c18, https://bugzilla.mozilla.org/show_bug.cgi?id=1215933#c58). Examples for other protocols (that i know of) would be sieve or the newer jmap, but i don't know if those are any good.
• I assume getting rid of existing bugs is not something one would want to write into a roadmap, although i would understand! xD
• Global message filter option to use a filter shared across accounts (bug 34973)
• ...

The good thing about Bug 297852

• Solutions exist. See comments above.
• The goal and the proposed means are not really controversial. Unlike some other long term enhancements, there seems to exist consensus that boolean expressions would indeed be very useful and something intrinsically good. At least nobody has voiced anything against the goal in itself yet.

As of now more fine grained filters are basically impossible to design, because a combination of bug 770137, bug 1320676 and Bug 1215933. These three bugs combined (+ the limitation to be able to apply filters only to the "income" folder) basically prevent most of the more elaborate fine tuned kind of filtering you might think about coming up with the current user interface.

There has rarely anybody really been adding/editing the filter code.
At least that is the impression i got by reading through all these bugs related to filters. Maybe putting this enhancement on the roadmap could give it some prominence and attract helpers/developers?

Solved bugs/enhancements related to filters in
2021: 6
2020: 8
2019: 8
2018: 3
2017: 4
2016: 4
2015: 4
2014: 3
2013: 1
2012: 2

Total open Bugs/enhancements to fix in 2021: 188
(and this number does NOT include stuff posted in mail news core)

Comment 65

[Rachel Martin]

Attachment: Mac-search.png

(In reply to rwstahlhut from comment #33)

For UI ideas, take a look at how OS X implements more complex boolean logic
in the Finder window search (used for searches or to save search
strategies). The appearance is just like TB filters, but when you
option-click the PLUS for another step, you get an indented sub-step that
itself is just a recursive version of the level above.

Here you go. The search is: Name starts with A or B and ends with X or Y. This is the standard "grouping" with is supported by the TB backend already. Comment #11 points to bug 513277 where this test was added that shows the function:
https://searchfox.org/comm-central/rev/4dedf3bcc53db8db86356fbaa9deb9f6a5845a45/mailnews/search/test/unit/test_searchBoolean.js#151

Using "option +" on Mac to get another group is not so intuitive, that's why the same effect is achieved with a "Group" button in BMO's advanced search:
https://bugzilla.mozilla.org/query.cgi?format=advanced - Scroll down to "Custom Search".
As can be seen in the referenced test and the related IDL, grouping is already implemented in the backend:
https://searchfox.org/comm-central/rev/4dedf3bcc53db8db86356fbaa9deb9f6a5845a45/mailnews/search/public/nsIMsgSearchTerm.idl#33-34

Apart from unknown backend bugs yet to be discovered, implementing this comes don't to putting a frontend onto the existing backend. The UI option is also pretty clear since Mac Finder search, BMO custom search and the grouping in the TB backend are essentially all the same.

Comment 66

[Axel Grude]

I got another 5 FiltaQuilla / quickFilters users requesting this feature within the last 2 months. It is still highly desired. Is there a possibility that somebody from Core could look at this as it is probably the single most useful improvement that would be possible in the filter area. even if we just added some checkboxes for the three flags (beginsGrouping, endsGrouping, !booleanAnd). The checkbox could do the inverse of what the global (or last outer) any / all flag does.

The UI could also hide these controls by default and we could have a flag or advanced setting to reveal this UI so that we won't get a support nightmare...

Comment 67

[Rachel Martin]

Attachment: complex filters.png

Analogue to the Mac search (comment #65), we're implementing this now, at time of writing, still WIP:
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/297852-grouping-search-term.patch

Comment 68

[Rachel Martin]

Turns out that those who said that the backend supported complex expressions were wrong. At least they are not serialised out or parsed back in, for example to msgFilterRules.dat and elsewhere ☹️:
https://searchfox.org/comm-central/rev/391d7cce6c4d581892af7d2a3f5c30a107d4a572/mailnews/base/src/VirtualFolderWrapper.jsm#183-197
https://searchfox.org/comm-central/rev/391d7cce6c4d581892af7d2a3f5c30a107d4a572/mailnews/base/src/nsMsgUtils.cpp#1596-1622
https://searchfox.org/comm-central/rev/391d7cce6c4d581892af7d2a3f5c30a107d4a572/mailnews/search/src/nsMsgFilterList.cpp#734

Plus there is no documentation how nsIMsgSearchTerm is meant to be used to create complex expressions apart from this example:
https://searchfox.org/comm-central/rev/391d7cce6c4d581892af7d2a3f5c30a107d4a572/mailnews/search/test/unit/test_searchBoolean.js#184
Apparently nsIMsgSearchTerm is used to flatten an expression tree into a list while moving the operators (nodes) into the leaves. This is the entire "documentation" we could find:
https://searchfox.org/comm-central/rev/391d7cce6c4d581892af7d2a3f5c30a107d4a572/mailnews/search/src/nsMsgLocalSearch.cpp#343-345
Summary for the backend: While complex terms with groups can be turned back into an expression tree (ConstructExpressionTree), there is no support for serialising/parsing those complex terms.

Comment 69

[Howard Chu]

(In reply to Rachel Martin from comment #68)

Turns out that those who said that the backend supported complex expressions were wrong. At least they are not serialised out or parsed back in, for example to msgFilterRules.dat and elsewhere ☹️:

As stated in the initial bug description:
https://bugzilla.mozilla.org/show_bug.cgi?id=297852#c0

2. the current Filter expression language (i.e., what gets saved in
   msgFilterRules.dat) is ill-suited to actually representing complex filters.

I still believe adopting LDAP filter syntax would be the best way forward, but I never worked out what the migration would look like.

Comment 70

[Axel Grude]

2. the current Filter expression language (i.e., what gets saved in
   msgFilterRules.dat) is ill-suited to actually representing complex filters.

I still believe adopting LDAP filter syntax would be the best way forward, but I never worked out what the migration would look like.

That whole msgFilterRules.dat format is indeed very restrictive. I have a backup / restore functionality and it's hard to merge filters back when restoring, as there is no UID, last edited date or anything else but the filter name to deduct filter identity. I would love additional attributes that could be added by the JavaScript layer (and later through a mx API). I created my own JSON based format in quickFilters for the backup / restore functionality, which is a lot easier to read / edit and extend.

Comment 71

[Rachel Martin]

(In reply to Howard Chu from comment #69)

I still believe adopting LDAP filter syntax would be the best way forward, but I never worked out what the migration would look like.

Maybe it's not a good idea to attempt too much at once. We're now storing the beginsGrouping/endsGrouping as [ ] instead of ( ) in the file. The link in comment #67 now has the patch of the fully working solution.

Comment 72

[Richard Marti (:Paenglab)]

Attachment: Bug 297852 - Allow grouping in search term UI. r=#thunderbird-reviewers

Port of https://github.com/Betterbird/thunderbird-patches/issues/35

This also does a general overhaul of searchTerm.js:

• Removal of hideMatchAllItem() and updateRemoveRowButton()
• Removal of over-engineered global gTotalSearchTerms, badly tracking the length of the search term array
• Removal of gLoading, as a consequence of the gTotalSearchTerms removal
• Removal of gSearchRemovedTerms, we saw no use in tracking removed searchterms
• Moved badly conceived treecols for flex values out of the richlistbox, so the child count matches the richlist items
• Fixed clumsiness in removeSearchRow(): Just remove the row at the given index instead of searching for in the DOM
• Adding an empty search term for new rules, user might want to start off with a group instead
• Mitigation of https://bugzilla.mozilla.org/show_bug.cgi?id=1765077 (missing highlight, focus).
  Further clean-up: Removed nsIMsgFilter.AddTerm() and renamed nsIMsgFilter.GetTerm().

Comment 73

[DeaDSouL]

(In reply to Rachel Martin from comment #67)

Created attachment 9272624 [details]
complex filters.png

Analogue to the Mac search (comment #65), we're implementing this now, at time of writing, still WIP:
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/297852-grouping-search-term.patch

This looks so much better.
I came to request the same exact feature.

Because if for example I wanted to create this filter rule:
( (from,is,abc@mailcom) OR (from,is,xyz@mailcom) ) AND ( (subject,contains,A) OR (subject,contains,B) )

I need to create four rules in the current version, as:
(from,is,abc@mailcom) AND (subject,contains,A)
(from,is,abc@mailcom) AND (subject,contains,B)
(from,is,xyz@mailcom) AND (subject,contains,A)
(from,is,xyz@mailcom) AND (subject,contains,B)

I have more than 300 filter rules, just because the thunderbird doesn't support the complex filter rules yet.

Please make thunderbird supports it, even if we have to manually edit the "msgFilterRules.dat" file

Thanks

Comment 74

[Ulf Zibis]

(In reply to DeaDSouL from comment #73)

Please make thunderbird supports it, even if we have to manually edit the "msgFilterRules.dat" file

See also: Bug 944944 Bug 294632 Bug 1756327 Bug 1755819 Bug 1763718

Comment 75

[Rachel Martin]

Just for the record, we're constantly making updates to our solution at https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/297852-grouping-search-term.patch. The latest version doesn't apply cleanly to C-C any more due to dependencies with bug 683809.

Comment 77

[Ulf Zibis]

I like to suggest to apply this issue compatible to Sieve: https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)#Use
So please add bug 79525 to the "see also" list (unfortunately I'm not granted to do that).

Comment 78

[Thilo [:ThiloteE]]

I have been using Betterbird for roughly half a year now with the new complex filters UI and it works very well.
The development process and code changes can be traced by having a look at following bugs:

• https://github.com/Betterbird/thunderbird-patches/issues/35 (fixed)
• https://github.com/Betterbird/thunderbird-patches/issues/60 (fixed)
• https://github.com/Betterbird/thunderbird-patches/issues/62 (fixed)
• https://github.com/Betterbird/thunderbird-patches/issues/63 (not fixed, but users are notified, if situation is triggered; rare edge case)

Positive observations:

• I personally have have filter rules that tag, move and copy messages from inbox to sub-folders and local folders and it just works.
• I was able to reduce the amount of filters I have substantially. Instead I now have more rules within a single filter
• I was able to conveniently use complex filters e.g.: [If Tag is X AND (message body contains M OR N OR O OR P OR Q)], then do Z.
• I was able to work around the "impossible to automatically filter mail outside of inbox" problem using tags (and the filtaquilla addon).
• Jörg and his team also adapted the search UI (CTRL+Shift+F). It is possible to make use of "complex searches"

Negative observations:

• issue 63
• Speed. Filtering sometimes can take a little while. My new computer has new relatively up to date hardware (From 2020/2021) including a SSD and with many complex filters and many e-mails arriving in the inbox at the same time (e.g. after starting Betterbird), it can take a few seconds for messages to be moved into sub-folders. On the plus side though: As long as users refrain from adding complex filters, I would assume the speed to be comparable to original Thunderbird. Edit: Speed is great since a few months now. I believe my speed issues were caused by two conflicting filter rules and move operations. Unfortunately I am not able to reproduce what happened. Betterbird 102.14.0-bb39 (64-bit) is all good for now.

Recommendations:

1. Porting the Betterbird complex filter UI to Thunderbird is recommended. (My personal opinion as user (I don't know about code quality!))
2. Test filter performance. I read somewhere that the old filters engine was tested by filtering a million messages or something. :D Maybe something similar can be done here and compare the speed of the old with the new implementation.

Comment 79

[Ulf Zibis]

(In reply to Thilo.Ertel from comment #78)

I have been using Betterbird for roughly half a year now with the new complex filters UI and it works very well.

Great work.

Is this feature compatible with bug 79525 and https://addons.thunderbird.net/de/thunderbird/addon/sieve/?src=ss ???

Comment 80

[Thilo [:ThiloteE]]

@Ulf Zibis
You can try Betterbird with the new UI yourself and test if it is compatible with Sieve.
I have never used Sieve before. Tried to set it up once, but was not able to connect. I believe my mail-provider does not support Sieve.

Comment 81

[Richard Marti (:Paenglab)]

Alex, is this something we want?

Comment 82

[Alessandro Castellani [:aleca]]

Yes, it's definitely something we want.
I looked and betterbird's solution and both code and UI are not the best.
Let's talk about this a bit in our next front-end meeting and find a way forward.

Comment 84

[Axel Grude]

Just to ping and leave some relevant detail on this bug as there is currently code freeze in 128, and I am regularly getting users of my Add-ons quickFilters and FiltaQuilla requesting it.

Is this something we could attack for the next ESR? I am hoping that the Thunderbird code base will go on Github soon / this year and then I am open to contribute some patches around filters again, wouldn't mind being part of the team that implements this one; if you have a filters guy I can do requirements and QA as well if you like. PS: I have ported Thunderbird Filter Patches to Postbox before, but I haven't touched much C++ code in the last years, but getting better with ECMAScript, and am interested in getting started with Rust.

We might attack filter storage first, they really need some additional attributes (such as GUID, last edited, lasted used date, and custom attributes that can be added so we can expose this to Add-on developers later - in quickFilt4ers I have implemented a backup / restore feature but it has to match filters based on their names which is not idea) and would be much better stored as JSON rather than the current mess.

I think the Bb UI is somewhat useful, after some layout improvements that I requested:

hope the screenshot is visible (from Github)

Comment 85

[Howard Chu]

As I wrote originally, I would still recommend LDAP filter syntax https://bugzilla.mozilla.org/show_bug.cgi?id=297852#c0 since it's lightweight and we already have code in libldap that correctly and efficiently parses such things. JSON is much bulkier and you'd still need to invent a syntax to represent the Boolean operators themselves.

Comment 86

[Ulf Zibis]

(In reply to Howard Chu from comment #85)

As I wrote originally, I would still recommend LDAP filter syntax https://bugzilla.mozilla.org/show_bug.cgi?id=297852#c0 ...

What is the advantage of LDAP over Sieve [1]?
Many IMAP servers already have implemented it.
Does LDAP support server-side filtering?

[1] https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)

Comment 87

[Ulf Zibis]

(In reply to Axel Grude from comment #84)

Just to ping and leave some relevant detail on this bug as there is currently code freeze in 128, and I am regularly getting users of my Add-ons quickFilters and FiltaQuilla requesting it.

Does QuickFilters or FilterQuilla support server-side filtering, e.g. by the Sieve protocol?

Comment 88

[Axel Grude]

(In reply to Ulf Zibis from comment #87)

Does QuickFilters or FilterQuilla support server-side filtering, e.g. by the Sieve protocol?

No:

• FiltaQuilla adds other search terms (such as regular expressions) and actions (play sound, run external program, detach attachments...) - one of the problems here is that filter stuff is still not async, which makes using API methods difficult. I am hoping that the Tb Core team will address the async problem within the ESR128 cycle, then we can make filter APIs for mail extensions.

• quickFilters is a Management tool that supports creating (and auto-filling) filters (outlook style) through an assistant, so it builds on whatever Thunderbird has to offer. It also adds a toolbar to the message filters and some search functionality (find filters that apply to a folder)

Comment 89

[Howard Chu]

(In reply to Ulf Zibis from comment #86)

(In reply to Howard Chu from comment #85)

As I wrote originally, I would still recommend LDAP filter syntax https://bugzilla.mozilla.org/show_bug.cgi?id=297852#c0 ...

What is the advantage of LDAP over Sieve [1]?
Many IMAP servers already have implemented it.
Does LDAP support server-side filtering?

[1] https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)

Yes, LDAP supports server-side filtering, that is it's most heavily used function when serving an Address Book role.

It's important to be clear about what the aim is here. Boolean expressions in Filters is only targeting criteria for message selection, it has nothing to do with specifying what actions to take on selected messages. Sieve appears to be a larger scope, as it includes actions. I don't think anyone is proposing to replace MailNews' entire filter engine with Sieve.