Enable Boolean expressions in Filter UI (nested/grouped/mixed/combined/complex filters having AND and OR conditions)

NEW
Unassigned

Status

--
enhancement
14 years ago
2 months ago

People

(Reporter: hyc, Unassigned)

Tracking

(Blocks: 1 bug, {uiwanted})

Trunk
uiwanted
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [filter-mgmt][gs], URL)

Attachments

(2 attachments)

(Reporter)

Description

14 years ago
Since Bug 171073 has been fixed, the filter engine is now capable of processing
arbitrarily complex Boolean expressions. There are two problems though:
  1) the UIs have not been updated to allow users to take advantage of this
feature. The feature is only used internally, for managing virtual folders.
(There is already Bug 274192 requesting that the Search UI be updated, but
obviously the Filter UI must get a corresponding update.)
  2) the current Filter expression language (i.e., what gets saved in
msgFilterRules.dat) is ill-suited to actually representing complex filters. On
bug 171073 comment #13 I made a suggestion to borrow the operator syntax used
for the string representation of LDAP search filters. We could use anything
really, but standardizing on LDAP format may allow consolidation of this code
with Address Book searching, for example.
(Reporter)

Comment 1

14 years ago
(In reply to comment #0)
> Since Bug 171073 has been fixed, the filter engine is now capable

I use the word "now" loosely of course, as bug 171073 was fixed two and a half
years ago.

Anyway, my main concern is - is it acceptable to make a major change to the
syntax of filters, which will necessitate conversion of all existing
msgFilterRules.dat files to a new format?

We can mitigate some of the pain by incrementing the version number,
automatically converting the old format to the new format, and saving the new
format with a different filename. Then if someone needs to run an older version,
it will continue to work with the old filter definitions; they just wouldn't see
any new filters that were added with the new version.
*** Bug 297851 has been marked as a duplicate of this bug. ***
This strikes me as a good idea.  Some of the guts of the search logic in
mailnews/search/base is actually already shared between addressbooks and message
searching.  There are some rather strange artifacts in that code, including
separate but identical "and" operators for LDAP and local addressbooks.

As far as the filter files go, we've done exactly this sort of format upgrade in
the past.
*** Bug 301011 has been marked as a duplicate of this bug. ***

Updated

12 years ago
Duplicate of this bug: 356947
Depends on: 171073
QA Contact: filters

Updated

11 years ago
Duplicate of this bug: 426136
(Assignee)

Updated

11 years ago
Product: Core → MailNews Core
Duplicate of this bug: 494890

Updated

10 years ago
Duplicate of this bug: 510263
Duplicate of this bug: 530664

Comment 10

10 years ago
thanks for marking this as a DUPE, but is anything actually being done? This bug is now 4 years old.  If the functionality exists, is there an explanation somewhere of the syntax?
The functionality exists in the backend, but there is no user interface that exists that can use it (though the functionality for AND/OR filters is used for a variety of purposes in the backend). I am not aware of any ongoing efforts to fix that, though I'm sure contributions would be welcome.

I am also not aware either of any documentation of the syntax for that, other than the relevant .idl files which are quite sparse. However, I had to figure it out for myself in some work I did recently, and I fixed a serious bug there and also wrote a javascript unit test for it. So you might check out file test_searchBoolean.js as a demonstration of the syntax, or check bug 513277.

One workaround that is currently available is to write some simple javascript to do the actual test that you want, then implement that as a custom search term. I'll be releasing a TB3-compatible extension soon that implements a simple javascript interface for a search term, which also works in filters. Sorry I don't have more of that yet, but you can check http://mesquilla.com/2009/11/20/maybe-i-need-a-search-extension-searchaquilla/ where I mention that functionality briefly.

Comment 12

8 years ago
What is the difference of this bug and bug 274192? They seem to be both about Search AND Filters now. What about differentiating them?
Whiteboard: [filter-mgmt]

Updated

8 years ago
Blocks: 378343
How do you see this as being filter-mgmt?

Bug 274192 should be about advanced search only.
And this about Filters only.

It seems there might be some rationale for having common code. But I don't think we should let that delay delivering one bug or the other.
See Also: → bug 274192
Summary: Search/Filter and Boolean expressions → Filter and Boolean expressions
(Reporter)

Comment 14

8 years ago
Eh? This bug is over 6 years old, I find any concern over "delay" at this point to be laughable.

Comment 15

8 years ago
Wayne, I think you defined filter-mgmt as being about UI to do with creation and editing of filters. See your email of 22.10.2011.
I think comment 0 and comment 11 actually say this bug is requesting to add the UI for the existing backend features.
Is that not the case?
(In reply to :aceman (away 27.-2.) from comment #15)
> Wayne, I think you defined filter-mgmt as being about UI to do with creation
> and editing of filters. See your email of 22.10.2011.
> I think comment 0 and comment 11 actually say this bug is requesting to add
> the UI for the existing backend features.
> Is that not the case?

I've posted in mozilla.dev.apps.thunderbird so that anyone interested in this subject can participate. http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/233a96f96259c988#
Keywords: uiwanted
Summary: Filter and Boolean expressions → Enable Boolean expressions in Filter UI

Comment 17

8 years ago
The newsgroup lives on mozilla's private server, so going there via google (as above) gives read-only access: anything I try to post disappears.

But adding a newsgroup account for news.mozilla.org, then subscribing to mozilla.dev.apps.thunderbird works.

[Just trying to help anyone else who runs into the same problem.]

Updated

7 years ago
Duplicate of this bug: 461064

Updated

7 years ago
Duplicate of this bug: 418603

Updated

6 years ago
Duplicate of this bug: 846367
Posted image Nested Filter UI
Here is a suggestion for a User Interface that is fairly minimal as it is based on the existing Filter COnditions dialog. The idea is to allow nesting filters by allowing a new condition "Nested Filter applies to mail" (or is applicable, wording may vary). And then add a button to edit a nested filter or to select an existing filter.
Attachment #720376 - Flags: ui-review?(bwinton)

Comment 22

6 years ago
Axel, is the attachment meant for this bug? This one is about boolean expressions (AND, OR) between filter rules.

Nested (macro) rules and actions are in different bugs.

Can you explain?
Flags: needinfo?(axelg)
(In reply to :aceman from comment #22)
> Axel, is the attachment meant for this bug? This one is about boolean
> expressions (AND, OR) between filter rules.
Well actually I thought this one was about a _UI_ for combining different boolean rules.

The paradigm of nested "filters" would be a simple way to display and generate complex conditions.

e.g.
A and (B or C)

here you would choose the model

Condition A and Z
As Z you would build a condition (B or C) - simply by clicking the Edit button. This would lead to a new (if you like anonymous) filter which has no action and will not be shown in the main filters list. It can display exactly the same interface the users are already used to so they can build the condition
"B or C". I can create a mock up for this "nested condition" to illustrate if you like.

Once they name this new condition and click Ok, they are returned to the original "nesting" filter. Of course to bring together the (already existing?) mechanism that stores AND / OR mixed filters and to transpose this into the UI would be the difficult bit.

My thoughts on this were to introduce the concept of building this filter "gently" without the user having to learn a completely new UI.

Alternatively, we could try to "indent" the nested conditions and change the operator from AND to OR and back on each nested level, but I think that is going to work out as a lot more difficult both to design and comprehend.
Flags: needinfo?(axelg)

Comment 24

6 years ago
But this looks like a new UI and would be probably very hard to add as it probably breaks many paradigms in the current filters.

The current filter editor already had a column to prepend AND or OR before each rule. But that was removed for now as the editing is not implemented. And there is also the question on how to edit "brackets" (grouping of the rules). Yes, your approach solves this part.
Comment on attachment 720376 [details]
Nested Filter UI

Wow.  I'm really conflicted on this one.

On the one hand, this seems like a fairly elegant hack given the current constraints, on the other hand, I'm not sure whether people will figure out that this is the way for them to do what they're trying to do.  When I think of "A and (B or C)", I don't think of the word "nesting", and I'm not sure whether we should accept the current constraints, or whether we should try to overcome them while we're making this change…

As some nits with the screenshot itself, the first two boxes should be one, and maybe just call it "Filter applies to mail".  (I'ld like something that made the whole thing into a sentence, if possible, like "Subject contains QuickPasswords"…  Hmm, how about "Message matches filter FromMailingListFilter"?)

Having the four buttons there does seem a little much.  Also, no-one will actually be typing in the name of another filter, so I think we could get away with a single button to let you choose a different filter, and have the "create new sub-filter" option on that dialog.

Finally, when we create a new sub-filter, we should set it to never run, and change the default action to either "Stop execution", or better than that, a new action: "return success to parent filter".

So, I guess, based on all of that, I'm going to say ui-r-, but if you fill out the whole process of creating subfilters a little more, and specify how to deal with all the edge cases, and tidy up the dialog a bit, that'll probably change to a ui-r+…  :)

Thanks,
Blake.
Attachment #720376 - Flags: ui-review?(bwinton) → ui-review-
(In reply to Blake Winton (:bwinton) from comment #25)
> Comment on attachment 720376 [details]
> Nested Filter UI
> 
> Wow.  I'm really conflicted on this one.
> 
> 
> As some nits with the screenshot itself, the first two boxes should be one,

except it might be easier if users also had the option of 
[Subcondition][V]  [does not apply][V]

> and maybe just call it "Filter applies to mail".  (I'ld like something that
> made the whole thing into a sentence, if possible, like "Subject contains
> QuickPasswords"…  Hmm, how about "Message matches filter
> FromMailingListFilter"?)

first of all, thanks for reviewing. :)
thing is, whatever is included as nested / subfilter, is in actual fact not a full filter but just a filter condition. The proposal is just to reuse the (already known) filter details dialog to edit the included condition. So I believe the real trick will be to find the right (intuitive) labels for these things. So having a select for a different filter might indeed be superfluous.

> 
> Having the four buttons there does seem a little much.  Also, no-one will
> actually be typing in the name of another filter, so I think we could get
> away with a single button to let you choose a different filter, and have the
> "create new sub-filter" option on that dialog.

I believe when the button for the condition is clicked, the dialog should be shown again, the question is whether we allow giving the condition a name or should it just try and cram all condition elements in there so it becomes explicit? THis could become very long poentially, consider an example:

A and (B or C or (D and E))

that would show up as 2 lines:

     sender is John Williams
AND  subcondition is (subject is music or body contains "film music" or subcondition is (body doesn't contain guitar AND subject contains "Sky"))

so it might indeed be better for the user to be able assign a meaningful name to the subcondition such as 

     sender is John Williams
AND  subcondition is (JW the composer)

subcondition "JW the composer":
      subject is music 
   or body contains "film music" 
   or subcondition is (not related to Sky's guitar player)



> 
> Finally, when we create a new sub-filter, we should set it to never run, and
> change the default action to either "Stop execution", or better than that, a
> new action: "return success to parent filter".

Yes. I also believe that "sub-filter" should not actually show up in the filter list at all (because it is just a condition). Since it is stored in the original filter anyway and is not an entity on its own this shouldn't actually present a problem. The main rationale behind the hack is that we reuse the interface which our users already know how to use, and we do not actually reinvent a back-end for mode complicated filter conditions.

> 
> So, I guess, based on all of that, I'm going to say ui-r-, but if you fill
> out the whole process of creating subfilters a little more, and specify how
> to deal with all the edge cases, and tidy up the dialog a bit, that'll
> probably change to a ui-r+…  :)

yes I will do, once I get a bit of time. Especially, I need to make a screenshot of the "nested" filter / condition editor. I am almost in favor or re-instantiating another modal edit filter as child dialog, as this reflects the hierarchical relationship between the filter and its contained conditions. The hardest thing here is to put this into words without complicating it.

Axel

Updated

6 years ago
Duplicate of this bug: 873766

Updated

6 years ago
Summary: Enable Boolean expressions in Filter UI → Enable Boolean expressions in Filter UI (nested/grouped/mixed/combined/complex filters having AND and OR conditions)

Updated

6 years ago
Duplicate of this bug: 886898
Posted image multiple-Filters.png
Here is another approach on how the nested filters could be displayed. It doesn't address the question on how to add the start / finish of the nesting but shows how it could be displayed - adding 2 resizing columns at the left and right which will show the parenthesis and the operator (any or all). I added some color to make the "nesting" more obvious, but that might be technically to difficult as implementation. This is based on the desriptions in 
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIMsgSearchTerm#beginsGrouping and endsGrouping. I assume these attributes denote open and closed brackets (although I am not sure how you would store double brackets which close 2 groupings as in the example). Advantage of this approach is that it shows all information on one screen and is very close to the implementation of the back end (AFAIK), even though it might be a bit less intuitive. I did not mark my first UI as obsolete as I am still not sure whether it is not better than this one. 

An open bracket would necessarily always denote toggling the operator (and - or - and) and could be showing the term any / all as shown in the screen shot.

Comment 30

5 years ago
I'd be happy just to have one overriding AND condition with a sequence of OR conditions.
For example

AND: Does not contain "confirmation"
OR: from is Orbitz
    from is Travelocity
    from is La Quinta
    <etc.>
ACTION: Move to "travel" folder

This would move information on travel deals into the "travel" folder, which I would look at when I'm interested in browsing for travel deals.
I want confirmation messages to stay in my inbox, where I look for more urgent notices.
This is my fundamental desire, to separate information I would browse when I'm interested, versus notices that I want to see as soon as they come in. (Like "polling" versus "interrupt-driven").

Updated

5 years ago
Duplicate of this bug: 418620

Updated

5 years ago
Duplicate of this bug: 274192

Updated

5 years ago
Whiteboard: [filter-mgmt] → [filter-mgmt][gs]

Comment 33

5 years ago
For UI ideas, take a look at how OS X implements more complex boolean logic in the Finder window search (used for searches or to save search strategies).  The appearance is just like TB filters, but when you option-click the PLUS for another step, you get an indented sub-step that itself is just a recursive version of the level above.  The option-click is not obvious, which keeps newbies from getting confused (but probably hides it from a lot of non-newbies too). Anyway, the logic is AND/OR as complex as you'd like.  I think it would work well for TB.  It is intuitive to use and would not require the development of a new filter interface concept. The great OS X utility software Hazel does something very similar.  It works well there too.  FYI.

Updated

5 years ago
Duplicate of this bug: 348893
(In reply to rwstahlhut from comment #33)
> For UI ideas, take a look at how OS X implements more complex boolean logic
> in the Finder window search (used for searches or to save search
> strategies).  The appearance is just like TB filters, but when you
> option-click the PLUS for another step, you get an indented sub-step that
> itself is just a recursive version of the level above.  The option-click is
> not obvious, which keeps newbies from getting confused (but probably hides
> it from a lot of non-newbies too). Anyway, the logic is AND/OR as complex as
> you'd like.  I think it would work well for TB.  It is intuitive to use and
> would not require the development of a new filter interface concept. The
> great OS X utility software Hazel does something very similar.  It works
> well there too.  FYI.

Could you post a screenshot with an example? Not everybody owns a Mac (including me)

Updated

4 years ago
Duplicate of this bug: 1156676

Updated

4 years ago
Duplicate of this bug: 1175822

Updated

4 years ago
See Also: → bug 323773

Comment 38

4 years ago
Wayne: 1175822 isn't duplication. This is approach of other side.
Logically from email message side this is the same. But from user side this is different.
My approach is, filters next to filters are in 'or' logic. But next filter isn't executed when first is completly true and takes any action with message. Let we name it vertical behavior.
Second, horizontal behavior idea, filter has its subfilters. subfilter executes only when master filter is true. This is like 'and' logic. master filter can group more subfilters. subfilters are in 'or' logic ech to other in the same level, like files in one flat directory.
This cause, I can prepare simpler common rules in one master filters, and additional detailed rules in subfilters. Then common rule is executed only once, saves cpu work.
Currently I must duplicate common rules inside subfilters, because subfilters are not implemented.

Do you stil think 1175822 is duplicate ? I think 1175822 is more complex behavior, and on UI side can better use currently implemented booleans in filtering.
(In reply to grzegorz.szyszlo from comment #38)
> Wayne: 1175822 isn't duplication. This is approach of other side.
> Logically from email message side this is the same. But from user side this
> is different.
> My approach is, filters next to filters are in 'or' logic. But next filter
> isn't executed when first is completly true and takes any action with
> message. 

I think this is only the case if the email is moved to a different folder, but I might be wrong? Outlook has an option "stop processing filters on email" (paraphasing), would that be helpful in Thunderbird? It does make sense not to have additional filter actions on an email that has been moved / archived to a different folder.

> Let we name it vertical behavior.
> Second, horizontal behavior idea, filter has its subfilters. subfilter
> executes only when master filter is true. This is like 'and' logic. master
> filter can group more subfilters. subfilters are in 'or' logic ech to other
> in the same level, like files in one flat directory.

I wonder, would it be helpful if you could add filters to other folders than the INbox? I was always wondering whether there was a practical application for "cascading" filters down by copying mail to some other "parent nodes" and then doing more detailed filtering from there, rather than having all filters in one long hard to manage list.

Comment 40

4 years ago
(In reply to Axel Grude [:realRaven] from comment #39)
> I think this is only the case if the email is moved to a different folder,
> but I might be wrong? Outlook has an option "stop processing filters on
> email" (paraphasing), would that be helpful in Thunderbird? It does make
> sense not to have additional filter actions on an email that has been moved
> / archived to a different folder.

I think Move implies "stop filter execution" in TB too.

> I wonder, would it be helpful if you could add filters to other folders than
> the INbox? I was always wondering whether there was a practical application
> for "cascading" filters down by copying mail to some other "parent nodes"
> and then doing more detailed filtering from there, rather than having all
> filters in one long hard to manage list.

Yes, there is a bug requesting per folder filters (e.g. bug 294632).

Comment 41

4 years ago
(In reply to Axel Grude [:realRaven] from comment #39)
> (In reply to grzegorz.szyszlo from comment #38)
> > Wayne: 1175822 isn't duplication. This is approach of other side.
> > Logically from email message side this is the same. But from user side this
> > is different.
> > My approach is, filters next to filters are in 'or' logic. But next filter
> > isn't executed when first is completly true and takes any action with
> > message. 
> 
> I think this is only the case if the email is moved to a different folder,
> but I might be wrong? Outlook has an option "stop processing filters on
> email" (paraphasing), would that be helpful in Thunderbird? It does make
> sense not to have additional filter actions on an email that has been moved
> / archived to a different folder.

This is not my idea. You think about 'action' stage. I think about 'make decision' state.
It is no difference between moveint email from inbox to other folder, and marking email with star
or doing whatever. Simple example. I want make something with mail incomming from mailing group mailgroup@anywhere.com sent by user user1@user1domain.com. Then I want do somethink different from the same mailing group, bot sent by user user2@user2domain.com. What I do usually? I make two filters. First with rule replyto: mailgroup@anywhere.com and sender: user1@user1domain.com and any action. Second filter rule is replyto: mailgroup@anywhere.com and sender: user2@user2domain.com and any action.
You see, replyto: mailgroup@anywhere.com is common.
I think making parent filter0 with only replyto: mailgroup@anywhere.com without any action, or with any default action. Then making two slave/sub filters with only filter1 user1@user1domain.com and any action and  filter2 user2@user2domain.com and any action.
Then if filter0 doesnt match, then filter1&filter2 are not run.
Consider idea, when filter0, filter1&filter2 are more complicated. My approach cause simplification for thunderbird user. Why? because filters are naturally grouped by this function. it isn't flat list. secondly, user on filters configure only important configuration on its level.

> > Let we name it vertical behavior.
> > Second, horizontal behavior idea, filter has its subfilters. subfilter
> > executes only when master filter is true. This is like 'and' logic. master
> > filter can group more subfilters. subfilters are in 'or' logic ech to other
> > in the same level, like files in one flat directory.
> 
> I wonder, would it be helpful if you could add filters to other folders than
> the INbox? I was always wondering whether there was a practical application
> for "cascading" filters down by copying mail to some other "parent nodes"
> and then doing more detailed filtering from there, rather than having all
> filters in one long hard to manage list.

No. no like that. I investigate only the case when we filter out inbox only.
I had a closer look at the filter parser, and apparently it doesn't currently support parsing additional ( and ) for setting the beginsGrouping and endsGrouping attributes:

http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgFilterList.cpp#727

Would writing a patch for support of this be in scope for this bug? 

Example / Test Case:
(AND (subject,contains,FS Notification) 
  OR (\"list-id\",contains,test)) 
 AND (from,contains,fastspring.com)

simplified: (A OR B) AND C

expected Parsing result:

A.booleanAnd = true     // is this ignored?
A.beginsGrouping = true
A.endsGrouping = false

B.booleanAnd = false
B.beginsGrouping = false
B.endsGrouping = true

C.booleanAnd = true
C.beginsGrouping = false
C.endsGrouping = false

Rules for writing a condition: 
beginsGrouping => "(" before booleanAnd
endsGrouping => ")" at end of a term - which is after the condition list)

(Parsing might be easier if we use different parentheses, such as { } or [ ] )
(In reply to Axel Grude [:realRaven] from comment #42)
> I had a closer look at the filter parser, and apparently it doesn't
> currently support parsing additional ( and ) for setting the beginsGrouping
> and endsGrouping attributes:
> 
> http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/
> nsMsgFilterList.cpp#727
> 
> Would writing a patch for support of this be in scope for this bug? 
> 

Yes.

The tricky part would be making it backwards compatible. Perhaps there is some way to add these additional terms that would allow the old format to be used? I have not really thought that through much, not sure if that is a Good Idea or not.

Updated

3 years ago
Duplicate of this bug: 1211631

Comment 45

2 years ago
I've wanted this in Thunderbird for many years. While the base concept is pretty simple and obvious, apparently, the implementation is not.

Now that Thunderbird is striking out on its own, maybe this could be factored in when designing the new improved version.

Updated

2 years ago
Duplicate of this bug: 1375583

Updated

4 months ago
Duplicate of this bug: 1392704

Comment 48

4 months ago
I don't know the code for this project, but from a data representation standpoint, any complex algebraic expression can be stored in a simple linked list/array/vector using polish notation consisting of only the conditions themselves, the binary operators OR, AND, XOR and a unary NOT operator.

Example2:
    cond1 AND (cond2 OR cond3)

Becomes:
    AND, cond1, OR, cond2, cond3

Or using the example above:
    ((subject,contains,FS Notification) OR (\"list-id\",contains,test))
     AND (from,contains,fastspring.com)

Becomes:
    AND,
    (from,contains,fastspring.com),
    OR,
    (subject,contains,FS Notification),
    (\"list-id\",contains,test)

The filter conditions become the operands to the operators and these lists are easily processed by each object having an "eval" function that takes a list iterator by reference and returns its result.  The eval is then called recursively until all operators and operands are evaluated.  C++ example:

    /* operator or operand */
    template<typename T, typename O> class Oper {
        typedef std::vector<Oper<T, O> > vector_type;
        typedef vector_type::iterator iterator;
        
        virtual T eval(iterator &i, const iterator &end, const O &obj) = 0;
    };

Every call to eval increments the iterator (throwing or asserting if it reaches one-past-the-end).  Binary operators will do something like a = iter->eval(); b = iter->eval(); and then return their result of a OP b.  Unary operators behave similarly.  At the root of the operation, you just call eval() on the head of the list and verify that the the resulting iterator is the last valid element.

    Oper<bool, Message>::vector_type my_filter = something;
    Oper<bool, Message>::iterator i = my_filter.begin();
    bool result = (*i)->eval(i, my_filter.end());
    assert(i == my_filter.end() - 1);

AND might look like this:
    template<typename T> class OperandAnd : public Oper<T> {
        T eval(iterator &i, const iterator end, const O &obj)
        {
            ++i;
            assert(i != end);
            return (*i)->eval(i, end, obj) && (*i)->eval(i, end, obj);
        }
    };

Of course, you should be aware of C++ operator precedence and evaluation order here.
(Reporter)

Comment 49

4 months ago
Daniel - that's all great, but please read the ticket you're responding to.

The filter engine already implements this stuff. LDAP filter syntax is already in Polish notation.
The only thing missing is a UI to allow users to use the full capability of the filter engine.

That's what this ticket is about.

Comment 50

4 months ago
HAH!  "Since Bug 171073 has been fixed, the filter engine is now capable of processing
arbitrarily complex Boolean expressions"  You mean I should READ? :)

I guess I'll have to live with my 80-odd email filters for now.  Sorry about that.

Updated

2 months ago
Blocks: 1192478
You need to log in before you can comment on or make changes to this bug.