1127267 - Simplify and reorganize header filter labels of advanced "Search messages" and "Filter Rules" dialogues for consistency

Status: NEW

(Thunderbird :: Search, enhancement)

Description

[Thomas D. (:thomas8)]

Attachment: XUL Demo (V.1): Proposal for new criteria dropdown in Search Messages & Filters

+++ This bug was initially created as a clone of Bug #586101 +++

This bug wants to improve the UX of the advanced "Search messages" and "Filter Rules" dialogues for both normal and advanced users; more specifically, to simplify labels, reorder, and group the list of search criteria offered to users in the header selector dropdown.

Please find attached a full-fledged XUL demo which visualizes this proposal in direct comparison with current implementation.
It's actually simple and straightforward when you look at it, and much better in terms of ux-consistency, ux-discovery, and ux-efficiency; so I recommend  checking out the annotated demo before going into the full version of philosophical details and UX reasons below.

To view the XUL demo in FF:
- Install Remote XUL Manager:
  https://addons.mozilla.org/en/firefox/addon/remote-xul-manager/
- add a permission for bugzilla.mozilla.org (and <file> for local xul files): about:addons > Remote XUL Manager > Settings button

I'm looking at this purely from a front-end UI perspective; backend is another story. Some backend fixes in nsMsgSearchAttribs are required to make this work (existing bugs), where as a rule of thumb I suspect most of the backend searches should be modelled in analogy to the frontend, notwithstanding possible combinations.

This RFE has been inspired by philosphy [1] and ideas from Kent James (:rKent), and also addresses some of his critical feedback on the matter.

*** STR ***

1a) Open the Advanced "Search Messages" dialogue (from main 3-pane, Ctrl+Shift+F or Edit > Find > Search Messages...)
1b) Open the "Filter Rules" dialogue (Tools > Message Filters > New...)

2) From the criteria input area, open the header selector dropdown box (showing "Subject")

3) Look at the dropdown with the eyes of an average user trying to figure out how to define useful searches like searching in "Sender" or "Recipients" of a message, or both (after you've come to know and love these terms from convenient and highly efficient Quick Filter bar).

4) Look at the dropdown again with the eyes of an advanced user trying to define more advanced searches with precise control over which headers to search or not (more so considering that we don't offer nested searches yet where users could freely combine AND and OR criteria using brackets or such).


*** Actual result (see XUL demo) ***

To quote a critical review of TB once found in Linux Magazine [1],

"It’s too complex for casual users, and not full-featured enough for business users."

Exactly that for the header selector dropdown in its current arrangement; it's a pretty unstructured line-up of disconnected options (for everyone), and it's incomplete (for advanced users). More specifically:

a) unstructured list with no recognizable order
b) lacking natural language search criteria
c) captions are not ux-consistent with quick filter bar
d) random mix of basic criteria and advanced criteria
e) random mix of freeform and closed-set search criteria
f) 17 items; some important criteria are missing (feature-incomplete)


*** Expected Result (see XUL Demo!!!) ***

In a nutshell, simplify labels, reorder, and group the list of search criteria so that's it's better for both normal and advanced users: more ux-consistent with Quick Filter Bar, more ux-discoverable, and more ux-efficient. Address all the current shortcomings listed in Actual Result.

a) reordered, structured list with clear subsections (separators) - easier to parse
b) natural language search criteria (with tooltips for technical details!)
c) captions are ux-consistent with quick filter bar
d) basic/frequent criteria first, then advanced/rare criteria
e) freeform criteria and closed-set search criteria grouped together
f) 21 items; includes all important criteria (feature-complete)

#1: Integrating pending changes of filter criteria, the proposal simplifies/adjusts the labels {and functionality} of 3 existing criteria using the same natural language terms seen in Quick Filter Bar (ux-consistency):

Current criteria (too technical;{incomplete}) ->  New criteria (as in QFB) [Menuitem's tooltip]
-----------------------------------------------------------------------------------------------
From {or Reply-To}*                 ->  Sender                [From or Reply-To]
To or CC {or BCC}*                  ->  Recipient             [To, CC, or BCC]
From {or Reply-To}*, To, CC, or BCC ->  Sender or Recipient   [From, Reply-To, To, CC, or BCC]

*: Three bugs pending to add missing headers to some search criteria:

Bug 586101 - Add "BCC" to "To or CC" Quickfilter (Recipient)
Bug 600928 - Add "Reply-To" to "From" Quickfilter (Sender)
Bug 535879 - Add "Reply-To" to "From, To, CC, or BCC" Quickfilter (All Addresses / Sender or Recipient)

#2: Preserve/complete the existing set of primary header criteria (To, From, Reply-To, CC, and BCC), to be feature-complete both for normal and advanced users; preserve "To or CC" to work around our lack of precedence operators (i.e. there's no other way advanced users can currently create such OR-searches if they also need AND at the same time). Accordingly, net addition of 4 new criteria:

- Reply-To (This is just missing for no reason although it's not structurally different from, say, CC).
- BCC (yes, it's needed for advanced users who want to filter their sent messages, and they'll know what they are doing; no harm for normal users if we tuck this away at the far end of the list and offer better choices at the top)
- Recipient (To or CC or BCC; Bug 586101); keeping "To or CC" is still required after that to avoid AND/OR precedence problems (see details below)
- Sender (From or Reply-To; Bug 600928, Bug 535879); keeping "From" is still required after that because depending on scenario, "Reply-To" might not always be wanted for Sender searches; all primary headers like To, From, Reply-To, CC, or BCC should obviously be offered as this is an "Advanced Search".

Why do we still need "To or CC" when we already have "To, CC, or BCC"?

If we only offer "To or CC or BCC", there's technically no way to search for just "To or CC" is "Peter" AND "Subject" is "holiday". Depending on scenario, both business and private, it can make a big difference if recipient was CC'ed or BCC'ed, so there's no reason to exclude such more precise searches by design. We can negotiate eliminations again when we finally offer flexibility of nested/mixed AND/OR searches...

[1] Linux Magazine Article "Is Thunderbird Too Little, Too Late?" (interesting read regardless of opinion)
http://www.linux-mag.com/id/7788/
Thanks to Kent James who raised this on tb-planning in a thread titled "Role of addons" (2012-09-14).

To which he (rkent) has this remarkable answer (which applies seamlessly to this bug):
> I think that our sweet spot (to use an overused term) is the advanced,
> non-enterprise user. The complex features absolutely need to be there for
> our main users, which means that they need to be in the core code. But we
> still need some method to push the boundary toward the simpler users. That
> means simplifying the user interface.

+1 :)

Comment 1

[José Fortes]

I truly much appreciate the effort in improving Thunderbird search interface, but I'd suggest the discussion shouldn't block for long what it seems a relatively simple and noncontroversial solution to the original years-old statement of Bug 586101, that is adding BCC to Recipients for Quick Filter on Sent folder.

Comment 2

[Thomas D. (:thomas8)]

(In reply to Thomas D. (currently busy elsewhere; needinfo?me) from comment #0)
> Created attachment 8556401 [details]
> XUL Demo (V.1): Proposal for new criteria dropdown in Search Messages &
> Filters

> To view the XUL demo in FF:
> - Install Remote XUL Manager:
>   https://addons.mozilla.org/en/firefox/addon/remote-xul-manager/
> - add a permission for bugzilla.mozilla.org (and <file> for local xul
> files): about:addons > Remote XUL Manager > Settings button

Due to changes in BMO attachment URLs, and what I consider a bug in Remote XUL manager (https://*.bmoattachments.org can't be set as an exception), viewing XUL attachments now requires one permission entry in remote XUL manager for every bug:

- In Remote XUL manager settings (Tools > Web Developer > Remote XUL Manager), add this permission:
https://bug1127267.bmoattachments.org

To view and play with local XUL files:
- In about:config, set dom.allow_XUL_XBL_for_file to true.
- add <file> to Remote XUL manager's list of permissions

Comment 3

[Richard Marti (:Paenglab)]

It would be simpler, when you make a screenshot of both menulists side by side and attach it here.

BTW what I've seen makes sense.

Comment 4

[Thomas D. (:thomas8)]

As hinted before, this proposal improves the UX for both regular and advanced users:
* ux-consistency between natural language filter terms in quick filter bar and advanced search/filters
* ux-discovery, ux-efficiency: for every type of user, both simple and advanced search criteria are now easier to discover and faster to use because the list of criteria has been reordered and separators provide structure by grouping related criteria, which allows faster visual parsing of the list

Comment 5

[Thomas D. (:thomas8)]

Attachment: HTML Demo (V.1): Proposal for new criteria dropdown in Search Messages & Filters

For ease of access, here's an HTML version of the XUL demo, featuring the new list of filter criteria:
- relabeled
- reordered
- structured by separators
- feature-complete

Richard, can you please offer some UI feedback?

@everyone: feedback welcome! (Pls appreciate my comments above which explain the philosophy of this in detail).

Comment 6

[Thomas D. (:thomas8)]

Attachment: Screenshot 1: Proposal for new criteria dropdown in Search Messages & Filters

(In reply to Richard Marti (:Paenglab) from comment #3)
> It would be simpler, when you make a screenshot of both menulists side by
> side and attach it here.
> 
> BTW what I've seen makes sense.

For completeness, here's the comparative screenshot. For ease of analysis, I've added numbers to the menu items so that the reshuffling and additions become transparent.

Please refer to the HTML Demo of attachment 8846501 [details] to see the technical tooltips on the natural language search items (Sender, Recipient, Sender and Recipient). For advanced users, the tooltips will provide clarity as to which headers we actually search. Otherwise, it will just work as expected after we fix the respective bugs (see comment 0 for details).

Please note that in terms of ux-discovery, it's *salience of frequently used items* that matters, not absolute length of criteria list. In a dictionary of 1000 entries, it'll be easy to find "Adam" if there are only 5 entries for "A" which are toplisted at the beginning of the book. Same here. Most frequent items are toplisted, and similar items grouped, hence everything much easier to find than before. E.g., "Date" and "Age in days" were separate before (only God knows why), now together.

Comment 7

[Richard Marti (:Paenglab)]

Comment on attachment 8846501 [details]
HTML Demo (V.1): Proposal for new criteria dropdown in Search Messages & Filters

Yes, this order and grouping makes sense. The tooltips aren't needed. First I'm not sure they are supported on menulists and second the items are self explanatory enough for a first search. If a user wants more specialized search, he then will use the more specific entities.

Comment 8

[:aceman]

I like this too. But can we first move bug 586101?

Comment 9

[:aceman]

Thomas, considering bug 586101 comment 85, could you update the proposal marking which items (headers) would be part of the "basic search" and which would only be exposed for the "advanced search". I think we would have only the one list of headers as today, but some rows would be initially hidden. Operating (in some way) the "show advanced headers" item would expose all the headers. All other functionality would be unchanged.

Comment 10

[:aceman]

Attachment: WIP patch for the functionality of exposing advanced headers

This uses onmouseover event on the menuitem for now as I didn't find any better way. Any click on the item causes the menupopup to close and I didn't find a way to prevent it.

Comment 11

[Richard Marti (:Paenglab)]

Comment on attachment 8848762 [details] [diff] [review]
WIP patch for the functionality of exposing advanced headers

Yeah, it's a bit surprising when it opens on mouseover. A click would be preferred, but when not implementable then we can use this.

The expanded list should still have the separator at the same place. This helps the user to see what are the advanced items.

I think the menu should hide the advanced items again after closing and reopening the menulist. Only when a item from the advanced part is chosen and the user opens the menulist, then the advanced items should be shown.

Comment 12

[Magnus Melin [:mkmelin]]

I don't think you should be hiding this behind a mouseover. That's not keyboard accessible, and it's surprising non-standard behaviour. But then again, I don't think those options need to be hidden anyway. All of this is in a way pretty hidden away from novice users so I'd assume people don't really mind that the list is somewhat long.

Comment 13

[Thomas D. (:thomas8)]

(In reply to Kent James (:rkent) from bug 586101 comment 85)
> I like the term "Recipients" for "To, CC, and BCC" but the whole proposal
> adds 4 additional search terms to an already complex table,

Yes, while radically *reducing the complexity* (former chaos) of same table. As I already explained in comment 6, absolute length of a list doesn't matter much, but *salience of the most frequently used items* does. Net result: /In spite of adding some search terms/, UX-discovery has been *massively improved* by reordering and structuring the list with separators. Plus, my comment 0 and ff. explain in detail the rationale of adding these and not removing similar items. Can anyone advocating for less kindly refer to those detailed arguments first...

> plus directly exposes BCC as a search term

Sorry, but imho there's nothing special about BCC; it's one of the default headers which are part of our everyday email culture, more so for advanced users which are the target group of this feature. In fact, we as privacy-concerned Mozillians should actually promote BCC wherever possible. And again, can we please stop underestimating the cognitive abilities of our advanced users: I believe even users who might be initially unaware that BCC headers can't be found in received messages are bound to find out very quickly when their searches don't return the expected results. Advanced users who don't care about BCC are not going to use it because we offer better choices at the top of the list, which easily cover the most common use cases. 

As a compromise, as several others have suggested before, what if we just add a hint in the menuitem label?
> BCC (if available)
We don't want these headers to be translated so it'll be roughly the same length for other languages.
Plus I don't share Richard's aversion against tooltips: they don't hurt in any way because they are hidden when you don't need them, but will appear at your fingertips exactly when you need them. Tooltips work on menuitems (as seen in my XUL demo), so we could easily add this tooltip for ux-error-prevention:
> "Please note that BCC headers are not available on received messages."
Done. Problem solved.

> It seems to be driven also by "Crucial functionality should not be left to
> addons. Anything search is crucial."

I realize that this comment of mine can be misunderstood if taken as an absolute. I do consider offering all standard message headers as filter criteria crucial functionality /for advanced users/. I for one would find it irritating and unprofessional having to add "BCC" as a custom filter, which feels flimsy and unstable. And if BCC is considered special, what about "Junk score origin" and "Junk percentage"? Let's face it, this is really for advanced users, notwithstanding that this bug will make things a lot easier for the wandering casual user, too.

Comment 14

[Thomas D. (:thomas8)]

(In reply to Richard Marti (:Paenglab) from comment #7)
> Comment on attachment 8846501 [details]
> HTML Demo (V.1): Proposal for new criteria dropdown in Search Messages &
> Filters
> 
> Yes, this order and grouping makes sense.

Thank you, I reshuffled this a lot to improve ux-discovery and ux-efficiency.

> The tooltips aren't needed.

Objection, your honor!

> First I'm not sure they are supported on menulists

They are supported, as seen on my XUL demo.

> and second the items are self explanatory enough for a first search. If a user
> wants more specialized search, he then will use the more specific entities.

Basic and advanced items are complementary; due to our shortcomings wrt precedence operators, there is no way to generate the functionality of, say, "Recipients" (as a multiple OR condition), using the single headers (To - CC - BCC), as soon as you want to add just one AND condition, say "AND subject is...".

I think in this case tooltips are required to provide clarity about which headers we are actually searching.
- Sender: Users can't guess that we are now including "Reply-To" in this criterion (pending bugs). Whilst for many users we're probably doing the right thing, some users might even face false positives if we don't offer that information.
- Recipient: Users can't guess that we are now including "BCC" in this criterion (pending bugs). Historically, we haven't (Bug 586101 and friends), which is another reason to make this explicit now on the tooltip.
- Sender or Recipient: dito.

On the BCC menuitem, an explanatory tooltip would also go a long way for ux-error-prevention.

Whilst I'd agree that generally speaking, labels should be as self-explanatory as possible, and tooltips should not be an end in itself, I'm yet to hear any significant reason *against* adding tooltips where they add valuable information.

Comment 15

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #12)
> I don't think you should be hiding this behind a mouseover. That's not
> keyboard accessible, and it's surprising non-standard behaviour. But then
> again, I don't think those options need to be hidden anyway. All of this is
> in a way pretty hidden away from novice users so I'd assume people don't
> really mind that the list is somewhat long.

+1.

Indeed it is unlikely for John Doe to find or use Advanced Search or Message Filters, so Magnus has pointed out yet another argument which addresses the concerns of bug 586101 comment 85. For the actual users of these dialogues and their workflows, hiding entries in this list really doesn't add any value wrt UX, it actually makes UX-discovery and UX-efficiency worse. Please let's not underestimate the cognitive abilities of our users; there's no magic required to visually parse a 21 items list which is now grouped into 3 distinct sub-sections:
5 natural-language items at the top (that's where most users will find what they need, no need to parse more)
10 less used criteria bundled in the middle (with the most likely candidates, Tags and Date/Age, toplisted)
6 single headers tucked together at the bottom, again as a clearly distinct group.
For comparison, before this bug we expected our users to parse a 17 items monolithic list without any logical structure or sorting.

I think given the positive comments of Magnus and others, plus my painstaking explanations of the underlying rationale, we're good to proceed with the original proposal of this bug, per Kent's disclaimer in bug 586101 comment 85:

> This is only my opinion, and I do not have a veto over this. Please do
> whatever the majority of contributors want.

Comment 16

[Thomas D. (:thomas8)]

Richard, fyi:

Nit: We need to re-label "Body" into "Message Text" (also in quick filter UI).

* "Body" is pretty geeky and doesn't fit nicely with the new natural language terms (Sender, recipient, etc.)
* "Body" is actually technically wrong, because what we really mean is "Message text", not the entire body (although, as correctly pointed out by Kent, there are bugs where we search even the raw mime flavor of attachments...):

https://www.lifewire.com/what-is-the-difference-between-email-body-and-header-1171115

> The email body is the main part of an email message. It contains the message’s text,
> images and other data (such as attachments)