972690 - Recipient autocomplete name search appears "no longer alphabetical" (should prioritize startsWith matches of display name words)

Status: RESOLVED DUPLICATE of bug 970456

(SeaMonkey :: MailNews: Composition, defect)

Description

[gwelsh]

Attachment: AddressSM223-vs-SM224.jpg

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23 (Beta/Release)
Build ID: 20131210202506

Steps to reproduce:

typed   the  letters   me   in the "To" field of message composition window


Actual results:

auto-complete suggestions are not in any sensible order and do not start with  me, except for third suggestion


Expected results:

results should start with  letters   me   like they do in  SM2.23.

see attached file showing a screen shot of both SM 2.23 acting correctly and SM 2.24 not acting correctly when typing the (example) letters   "me".

Comment 1

[Tilmann Reh]

Same here with Windows.

Since that change, using Nicknames is absolutely useless, they simply don't work any more. The previous method was much better.

If the search algorithm resp. search order has been changed for whatever reason, at least the nicknames must get higher priority than any other match.

Comment 2

[Thomas D. (:thomas8)]

(In reply to gwelsh from comment #0)

> Expected results:
> results should start with  letters   "me"   like they do in  SM2.23.

(In reply to Tilmann Reh from comment #1)
> If the search algorithm resp. search order has been changed for whatever
> reason, at least the nicknames must get higher priority than any other match.

Gwelsh and Tilmann, thank you for voicing your concerns that some of your personal use patterns which appeared to work before no longer do. So let's have a look what is really going on here.

Both Gwelsh and Tilmann describe scenarios where they expect a stable 1 on 1 relationship between search terms and the topmost result of recipient autocomplete: For search term "foo", users expect that TB should always return a certain contact "Foo bar <[e-mail]>" as the topmost result.

There are two design concepts that allow for such more stable relationships between search terms and results, and both of them are NOT functional in TB/SM (neither before nor after the change of result set observed here):

Nicknames: The only 100% failproof way to ensure stable (static) 1 on 1 relationship. User defines unique "alias" search term for specific contacts. Currently broken:
Bug 325458 - Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries

Frecency: Semi-stable (dynamic) 1 on 1 relationships based on user's use patterns, with an algorithm considering "frequency" and "recency" of how user picks certain results after certain search terms. So if after searching for "foo", user has frequently AND recently picked "Foo bar <[e-mail]>" from results list, TB/SM will remember that and rank that contact higher. This feature is currently NOT implemented (or partially at best, as we do have a dull frequency counter aka popularity index):
Bug 382415 - Popularity index of recipient autocomplete doesn't honor timeline (use frecency for email contacts)

What this means is that both scenarios described in comment 0 and comment 1 have never been scenarious with "stable" behaviour but it's rather accidental and unreliable that they have worked so far for your particular datasets. E.g. if Gwelsh had another contact "me1" which he might have used 100 times 10 years ago, that old contact would still always show first when he searches for "me", regardless of alphabetical order, and even if he were to chose newer "me" contact this year up to 99 times (bug 382415). Same for Tilmann: You are just lucky that your nicknames have worked so far (because they were either unique enough or had high popularity index), but before or after the change, nicknames have never worked as they should (bug 325458); as a workaround, just make nicknames unique e.g. "#me" instead of just "me".

FYI: The order of result sets was NOT changed, but indeed single-word searches (especially with only 2 or 3 characters, which really isn't unique enough) might have slightly larger result sets now depending on your data structure and search terms. The algorithm had to be changed because we had lots of users rightly complaining that expected results were not found, even for obvious cases like FirstName="Anne Mary" or Email=<[e-mail]>, where before the change, searching "Mary" would fail because of beginsWith per field logic (now replaced with contains logic). We're in the process of making multiword searches more efficient which will go a long way to make search much more powerful, versatile, and precise than now. Summary and explanation of these changes in Bug 558931 Comment 109; contains logic was introduced in bug 529584.

Comment 3

[Thomas D. (:thomas8)]

Per my explanation in comment 2, I believe this bug is actually invalid.

Comment 4

[Tilmann Reh]

Thomas, thanks for your explanation.

If it really was an "accidental" feature, it has been surprisingly stable for many many years... I have been using Mozilla Suite since 2004, then SeaMonkey from 1.0 until today - and have *always* used nicknames this way.

I completely agree with the many statements in bug 325458 - so when will this bug finally be fixed? After all, it's eight years (!) old, and it shouldn't be that hard to fix it, IMHO...
If that's fixed, I don't mind the other changes in the search method. Otherwise, please revert to the previous, well functioning (even if accidentally...) method until that bug is finally fixed. Thanks.

Comment 5

[gwelsh]

So there is no alphabetical logic imbedded in the auto-complete (search), and never was?  Well then that's still a bug.  If every cell phone I have ever used, can show me names starting with any letter I type, then SM should be able to do it too.

Can we change the title of the bug to "auto-complete name search is not alphabetical"?

Comment 6

[:Hb]

Latest Thunderbird behaves similar. So this bug aims at product: MailNews Core with component: Composition.

Comment 7

[Thomas D. (:thomas8)]

(In reply to gwelsh from comment #5)
> So there is no alphabetical logic imbedded in the auto-complete (search),
> and never was? Well then that's still a bug. 

Nobody said there's no alphabetical logic, but currently the 1st order criterion for autocomplete results is popularityIndex (without which your desired effects of toplisting your favourite contacts which you got used to wouldn't even happen right now).

> If every cell phone I have
> ever used, can show me names starting with any letter I type, then SM should
> be able to do it too.

I doubt if we can just copy UX concepts from cell phones for an all-purpose mailing application (e.g., think of enterprise use), but I get what you're trying to say. But again, I suspect what you really want is static or semi-static relationship between your searchwords like "me" and the topmost result contact. For that purpose, alphabetical won't help at all; what you really want is bug 382415 and bug 325458, both of which are excellent solutions to your usecase (and both bugs existed before we changed the search algorithm, they might just be more exposed now for existing users depending on your personal dataset and use patterns. In fact over time, if you keep using certain contacts more often than others, it will adapt to some extent because of popularityIndex).

> Can we change the title of the bug to "auto-complete name search is not
> alphabetical"?

Dito, alphabetical as such won't help much if not complemented by frecency and nicknames.
Prioritizing search results is a very complex issue and there are no simple solutions (your google results aren't alphabetical, are they?). For some problems, see Bug 970456 Comment 8, and Bug 970456 Comment 12.

But if we try hard enough (for which manpower and time is missing), we might come up with something along the lines of the following bug:

Bug 970456 - Recipient autocomplete: Explore idea of prioritizing "beginsWith" wordwise matches over "contains" matches: search for "be" should toplist "Ben" and "Betty" over "HolBErt" (having "be" in the middle of word)

This bug 972690, if not invalid, might otherwise be a duplicate of bug 970456.

Comment 8

[Thomas D. (:thomas8)]

Sorry, unintended field change (corrected here).

Comment 9

[gwelsh]

> I suspect what you really want is static or
> semi-static relationship between your searchwords like "me" and the topmost
> result contact. 

No, I want the auto-complete to work like a cell-phone.  I, in fact, thought that WAS how it worked.  I 100% do NOT need it to search at all, I expect it to list names starting with the letter I type, and then narrow down by the next letter I type.  I never realized there was any search component at all.  If I were opening the address book and trying to find something I don't remember then a search component is useful, but when I am just starting to type a name, in the composition window's TO field, it should just list names starting with that letter.

Comment 10

[Philip Chee]

Setting dependency to Bug 970456 for tracking purposes.

Comment 11

[john]

The simplicity that gwelsh seeks in comment 9 is appropriate and useful, as is the popularity tool (aka frecency).   

Given that the simple "gwelsh" search is easily programmed (searches for first/last or email), and works like a simple dictionary, why not place it in the code, and craft a preference that toggles the simple search versus the complex search/sort

Comment 12

[gwelsh]

I was quite shocked when I discovered what was going on, i.e.  the autocomplete was not an autocomplete at all, but a search function, that used to be quite weak and therefore seemed to just autocomplete but was recently "improved" and therefore giving all sorts of completely unrelated (from my perspective) matches.

Comment 13

[gwelsh]

I see Bug 970456 is fixed and target milestone is TB 36.  Do we just wait and see if that spills-over into SeaMonkey 2.36 ?  Or is that not how it works?

Comment 14

[gwelsh]

I just installed SeaMonkey 2.34a1 and this bug is FIXED in that version.

Comment 15

[Thomas D. (:thomas8)]

Per reporter's comment 14, this bug has been FIXED for SeaMonkey.
So I reckon it must have been fixed by Bug 970456. Which makes this a duplicate.
Anything else after that, pls file new bugs/RFEs.

FTR: AutoComplete result sorting has never been fully alphabetical, but (before Bug 970456) gave absolute priority to popularityIndex (external OS X and LDAP ABs excluded).

Users like reporter were seeing effects of Bug 529584 where field-startsWith algorithm was expanded to field-contains algorithm, avoiding search failures for many real world scenarios but also expanding result sets especially for very short searchwords (like 1 or 2 characters). Due to higher popularityIndex, contacts matching less significantly in the middle of words could nevertheless end up on top of the result set, often contrary to users' expectation. Bug 529584 was complemented by Bug 558931 which allows rapid and radical reduction of the number of autocomple results by entering two search words, where a "words" can be any significant character sequence from anywhere in the contact's names or address. E.g. searching for "gw ha" can now succeed to filter just Gwelsh (at) hawaii.tld from thousands of AB contacts. As I explained in previous comments, further improvements like real "frecency" algorithm etc. are needed and on record.

(In reply to john ruskin from comment #11)
> The simplicity that gwelsh seeks in comment 9 is appropriate and useful, as
> is the popularity tool (aka frecency).   
> 
> Given that the simple "gwelsh" search is easily programmed (searches for
> first/last or email), and works like a simple dictionary, why not place it
> in the code, and craft a preference that toggles the simple search versus
> the complex search/sort

John, feel free to file that as a new RFE if you think it's still needed.
Please note we already offer dictionary-style alphabetical result sets in contacts side bar (F9 in composition), which I think covers that usecase pretty well (or you can even start out from alphabetical listings in the main AB).

Recipient autocomplete was designed to fill a recipient input field with a *single*(!) email address, which needs to be retrieved efficiently from an unlimited number of records in user's ABs. So imho that purpose is really much better and more efficiently realized with an intelligent search algorithm than with alphabetical listings. Alphabetical listings without other means of prioritization will always fail for any scenarios where you have more than one alphabetical match, e.g. several matching contacts with same or similar display name. So you'll end up typing many more characters than necessary to narrow down your search using an alphabetical "startsWith" approach. As opposed to that, multiword search and frecency algorithm (as in FF location bar) will allow users to just type whatever they remember best (without considering coincidental field data structure), rapidly narrow down result sets, and TB will learn over time to toplist the best match depending on users personal preferences (derived from frequency and recency of contact usage, also involving the combination of search input and final contact picked by user). Note that dynamic frecency is not fully implemented yet.

Alternatively, explicit, permanent and stable definition of unique short nickname field value for a given contact can also be a highly efficient method of retrieving addresses with typing just two or three characcters using autocomplete (some bugs standing; hence currently nickname should be unique against any other field content anywhere in your ABs). E.g. you could define "g#" in nickname field of "Gwelsh hawaii" contact, then in recipient autocomplete, just type "g#", press Enter, - done! Works reliably and exclusively 100% of the time, regardless how many other contacts called "Gwelsh" might be populating your ABs.