Closed Bug 325458 Opened 19 years ago Closed 10 years ago

Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries, for searchphrase==nickname [To, CC, addressing field/area, toplisted, priority, results]

Categories

(Thunderbird :: Message Compose Window, defect)

Product:
Thunderbird
Component:
Message Compose Window
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(thunderbird37 fixed, thunderbird_esr31 affected)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 38.0
Tracking Flags:
Tracking Status
thunderbird37 --- fixed
thunderbird_esr31 --- affected

People

(Reporter: brian.atkins, Assigned: sshagarwal, Mentored)

References

(Blocks 1 open bug,
URL
)

Details

(4 keywords, Whiteboard: [good first bug][lang=js])

Attachments

(1 file, 3 obsolete files)

Patch for check-in
6.77 KB, patch
sshagarwal
: review+
sshagarwal
: feedback+
mkmelin
: approval-comm-aurora+
mkmelin
: approval-comm-beta+
Details | Diff | Splinter Review 
I should have added a possibly important point.  I have multiple address books, which appear, and I assume are searched in alphabetical order.  The entry for "Wilber" appears in an address book at the top, which begins with "A".  The entry with "wil" as the nickname appears in an address book further down, which begins with "I".

Perhaps the solution is to allow the user to set the order by which the address books are searched?
See bug 323364, which references several other bugs about list ordering.

I'm not convinced this is a major bug; it sounds like an enhancement request 
to me.
OS: Windows XP → All
Hardware: PC → All
Version: unspecified → Trunk
Severity: major → normal
(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8)
> Gecko/20051111 Firefox/1.5
> Build Identifier: Thunderbird 1.5

This anomoly appeared after I updated to v1.5 Before typing "Jim" followed by the ENTER key produced the ONE email with nickname "Jim". Now my mail is getting  addressed to an email address beginning with "Jim..."  Why on earth would anyone updating the app change this behavior. I can't use nicknames at all now because they DON"T WORK!
> This anomoly appeared after I updated to v1.5 Before typing "Jim" followed by
> the ENTER key produced the ONE email with nickname "Jim". Now my mail is
> getting  addressed to an email address beginning with "Jim..."  Why on earth
> would anyone updating the app change this behavior. I can't use nicknames at
> all now because they DON"T WORK!

I totally agree. I send several email sto the wrong person in the last days.
I changed back to Mozilla Suite, which does not has not this in my opinion  
really serious bug.
Total agreement, this is a serious *regression* from ol'good Mozilla 1.7.
And also from any decent mail client by the way: only an idiot would prefer to have all substring matches from LDAP or collected address book take precedence over the hand-crafted, ultra-short nicknames.

So currently for me Thunderbird is unusable, because of theis very nickname issue.

Since this bug was reported 8 months ago, and judged "just a feature request" by the experts, I'm going to drop Thunderbird, and avise many others to think twice before leaving mozilla.
QA Contact: message-compose
Regardless of whether this is an enhancement request or a bug, I think it is an important issue. If I assign a nickname to an entry in the address book, this means i want to refer to that user by that nickname. Other matches should still show in the autocomplete list, but I do think that the entry with matching nickname should take precedence.
The current TB trunk (3.x) builds, and I believe TB 2.0 have been changing to sort by how much you use particular addresses (essentially "popularity"). IMHO this is far better than sorting by nickname, however Bryan may have some thoughts on this.
Assignee: mscott → nobody
Severity: normal → enhancement
Somewhere I described this example of why nickname should trump popularity:

My mother is named Mary.  I have a friend named Mary.  Mom's AB entry has a nickname "Mom" -- and I write her far more often than I do my friend.  My friend's AB entry has a nickname "Mary".  But if I open a compose window and type Mary, it's my Mom who gets the first match because she's "more popular."

That's just wrong.  Nicknames should take highest priority.

xref bug 331817
Status: UNCONFIRMED → NEW
Ever confirmed: true
Currently "nickname" in the address book does not work. I am really annoyed that this bug was never considered serious enough for someone to fix. Until it is, someone should ELIMINATE the nickname option.  It is dangerous for people typing a "nickname" in the address "To" assuming it will actually work.
Attached patch patch (obsolete) — — Splinter Review 
set PopularIndex based on the different matching case.
Assignee: nobody → brian.lu
Attachment #342561 - Flags: review?(bugzilla)
Attachment #342561 - Flags: review?(bugzilla) → review-
Comment on attachment 342561 [details] [diff] [review]
patch

Sorry, this just isn't the right fix.

One of the problems is it ignores the popularity index (i.e. overwrites it).

I think what we really need here for this bug, is to consider our current algorithms and how we can improve them. I'm hoping Bryan (our User Experience guy) can set up a discussion on that.
This sort order is not going to work effectively across all the different ways that people are searching for others.  One type of use requires the nickname is more important and the other type requires first or family names.

The real goal here is that when a user types 'wil' into the auto-complete for the first time we show all the possible matches on different (and relevant) fields for that term.  When the user selects the desired person from the auto-complete list Thunderbird associates the term 'wil' with that person.

This idea is called 'frecency', by the Firefox team, which was built into the FF3 awesomebar auto-completion.  It gives the user a sense of the application learning their habits and allows them to find the same person by the same string over and over again.

We need to have a similar algorithm to this built against our contact database.  I believe Gloda already has an idea of 'frecency' built into it, however we need to keep tweaking the data points it uses.

https://developer.mozilla.org/En/The_Places_frecency_algorithm

We should probably open a new bug for a frecency index and I recommend against this path.
(In reply to comment #12)
> This sort order is not going to work effectively across all the different ways
> that people are searching for others.  One type of use requires the nickname is
> more important and the other type requires first or family names.
I don't understand when first or family names should ever be higher precedence than nickname. As I understand it, nickname should override Thunderbird's "frecency"; it essentially indicates that a user wants to always associate a specific word with a contact and that this word should always be used to gain quick access to this contact.
(In reply to comment #11)
> (From update of attachment 342561 [details] [diff] [review])
> Sorry, this just isn't the right fix.
> 
> One of the problems is it ignores the popularity index (i.e. overwrites it).
> 
searching "PopularityIndex" in mxr.mozilla.org, I can't find anywhere else that use it expect here. What's the meaning of " PopularityIndex"?  Isn't it how "popular" the AB item is?
(In reply to comment #14)
> (In reply to comment #11)
> > (From update of attachment 342561 [details] [diff] [review] [details])
> > Sorry, this just isn't the right fix.
> > 
> > One of the problems is it ignores the popularity index (i.e. overwrites it).
> > 
> searching "PopularityIndex" in mxr.mozilla.org, I can't find anywhere else that
> use it expect here. What's the meaning of " PopularityIndex"?  Isn't it how
> "popular" the AB item is?

Its values are set up in nsMsgCompose.cpp it is designed to be incremented each time the user sends a message to a contact. There are problems with it (for instance it just increments, there's no frequency option in there) which are covered in different bugs. The only actual use currently is in the autocomplete code.
Just something to try to drag this back to fundamentals here.

As a Luddite who just the other day "upgraded" from Netscape Mail 7.2 to Thunderbird, this bug is the first thing I noticed (yes, it is a bug, in the category of "something that used to work and is now broken").  To be fair, I was happy to discover just how similar Thunderbird was to my old favorite clunker 7.2.  But this bug drives me crazy.  It is *way* too easy to improperly address email by mistake.  No, I don't want the software "thinking" for me about who "pete" is based on popularity or the alphabet or anything else.  I know who "pete" is, I have told the software who "pete" is, and that is what matters.

I would say that the only reason that there hasn't been more noise about this is because it is not obvious to the average user (like me) just how one goes about complaining about something like this.  It took me a long time to find this forum.  But here are some others I found along the way that show we-who-would-like-this-fixed are not outliers.

http://fixunix.com/mozilla/435688-nickname-priority.html
http://forums.mozillazine.org/viewtopic.php?f=39&t=371872

Thanks for listening.
Keep on shouting until someone fixes this bug.  It's bad design to offer nicknames, then place them bottom priority in the UI. I sent mail to the wrong person without noticing the bug when I switched to Tbird. Nearly went back to my previous mailreader and can't believe it's still unresolved!

You couldn't have said it better:  "... this bug drives me crazy.  It is *way* too easy to improperly address email by mistake.  No, I don't want the software "thinking" for me about who "pete" is based on popularity or the alphabet or anything else.  I know who "pete" is, I have told the software who "pete" is, and that is what matters."
I just installed TB 3 and because the address popularity counts seem to have been cleared out in the process, the lack of nickname precedence is even more noticeable - I may just have to turn off autocomplete altogether so I stop embarrassing myself with messages sent to the wrong recipient. :)  It would be great to see a fix for this soon.  Bug 530865 seems to be related.  Thanks!
(In reply to comment #17)
> Keep on shouting until someone fixes this bug.

Please do not, this is not how effective communities can work.  I can understand wanting changes to happen (I have bugs in Firefox that aren't fixed) but working together as well behaved adults is what makes progress.


If we had a patch that merely weighted the nickname matches above the popularity index then I think that would be an acceptable update to our current system.  I believe a frecency index would be a much better value to use however that's for another bug to tackle.
@Chris, thanks I hadn't seen the seamonkey version, bug 530865
An earlier commenter said they were a Luddite who only recently upgraded to Thunderbird 3.  Count me among the same crowd.  I had been using a shareware mail client from roughly 2002.

That client allowed me to address an e-mail by typing a comma-delimited list of nicknames into any of the relevant fields (TO, CC, or BCC).  Thunderbird also supports parsing a comma-delimited list of email addresses into multiple entries into its addressing user interface.

Expanding on the example above, assume you have nicknames "wil", "jan", and "bob".  You should be able to type "wil,jan,bob" to quickly address a mail to all three.  When focus leaves the field, the parser should expand each based on a nickname match, resulting in three entries in the addressing user interface.

The current behavior will break apart the comma-delimited list into three entries, but the nicknames will be retained (that is, the user interface will literally say "To: wil", "To: jan", and "To: bob").

It seems the current addressing design in Thunderbird is to attempt to resolve the address you're typing before you leave the current text field.  This explains the "(user input) >> (resolution proposal)" user interface gimmick in the text input field.

I think that hitting tab when a proposed resolution is displayed--even if only momentarily or perhaps as an event that fires prior to losing focus--acts as a user selection of the proposed resolution.  The trouble is that parsing of a comma-delimited list is apparently executed after that resolution process.  The resolution process finds no match for "wil,jan,bob" and therefore assumes that represents the totality of the desired input and passes the value over to the comma-delimited parser.

I consider nicknames to be a special case of address resolution.  As the proponents above have said, at the least nicknames should be treated as the #1 priority in finding a matching address.  But I would recommend going further: namely, nickname resolution should occur -after- the comma-delimited parsing of the user's input occurs.
To repeat Brian's perfect comment after months patiently waiting for a solution:
"You may not think this is important, but anything that causes mail to get sent
to the wrong user inadvertently is just about he worst possible problem a
mailer can have."
How can this be classified as Enhancement.
https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance
It is certainly a loss of function, so it's somewhere between Normal and Major. (and I say Major).
Other than soliciting votes (which I have done in Moz community support) how can one get the impotance correctly defined?
(In reply to comment #12)
> This sort order is not going to work effectively across all the different ways
> that people are searching for others.  One type of use requires the nickname is
> more important and the other type requires first or family names.
>
What other use of nicknames is there other than addressing messages?  I don't follow the notion that one might want another field to have higher priority in this context.
I've inquired about getting this bug reclassified as a loss of functionality:

http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/11146031bdc47ecd#

Chris
Who experience this regression in TB3 should have look into bug 497722.
bug 497722 is not the same as this bug.  Further, I installed 3.01, and bug 497722 is not fixed.  Entering a letter into the "to" field still produces random results.
(In reply to comment #28)
> bug 497722 is not the same as this bug.

Correct.

> Further, I installed 3.01, and bug
> 497722 is not fixed.

Yes it is not fixed in 3.0.1 as it was fixed after 3.0.1. It will be fixed in 3.0.2 (not released yet).
My brief testing indicates that trunk build 3.2a1pre has fixed this bug.  The "to" field now fills in like it did under Tbird 2.0.
This bug is still present as of 3.1.7 - ie. Use of a unique nicknames doe not result in the associated email address appearing as the first item in the list of suggested email addresses after the autocomplete function runs. I suggest tha in the tab:

  Tools -> Options -> Composition -> Addressing:

an option of the form:

  [ ] Search Nickname field first? 

..  be added, and of course the code updated to work that way.
I agree this is a serious misbehavior and as far as I can tell (from the end-user point of view, not from source), this is a regression.  In Thunderbird 2, if I typed "Bob", and the auto-completion served up a name I did not prefer, I could go into the address book, add "Bob" as the nickname for my favorite Bob, and thereafter typing "Bob" would *always* complete to the desired one.  When I moved to Thunderbird 3.1, this was lost.  (Still true as of 3.1.7)

I don't know what other purpose nicknames even serve, if not to be used for completion.  

As far as I can tell from reading the various here-linked bugs on auto-completion order, none of them address the apparent regression in nickname handling.

(I think a future change to use frecency might work out, and one might argue that a nickname should give weight in frecency much like bookmarking a URI might, but given this is seems to be a regression, and it's serious enough (misdirected email), I would hate to see the fix for this entangled into a larger issue of frecency.)
6 years is far too long to either:
(a) Remove the nickname from the email options
(b) Make it work (when I type Ann followed by a carriage return, it automatically fills in the address for my nickname Ann - and no other!
Just sent an email to the wrong person due to this bug.  Here's the situation:

I have the following two people in my Personal Address Book:
First: Zhichao
Last: Zhang
Display: Jack Zhang
Nickname: Jack
Email: zzc@company1.com

First: Jack
Last: Lastname2
Display: Lastname2, Jack
Nickname: (blank)
Email: Jack.Lastname2@company2.com

My email address is erich@company1.com (same company as first Jack)

I occasionally send email to Jack from Company 2, but (obviously since I've added a nickname for Jack number 1) when I type "Jack" in the To: field, I want email to go to the Jack in my own company.  As you can see, he has taken an American name (Jack) to go by instead of his real name (Zhichao).

Well, when I type "Jack" in the To: field, Thunderbird only gives me one option, Jack from Company 2!  If I want to send email to Jack #1 I have to type "zh" and select his name from the multiple choices, or type "zzc" and his email is the only choice.  It seems Thunderbird completely ignores the nickname, as it doesn't even offer Jack Zhang.

Ok, further testing reveals if I type slowly ("Jac", then wait) both Jacks are provided as options.  If I then further type the "k", both Jacks remain as options for me to choose from.

I would like AT A MINIMUM for both Jacks to be options if I type "Jack" quickly, and of course I agree with everyone above that Jack Zhang should be the first option when typing "Jack".
Isn't there any way to get this bug fixed?  It's so obviously idiotic, and leads to such embarrassing and/or inconvenient results.

Help!!!!
(In reply to Tom Kreutz from comment #36)
> Isn't there any way to get this bug fixed?  It's so obviously idiotic, and
> leads to such embarrassing and/or inconvenient results.
> 
> Help!!!!

It would be nice if Mozilla Messaging provided a convenient way to crowd-fund specific features. Something simpler than kick-starter and perhaps connected to bugzilla.
See Also: → 295428
Summary: Nickname not highest precedence for matching address book entries → Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries [To, CC, addressing field/area]
Whiteboard: [patchlove]
(In reply to Brian Hauer from comment #21)
> Expanding on the example above, assume you have nicknames "wil", "jan", and
> "bob".  You should be able to type "wil,jan,bob" to quickly address a mail
> to all three.  When focus leaves the field, the parser should expand each
> based on a nickname match, resulting in three entries in the addressing user
> interface.
> [snip]
> I consider nicknames to be a special case of address resolution.  As the
> proponents above have said, at the least nicknames should be treated as the
> #1 priority in finding a matching address.  But I would recommend going
> further: namely, nickname resolution should occur -after- the
> comma-delimited parsing of the user's input occurs.

+1 to all of comment 21 and the underlying understanding of nicks as 1 on 1 aliases. However, focus in this bug is on giving resolved (single) nicks precedence in autocomplete results. So resolving multiple comma-separated nicks is probably beyond the scope of this bug. More reminiscent of (but perhaps not the same as) bug 295428. I'd recommend filing a separate bug for resolving a comma-separated list of nicknames.
(In reply to Thomas D. from comment #38)
> So resolving multiple comma-separated
> nicks is probably beyond the scope of this bug. More reminiscent of (but
> perhaps not the same as) bug 295428. I'd recommend filing a separate bug for
> resolving a comma-separated list of nicknames.

OT: Resolving other comma-separated things on recipients line, see also Bug 528503, bug 392932.
(In reply to Thomas D. from comment #38)
> So resolving multiple comma-separated nicks is probably beyond the scope of this
> bug. I'd recommend filing a separate bug for resolving a comma-separated list of
> nicknames.

OT: also about resolving multiple comma-separated nicks on one line:
https://getsatisfaction.com/mozilla_messaging/topics/five_reasons_i_downgraded_to_v2
https://getsatisfaction.com/mozilla_messaging/topics/address_entry_has_gotten_worse_with_new_updates
and probably more
This bug (and related bugs on nicknames) are supported by plenty of complaints at getsatisfaction:
265 topics found for nickname

https://getsatisfaction.com/mozilla_messaging/searches?query=nickname&x=1&y=3&style=topics

I stopped tagging gs reports after the first 2 or 3 result pages.
URL: http://getsatisfaction.com/mozilla_me...
Whiteboard: [patchlove] → [gs][patchlove]
See also:

Bug 295428 - Names and nicknames should still work in address book without autocomplete

Bug 118624 - AB quick search and contacts side bar does not search/match Nickname field, but should (ux-inconsistent with autocomplete)

Bug 400713 - Allow showing any fields from address book as columns in compose message contacts pane (sidebar) [column picker, customize, sorting by organization, nickname etc.]

Bug 247335 - Add separate "First Name", "Last Name" and "Display Name" fields to address book column picker for optional display in AB list pane
See Also: → 400713, 247335, 118624
How does such an obvious bug get filed and debated for year (2013), after year (2012), after year (2011), after year (2010), after year (2009), after year (2008), after year (2007), after year (2006)... but never actually fixed?

This worked in Thunderbird 2, which means it's a regression bug, not an enhancement.  There has been ample evidence that this seemingly-minor issue is a major source of frustration to many users, even to the point of abandoning Thunderbird entirely.  It may be a little thing, but that doesn't mean it doesn't matter.

Instead of debating the best algorithm for years, why not at least add a special-case check for an exact match of a unique nickname, before going into the current algorithm?  That would satisfy the critical use case, making most of the frustrated users happy.  A special-case should be trivial to add, for anyone who knows the code, so why hasn't that much been done after nearly 8 years of this ticket being marked as NEW?
(In reply to Deven Corzine from comment #43)
> How does such an obvious bug get filed and debated for year (2013), after
> year (2012), after year (2011), after year (2010), after year (2009), after
> year (2008), after year (2007), after year (2006)... but never actually
> fixed?

Deven, thanks for your feedback. I appreciate the frustration, as I've often felt it myself (being an active volunteer TB QA/UX contributor since 2007). However, we need to face the fact that TB has a track record of unresolved bugs for a variety of reasons. Where this bug is just one out of many, and the choice of "important" bugs to start from is pretty vast.

TB is now maintained almost exclusively by volunteers, so getting this bug fixed requires a volunteer who is a) able, b) willing, and c) has the time to actually fix this, and d) an agreement about the exact goal of this bug, the desired UX how it should be fixed. a) and c) are the hard parts, whereas I'd consider d) a non-issue in this case (although looking at the entirety of comment 0, to which comment 12 responds, I suspect misunderstandings about d) also contributed to the long latency of this bug).

> This worked in Thunderbird 2, which means it's a regression bug, not an
> enhancement.

Deven (and others before), thanks for pointing that out. Understanding the nature of a bug and documenting that as precisely as possible in the bug report is crucial.

Indeed, this looks like a bug because it violates the very purpose of the nickname design, which is a 1 on 1 relationship between the (ideally unique) nick and the card, to allow user to retrieve and distinguish that card efficiently and consistently from all other cards. (As a caveat, note that TB is not enforcing uniqueness of nicks!). As has often been said in this bug, users correctly expect from the intrinsic purpose of nickname design, that nicks should have absolute priority in searches, which includes priority over frecency. Frecency is a way of automatically establishing dynamic nicks, but certainly nicks that have been purposefully and explicitly defined by user must take precedence over that (why should I add "wil" as a nick to my uncle's card if I don't want "wil" to find that very card?). Users who want the frecency algorithm to take highest precedence probably wouldn't (and shouldn't) define explicit nicks.

Due to violation of these imo correct, established, and widely-evidenced user expectations, this is a bug in terms of UX error prevention where users can easily inadvertently send the msg to the wrong recipient(s), which is a major violation of their privacy. As such, this bug also undermines UX trust in TB as an application, as repeatedly stated here in line of comment 0 and comment 43:

> You may not think this is important, but anything that causes mail to get sent to
> the wrong user inadvertantly is just about he worst possible problem a mailer can have.

> There has been ample evidence that this seemingly-minor issue
> is a major source of frustration to many users, even to the point of
> abandoning Thunderbird entirely.  It may be a little thing, but that doesn't
> mean it doesn't matter.
> 
> Instead of debating the best algorithm for years, why not at least add a
> special-case check for an exact match of a unique nickname, before going
> into the current algorithm?  That would satisfy the critical use case,
> making most of the frustrated users happy.  A special-case should be trivial
> to add, for anyone who knows the code, so why hasn't that much been done
> after nearly 8 years of this ticket being marked as NEW?

See my remarks further up in this comment. Getting bugs fixed in TB is often less trivial than it looks, for a variety of reasons... 

FTR: I agree that the obvious solution to this bug (as required by the intrinsic principles of "nicks" design concept) is to list fullstring matches on nicks at the very top of the recipient autocomplete dropdown, so that they take highest precedence (even before frecency if we implement that). Btw FF works the same way for keywords: Keywords (not tags, nor frecency) take absolute highest priority over any other matches. Think of keywords and nicks as an ALIAS for the real thing.
Severity: enhancement → major
Keywords: privacy, ux-error-prevention, ux-trust
(In reply to Deven Corzine from comment #43)
> This worked in Thunderbird 2, which means it's a regression bug, not an

Fwiw, notwithstanding the annoyingness of this bug, it is NOT a regression.
I just tested this on TB version 2.0.0.24 (20100228), and nicks do NOT get absolute precedence in composition's recipient autocomplete (but it's easy to get fooled into thinking they do).

I tested with this scenario (basically):

Card1:
firstname: n1n
email: asdf@asdf.com
nick: n2n

Card2:
firstname: foo
email: asdf@asdf.com
nick: n1n

STR
1 compose new msg
2 in TO recipient field, type "n1n"

Actual Result:
- card 1 is at top of autocomplete list
- ...

Expected Result:
- card 2 having fullstring match of nick should be at top of autocomplete list
(In reply to Thomas D. from comment #45)
> (In reply to Deven Corzine from comment #43)
> > This worked in Thunderbird 2, which means it's a regression bug, not an
> 
> Fwiw, notwithstanding the annoyingness of this bug, it is NOT a regression.
> I just tested this on TB version 2.0.0.24 (20100228), and nicks do NOT get
> absolute precedence in composition's recipient autocomplete (but it's easy
> to get fooled into thinking they do).
> 
> I tested with this scenario (basically):
> 
> Card1:
> firstname: n1n
> email: asdf@asdf.com
> nick: n2n
> 
> Card2:
> firstname: foo
> email: asdf@asdf.com
> nick: n1n

Fwiw, I actually had these cards in different AB's (card 1 in Personal AB, card 2 in Collected Addresses). It shouldn't matter anyway.
Given that all other nicks out there follow the inherent UX-concept of an ALIAS name or shortcut (where the nick is synomymous with its parent), and given that the FF equivalent of nicks aka "keywords" follow this inherent design principle correctly (nicks getting absolute priority in awesome location bar, even before frecency), this is also an issue of at least external ux-concistency.
Keywords: ux-consistency
Whiteboard: [gs][patchlove] → [gs][patchlove][ux-concept: nick=ALIAS name, like FF keywords]
:aceman just fixed bug 529584 which is also about the result set in recipient autocomplete, so the patch there (attachment 803332 [details] [diff] [review]) might be good starting point to identify the right spot in the code.

And to everyone following this bug, if you have an interest in getting this fixed, even if you're not a code, it would have been a good idea to ensure that it isn't "assigned" to somebody who stopped working on this in 2008, because that will tell any other potential coder that there's nothing to do here because it's already taken by someone... :|
Assignee: eagle.lu → nobody
Depends on: 529584
For non-LDAP cases, I understand that it's this addToResult(...) function which finally sorts the autocomplete results list:

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#276

Seems to first sort on popularity here (after having removed duplicates)...
http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#309

...and then on full addresses here (probably secondary sort):
http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#318
This looks like one of the main places where we call addToResult(...) function (see comment 49).
If that's correct, one way of fixing this bug might be this:

Around here...

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#146
146  let email = card.primaryEmail;
147           if (email)
148             this._addToResult(commentColumn, directory, card, email, true, result);

...perhaps we could get the nick field as we now get the email field, and then add matches of the search fullstring (is that available here?) against nick to a separate result array, say resultTop.
Then merge resultTop Array with result array.

We also need to sort multiple nick matches on something (because unfortunately it's possible to have multiple cards with same nick).
(In reply to Thomas D. from comment #50)
> This looks like one of the main places where we call addToResult(...)
> function (see comment 49).
> If that's correct, one way of fixing this bug might be this:
> 
> Around here...
> 
> http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/
> nsAbAutoCompleteSearch.js#146
> 146  let email = card.primaryEmail;
> 147           if (email)
> 148             this._addToResult(commentColumn, directory, card, email,
> true, result);
> 
> ...perhaps we could get the nick field as we now get the email field, and
> then add matches of the search fullstring (is that available here?)

fullString (user's search words) are not explicitly available here (only inside the searchQuery argument which is passed to _searchCards function); but I suppose we could easily make it explicitly available by just passing it as a separate argument into _searchCards function:

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#131
> _searchCards: function _searchCards(searchQuery, directory, result) {
-> _searchCards: function _searchCards(searchQuery, fullSearchString, directory, result) {

So then when we call that function, we can just pass fullstring, too:
http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#433
> this._searchCards(searchQuery, dir, result);
> this._searchWithinEmails(emailSearchQuery, fullString, dir, result);
-> this._searchCards(searchQuery, fullString, dir, result);

> against
> nick to a separate result array, say resultTop.
> Then merge resultTop Array with result array.
> 
> We also need to sort multiple nick matches on something (because
> unfortunately it's possible to have multiple cards with same nick).

Perhaps we can get away with the current sorting algorithm of _addToResult.
(In reply to Thomas D. from comment #44)

> users correctly expect from the intrinsic purpose of nickname design,
> that nicks should have absolute priority in searches, which includes
> priority over frecency. Frecency is a way of automatically establishing
> *dynamic* nicks, but certainly *static* nicks that have been purposefully and
> explicitly defined by user must take precedence over that.

Adding "frecency", the big brother of this bug (but not required to fix this), is
Bug 382415 - Popularity index of autocomplete doesn't honor timeline (use frecency for email contacts)
Blocks: autocompleteFrecency
See Also: → 970456
I just found reference to this bug a few weeks after filing Bug 972690.  I really hadn't thought of nicknames being more important than last names or first names, but as of SM 2.24 there seems to not be any relevance at all between what is entered and what shows up.  The results should at least START with the letters I am typing.  That's how it worked before ( up through SM 2.23).
Blocks: 972690
Aceman, I think this shouldn't be too hard and will be very similar and in same code area as your sophisticated patch in Bug 984875, attachment 8428850 [details] [diff] [review]. My comment 50 and comment 51 also provide a starting point how this could be done. Perhaps the combination of your insight and my starting points might enable you to come up with a draft patch here?
Flags: needinfo?(acelists)
Oh, and somewhat contrary to comment 0, my understanding of this bug is that most importantly, we want to toplist autocomplete results where nickname==searchstring (full string match). Haven't thought about partial nickname matches much, but strongly suspect we shouldn't undermine popularity more than necessary, so just toplisting full string matches would suffice for this bug (ignore other sorting suggestions found in comment 0).
(In reply to Thomas D. from comment #45)
> (In reply to Deven Corzine from comment #43)
> > This worked in Thunderbird 2, which means it's a regression bug, not an
> 
> Fwiw, notwithstanding the annoyingness of this bug, it is NOT a regression.
> I just tested this on TB version 2.0.0.24 (20100228), and nicks do NOT get
> absolute precedence in composition's recipient autocomplete (but it's easy
> to get fooled into thinking they do).
> 
> I tested with this scenario (basically):
> 
> Card1:
> firstname: n1n
> email: asdf@asdf.com
> nick: n2n
> 
> Card2:
> firstname: foo
> email: asdf@asdf.com
> nick: n1n
> 

For reproducing this bug, create two new cards to ensure that both cards have same value for popularityIndex, otherwise the correct card might appear toplisted but for the wrong reasons.

> 
> STR
> 1 compose new msg
> 2 in TO recipient field, type "n1n"
> 
> Actual Result:
> - card 1 is at top of autocomplete list
> - ...
> 
> Expected Result:
> - card 2 having fullstring match of nick should be at top of autocomplete
> list
status-thunderbird_esr31: --- → affected
tracking-thunderbird_esr31: --- → ?
Sorry, the flags should have another comment, so I'll toggle them back and forth once more to get that right.
status-thunderbird_esr31: affected → ---
tracking-thunderbird_esr31: ? → ---
Aceman, Suyash, can you look into this? (See starting points at the bottom of this comment)

STR are found in comment 45, please ignore comment 0 which has a wider scope not applicable for what we want here.

This bug should be fixed *as a matter of urgency* and landed into TB ESR31, because it renders the nickname feature practically useless without insider knowledge and is now more exposed after the landing of autocomplete twin bugs, bug 529584 and bug 558931.

Although some of the adverse user comments on those bugs are just inconsiderate noise and nonsense (even suggesting we should not search email addresses for recipient autocompletion!), for a certain subset of users there's a trend of complaining that it's now harder to find their favourite addresses with very short searchwords of just one or two characters, for which case they'll see more results not returned before, due to new contains algorithm (*foo*). It all depends on search patterns and AB data structure, but I suspect such users are typically older/longterm private users with small address books. In such user's perception (as evidenced on comments), the old beginsWith algorithm, combined with popularityIndex plain counter, was effectively acting as a shortcut system to their favourite recipients, where typing "Ang" *appeared* to reliably return their favorite contact, "Angus Miller". Of course that has never been true and would fail also in TB 24 even for simple scenarios where user might also have frequent conversation with "Angus Johnson", "Angelina", and "Angelo". But in the combination of narrow beginsWith algorithm and popularity Index, that problem was often hidden for small ABs in private use.

So what such users really want is a stable 1:1 shortcut relationship between their short searchwords ("Ang") and their favourite contact associated with that ("Angus Miller"). That's where we fail miserably because of this bug: Traditional nicknames like "Tom" will largely fail because the contact where Nickname=="Tom" will not be reliably toplisted in autocomplete results, so you end up with "Tony Tomson" toplisted instead if he has a higher popularityIndex.

The other bad problem now more exposed and hence urgently in need of fixing in this context (but much harder than this here), is Bug 382415 (replace popularityIndex with frecency algorithm for autocomplete based on user inputs and picked results), for which a temporary simpler improvement might be something along Bug 1058583 (aging algorithm for popularityIndex).

Aceman, Suyash, this bug here should be relatively easy to fix; starting points in code and outline of potential fix found in comment 50 and comment 51. 

(In reply to Thomas D. from comment #54)
> Aceman, I think this shouldn't be too hard and will be very similar and in
> same code area as your sophisticated patch in Bug 984875, attachment 8428850 [details] [diff] [review]
> [©] [details] [diff] [review]. My comment 50 and comment 51 also provide a
> starting point how this could be done. Perhaps the combination of your
> insight and my starting points might enable you to come up with a draft
> patch here?

(In reply to Thomas D. from comment #55)
> Oh, and somewhat contrary to comment 0, my understanding of this bug is that
> most importantly, we want to toplist autocomplete results where
> nickname==searchstring (full string match). Haven't thought about partial
> nickname matches much, but strongly suspect we shouldn't undermine
> popularity more than necessary, so just toplisting full string matches would
> suffice for this bug (ignore other sorting suggestions found in comment 0).
status-thunderbird_esr31: --- → affected
tracking-thunderbird_esr31: --- → ?
Summary: Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries [To, CC, addressing field/area] → Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries, for searchphrase==nickname [To, CC, addressing field/area]
Whiteboard: [gs][patchlove][ux-concept: nick=ALIAS name, like FF keywords] → [gs][ux-concept: nick=ALIAS name, like FF keywords][STR comment 45, ignore comment 0][patchlove]
I have a quite large email adress book : 525 "collected adresses" and 15000 "personnal adresses". My colleagues have a less large email adress book but both my thunderbird and my coworker's thunderbird is badly impacted with the new TB31 email autocompletion behaviour.

- it sometimes does not lead to any result and leaves the typed begining of the email as is, without autocompleting it. Ex : when typing 'secret' and TAB, TB leaves 'secret' instead of completing to 'secretariat@domain.ext'
- it sometimes require to type allmost the whole email before the autocompletion gives a result : we sometime have to type 'secretariat@' so as to get 'secretariat@domain.ext'
- it sometimes leads to bad results. When typing 'secret' TB autocompletes to 'redacteur@domain.ext' for some unknown reason (or because it got the same doamine name ????)

Please revert to previous satisfying behaviour or fix new behaviour !!!
Addition : some emails need to be accessed regularly, but have the same part before the @. Hence, i use the nickname or some free field of the email card in the adress book to as to state a unique nickname or keyword that differenciate each of these adress. Its good that all these fields are searched.
Now I can't send to a list!

Has something changed within the last few days?  I have a mailing list for a weekly club, and each week I send out a notice.  I used to be able to type the start of the name and it would send the email to the list.  Now it says that it is not a valid email address because it is not of the form user@host, so it doesn't send it to the list.  I couldn't figure out any way to send to the list.
When I blanked out the description of the mailing list, then it would send to the list again.
(In reply to yanlu from comment #60)
> I have a quite large email adress book : 525 "collected adresses" and 15000
> "personnal adresses". Thunderbird is badly impacted with the TB31 email autocompletion behaviour :
> - it sometimes does not lead to any result and leaves the input without autocompleting it. 
> - it sometimes require to type allmost the whole email before the
> autocompletion gives a result
> - it sometimes leads to bad results.

Seems fixed in TB 31.1.1 with https://bugzilla.mozilla.org/show_bug.cgi?id=984875 fix.
(In reply to yanlu from comment #64)
> (In reply to yanlu from comment #60)
> > I have a quite large email adress book : 525 "collected adresses" and 15000
> > "personnal adresses". Thunderbird is badly impacted with the TB31 email autocompletion behaviour :
> Seems fixed in TB 31.1.1 with
> https://bugzilla.mozilla.org/show_bug.cgi?id=984875 fix.

Yanlu, thanks for reporting back with user feedback which is generally welcome.
However, the autocomplete problems you mention are not related to this bug, so let's avoid them here. Moreover, let's avoid unspecified general statements like "Seems fixed" which might easily be misunderstood to mean this bug here is fixed, which is NOT the case.
But yes, we're in the process of fixing things in recipient autocomplete; some fixes have already been released, and more pending, e.g. Bug 970456.
Summary: Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries, for searchphrase==nickname [To, CC, addressing field/area] → Recipient Autocomplete: Nickname does not get highest precedence for matching address book entries, for searchphrase==nickname [To, CC, addressing field/area, toplisted, priority, results]
(In reply to j.mccranie from comment #62)
> Now I can't send to a list!

j.mccranie, problem appreciated, but it's not related to this bug so let's be focused. Feel free to file a new bug if this isn't on record yet. But I think I've seen a recent bug for that which I can't find right now.
(In reply to Thomas D. from comment #67)
>> (In reply to j.mccranie from comment #62)
>> Has something changed within the last few days?  I have a mailing list for a
>> weekly club, and each week I send out a notice.  I used to be able to type
>> the start of the name and it would send the email to the list.  Now it says
>> that it is not a valid email address because it is not of the form
>> user@host, so it doesn't send it to the list.  I couldn't figure out any way
>> to send to the list.
> > Now I can't send to a list!
> 
> j.mccranie, problem appreciated, but it's not related to this bug so let's
> be focused. Feel free to file a new bug if this isn't on record yet. But I
> think I've seen a recent bug for that which I can't find right now.

Should have been fixed in TB31.1.1 via bug 1060901.
Flags: needinfo?(acelists)
OT: (In reply to :aceman from comment #68)
> > > Now I can't send to a list!
> > j.mccranie, problem appreciated, but it's not related to this bug so let's
> > be focused. Feel free to file a new bug if this isn't on record yet. But I
> > think I've seen a recent bug for that which I can't find right now.
> 
> Should have been fixed in TB31.1.1 via bug 1060901.

Thanks for the reference on that mailing list bug, and from there, the other one I had in mind was Bug 1008718.
Aceman, Suyash, sorry for nagging, but given the added importance of this bug for the current autocomplete UX, could you answer to my Comment 59?
Flags: needinfo?(syshagarwal)
Flags: needinfo?(acelists)
The infrastructure created in bug 970456 should make it easy to prioritize anything we want. Unless deciding about the priority of each contact is slow and so unnacceptable for large result sets.
Depends on: 970456
Flags: needinfo?(acelists)
(In reply to :aceman from comment #71)
> The infrastructure created in bug 970456 should make it easy to prioritize
> anything we want. Unless deciding about the priority of each contact is slow
> and so unnacceptable for large result sets.

I want to believe so too, but aren't we sorting plain *results* there *after* the fact (aka email addresses, potentially two from each card), whereas for nickname we need to retrieve that at a much earlier point where we still have access to the card (and going back from email to card would be error-prone in current flawed design of AB)? In fact, since a card can have 2 mail addresses, both of them might have to be toplisted in case of nickname match, primary address first.
<rant> Such an incredible waste of resources!  For years, I have puzzled about the use of the "Nickname" field, and now I see that it actually fits into someone's unbelievably obscure search algorithm that nobody in the known world has ever relied upon.  Why anyone would go to all the work described here to organize each card to be searched by these esoteric methods is totally beyond reason.  

All that is needed is a simple alpha index and search.  If the developers are infatuated with exotic search methods to pad their resumes, fine, but leave us real world users with a simple alpha search. Getting carried away with complicated search methods as described above is complete nonsense!</rant end>
José, just because you don't know and/or use nicknames doesn't mean they are obscure, useless or don't make any sense. Particularly for me, nicknames (in most cases just the initials) are the (by far!) preferred and fastest way to enter a recipients address. Just enter two or three letters and pick the adress from the upcoming (very short) list. This has always worked perfectly fine for many years (since early Netscape), until it was broken due to some "enhancements" to the search function. We just want that to be repaired.
(In reply to José Josephus from comment #73)
> <rant> Such an incredible waste of resources! [...] Getting carried away
> with complicated search methods as described above is complete
> nonsense!</rant end>

I've addressed the obvious social and factual shortcomings of that comment via private mail, including a link to https://bugzilla.mozilla.org/page.cgi?id=etiquette.html.

(In reply to Tilmann Reh from comment #74)
> José, just because you don't know and/or use nicknames doesn't mean they are
> obscure, useless or don't make any sense. Particularly for me, nicknames (in
> most cases just the initials) are the (by far!) preferred and fastest way to
> enter a recipients address. Just enter two or three letters and pick the
> adress from the upcoming (very short) list. This has always worked perfectly
> fine for many years (since early Netscape), until it was broken due to some
> "enhancements" to the search function. We just want that to be repaired.

+1. Thanks Tilmann.
We've heard your plea and we're working to repair that in Bug 970456, while preserving the actual "enhancements" of the search functions which we introduced to fix other bugs and cater for legitimate user expections.
All this talk is about the recipient autocomplete in the address widget. All the other searches (contacts sidebar, search in addressbook window and Edit->Search addresses should still sort alphabetically (by the column that is selected)). So no need to worry.
(In reply to :aceman from comment #76)
> So no need to worry.

Are you joking?  None of the fancy nick-name, frequency searches work, the alpha searches produce gibbled results! The sidebar list is out of order.  The whole thing is a broken down, useless wreck!  A simple alpha search on "Display Name" would at least give us something to work with. Start listening with full attention, please!
Then please post it in the relevant bug, not one about autocomplete. Thanks.
I don't think I have to say anything now about this.
Thanks.
Flags: needinfo?(syshagarwal)
Exact match was the change that happened to me at some point in the past year.  I have a few short nicknames like 'jim'. In previous version, typing quickly jim[tab] instantly prefilled from the nickname.  In the newer versions, if there are any addresses in the book starting with jim, the behavior is to put a red 'jim' in the to box, and not allow sending at all.  I fail to see the point of the nickname if it is not used, and one should not have to wait .5s for the autocomplete to prefill to hit tab.
It's become a problem again in TB31.4 (or perhaps a little bit earlier) (with the visibly different compose window)
This should be pretty straight forward to fix, now that bug 970456 is fixed. You'd just adjust the scoring there to be higher if it's a match in nickname.
Mentor: mkmelin+mozilla
tracking-thunderbird_esr31: ? → ---
Whiteboard: [gs][ux-concept: nick=ALIAS name, like FF keywords][STR comment 45, ignore comment 0][patchlove] → [good first bug][lang=js]
I have submitted a new bug 1127258 which describes a new problem when selecting an address in TB31 (31.4); that's two addressing problems introduced with this version of TB which result in emails going to the wong person unless one takes special care.
(In reply to Magnus Melin from comment #82)
> This should be pretty straight forward to fix, now that bug 970456 is fixed.
> You'd just adjust the scoring there to be higher if it's a match in nickname.

Suyash, could you try this? It's pretty important to offer a 100% predictable way of using autocomplete shortcuts... Magnus has volunteered to mentor this ;)
Flags: needinfo?(syshagarwal)
Attached patch Patch v1 (obsolete) — — Splinter Review 
Oh, I thought it was supposed to be for someone looking for "getting started".
If its urgent, here's an attempt.

Thanks.
Attachment #342561 - Attachment is obsolete: true
Flags: needinfo?(syshagarwal)
Attachment #8556897 - Flags: review?(mkmelin+mozilla)
Comment on attachment 8556897 [details] [diff] [review]
Patch v1

Review of attachment 8556897 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/addrbook/src/nsAbAutoCompleteSearch.js
@@ +128,5 @@
>     * Gets the score of the (full) address, given the search input. We want
>     * results that match the beginning of a "word" in the result to score better
>     * than a result that matches only in the middle of the word.
>     *
> +   * @param aCard - The card whose score is being decided

lowercase t?

@@ +139,5 @@
> +
> +    // We will firstly check if the search term provided by the user
> +    // is the nick name for the card or at least in the beginning of it.
> +    let nick = aCard.getProperty("NickName", "");
> +    nick = nick.toLocaleLowerCase();

you can just put .toLocaleLowerCase() after getPropery above

@@ +140,5 @@
> +    // We will firstly check if the search term provided by the user
> +    // is the nick name for the card or at least in the beginning of it.
> +    let nick = aCard.getProperty("NickName", "");
> +    nick = nick.toLocaleLowerCase();
> +    aSearchString = aSearchString.toLocaleLowerCase();

remove the duplication of this below.

@@ +143,5 @@
> +    nick = nick.toLocaleLowerCase();
> +    aSearchString = aSearchString.toLocaleLowerCase();
> +    let nickIndex = nick.indexOf(aSearchString);
> +    if (nick == aSearchString || nickIndex == 0)
> +      return BEST;

You don't need the nick == aSearchString
But, this doesn't quite do what this bug requests. What is requested is that nick should act as a "super-best" match. so the score should be 1 better than if it matched in display name - as is they would be equal (BEST + 1)

Please also adjust the test for this. There should be one test where the nick is the beginning of another persons name.

::: mailnews/addrbook/test/unit/test_nsAbAutoCompleteSearch6.js
@@ +139,5 @@
>      card.setProperty("PopularityIndex", element.popularityIndex);
>      card.firstName = element.firstName;
>      card.lastName = element.lastName;
> +    if (element.NickName)
> +      card.setProperty("NickName", element.NickName);

You don't need the if I think. All the other props are also just set to undefined if not set. Also, please lowerCamelCase NickName
Attachment #8556897 - Flags: review?(mkmelin+mozilla)
Assignee: nobody → syshagarwal
Severity: major → normal
Attached patch Patch v2 (obsolete) — — Splinter Review 
Made the suggested changes.

Thanks.
Attachment #8556897 - Attachment is obsolete: true
Attachment #8557496 - Flags: review?(mkmelin+mozilla)
Comment on attachment 8557496 [details] [diff] [review]
Patch v2

Review of attachment 8557496 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks a lot Suyash for your quick response to my request to pick this up - I am really getting tired of referencing and explaining this bug (and workaround) in other bugs where users have been facing problems due to this major flaw which practically disables the entire nickname functionality for a range of scenarios.

I'll set feedback+ because we're on the right track here; but the devil - as always - in the detail (i.e. this patch as-is won't work yet as expected)...

::: mailnews/addrbook/src/nsAbAutoCompleteSearch.js
@@ +141,5 @@
> +    // is the nick name for the card or at least in the beginning of it.
> +    let nick = aCard.getProperty("NickName", "").toLocaleLowerCase();
> +    aSearchString = aSearchString.toLocaleLowerCase();
> +    if (nick.indexOf(aSearchString) == 0)
> +      return BEST + 1;

(In reply to Magnus Melin from comment #86)
> You don't need the nick == aSearchString
> nick should act as a "super-best" match. so the score should be 1 better
> than if it matched in display name - as is they would be equal (=> we need BEST + 1)

True, but still not sufficient for ensuring absolute top ranking for *full* nickname matches over *partial* nickname matches, e.g.:
Card1: Nickname="foo" (nick==searchphrase; nickIndex==0)
Card2: Nickname="foobar" (nick!=searchphrase; nickIndex==0)
Searchphrase: "foo"

Autocomplete result (wrong):
Card2 (partial nick match)
Card1 (full nick match)

To make the nickname design work reliably, full nickname match must obviously supersede partial nickname match, so we would need to adjust the scoring accordingly:

if (nick == aSearchString)
  return BEST+2;
if (nickIndex == 0)
  return BEST+1;

However, I'm not convinced yet if making initial nickname matches rank higher than anything else will really work out well, e.g.:

Card1: Display Name="Tom" popularityIndex=10
Card2: Nickname="TomFoo" popularityIndex=5
Card3: Nickname="TomBar" popularityIndex=4
Searchphrase: "Tom"

Autocomplete result:
Card2 (partial nickname match, pi=5)
Card3 (partial nickname match, pi=4)
Card1 (full display name match, pi=10)

So I'm not sure if all results with partial matches on nickname should rank higher than full match on display name; should they? (I think not). Worse, PLEASE let's be aware that any forced top scorings that are not based on frecency will further undermine even the current rudimentary frequency implemtation of "popularityIndex":

Card1: Display Name="Tom" popularityIndex=1000
Card2: Nickname="TomFoo" popularityIndex=5
Card3: Nickname="TomBar" popularityIndex=4
Card4: Nickname="Tim" popularityIndex=6
Card5: Nickname="TD" popularityIndex=7
Searchphrase: "T"

Autocomplete result:
(all partial nickname matches top-listed, ordered by popularity:)
Card5
Card4
Card2
Card3
(then, potentially far down the list:)
Card1 (initial Display name match; popularityIndex=1000(!))

So in this case, even for partial nickname matches from single-character(!) search phrase, ALL partial nickname matches will be toplisted (although they are not popular at all, and the partial match on their nicknames is entirely insignificant!), whereas highly popular Card1 with an inital match on Display name will rank lower than all the insignificant matches.

Undermining "frecency"-based sorting more is imo a slippery approach because we'll never find a one-for-all algorithm to predict which address the user actually wants, but "frecency" (linked up with the actual user input) will certainly be one of our best bets. I think. These things are actually pretty complex...

Please correct me if I'm wrong, but I conclude that it'll be safer if we just fix the main usecase of this bug, which is FULL nickname matches; let the rest be handled by our existing algorithm. So I suspect that this should be sufficient:

if (nick == aSearchString)
  return BEST+1;

Also, since full nickname matches (per inherent design) must really always be toplisted without exception (regardless of any other sorting algorithms we might think of), I wonder if we should leave some room in the score for pushing other matches into the "best+" range:

if (nick == aSearchString)
  return BEST+100;

Further technical reviews might want to test and improve performance against large ABs.
Attachment #8557496 - Flags: feedback+
(In reply to Michael Gile from comment #80)
> and one should not have to wait .5s for the autocomplete to prefill to hit tab.

Michael, that's Bug 1012397.
Nit: While we are here, can you please fix this comment:

http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAbAutoCompleteSearch.js#152
> 152     // ":xx:", "jd", "who".
152     // ":xx", "jd", "who".
(In reply to Thomas D. from comment #88)
> Comment on attachment 8557496 [details] [diff] [review]
> Patch v2
> 
> Review of attachment 8557496 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> True, but still not sufficient for ensuring absolute top ranking for *full*
> nickname matches over *partial* nickname matches, e.g.:
> Card1: Nickname="foo" (nick==searchphrase; nickIndex==0)
> Card2: Nickname="foobar" (nick!=searchphrase; nickIndex==0)
> Searchphrase: "foo"
> 
> Autocomplete result (wrong):
> Card2 (partial nick match)
> Card1 (full nick match)

That result can occur e.g. when Card2 has a higher popularityIndex than Card1.
It shouldn't occur because full nickname matches (by inherent design) must supersede any other match, regardless of popularity.
Comment on attachment 8557496 [details] [diff] [review]
Patch v2

Review of attachment 8557496 [details] [diff] [review]:
-----------------------------------------------------------------

Hmm, Thomas has some point. So here's what I think we should do:

if (nick == aSearchString)
  return BEST+1;
if (nickIndex == 0)
  return BEST;

r=mkmelin, with that and test adjusted to pass (if needed)
Attachment #8557496 - Flags: review?(mkmelin+mozilla) → review+
Made the suggested changes.

Thanks.
Attachment #8557496 - Attachment is obsolete: true
Attachment #8558325 - Flags: review+
Attachment #8558325 - Flags: feedback+
Keywords: checkin-needed
(In reply to Magnus Melin from comment #92)
> Comment on attachment 8557496 [details] [diff] [review]
> Patch v2
> 
> Review of attachment 8557496 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Hmm, Thomas has some point. So here's what I think we should do:
> 
> if (nick == aSearchString)
>   return BEST+1;
> if (nickIndex == 0)
>   return BEST;

Cool! Looks like it could be an elegant solution to problems mentioned in my comment 88. Thanks, great teamwork!
I've pushed this as https://hg.mozilla.org/comm-central/rev/5f277274b13f
Status: NEW → RESOLVED
Closed: 10 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Flags: in-testsuite+
Target Milestone: --- → Thunderbird 38.0
Blocks: 1121585
Magnus, can you please set the appropriate flags to get this into a TB release version as soon as possible, even for TB31 if possible?

I'm seeing more and more bug reports where users complain that nickname searches no longer work (they never have, but they appeared to... but now, depending on scenario / search words and data sets, users might see more irrelevant results, also because of irrelevant matches from collected addresses, resulting from our former mis-design (now fixed) to use collected addresses for remote-content permissions).

We should really land all autocomplete fixes fast, to avoid getting more unqualified complaints for things we have already fixed.
Flags: needinfo?(mkmelin+mozilla)
I don't think this is ESR material as it's a new behavior not fixing a regression.
Flags: needinfo?(mkmelin+mozilla)
(In reply to Magnus Melin from comment #98)
> I don't think this is ESR material as it's a new behavior not fixing a
> regression.

In view of 33 votes here and the quantity of bugs reported against autocomplete result sorting algorithm, I find that evaluation nitpicking. While in a strictly technical sense, this might not be a regression, it's actually worse:

1) Once upon a time we delivered a feature which apparently has never worked as intended and required by inherent design. That's a significant failure which shouldn't have to wait any longer when it's finally fixed 9 years after being reported.

2) We have only recently exposed this problem significantly more by expanding the scope of recipient search for certain scenarios. So, as seen from numerous reports, users are now more affected by this design problem, because of the changes made in recent version. Technicalities aside, that looks like a de-facto regression to me (as confirmed by users saying "it used to work until recently but now it fails").

I don't see any risks coming from this straightforward patch, except that we're undermmining popularityIndex a bit more again, but this works on exactly the same terms and conditions as Bug 970456 so there's nothing new here.

So I still think this 5-line-patch should land on ESR.
It is important to be nit-picky when one is in the role of deciding what to take on ESR, lest we get excessivly loosey goosey with our standards - which are looser than Firefox's, and thus have grey areas which necessitate judgement calls ultimately made by one person. 

But I support the request for ESR uplift for one reason only, if comment 97 paragraph 2 is correct and the current net effect behavior to the user is it *appears* to be a regression - that's all that matters.  So if patch has no known, and little to no likelihood of having, side effects, then I think we should take it. (Also, we have taken patches on ESR that do not fix recent regressions.)
As one of the affected users, I want to plead with whoever has the clout to get this fixed as soon as, I find myself misaddressing messages practically every day since the latest release, which to me feels like a crime, to be honest it feels to me like something which needs fixing between releases

Going through the history of this, I spotted somone else's comment that the word nickname appeared on no less than 265 different threads on the getsatisfaction system, which sure sounds to me like a lot of people care (and will currently be in a state of disappointment)

Is there some sort of QA postmortem going on as to how this managed to get released in such a poor state?  I don't mean to express (or provoke) anger with this point, I just think it's really useful to contemplate how the organisation has managed such a cock-up and how not to do it again.

Would it help if I volunteered to evaluate what is now proposed?
(In reply to steve hayes from comment #101)
> Going through the history of this, I spotted somone else's comment that the
> word nickname appeared on no less than 265 different threads on the
> getsatisfaction system, which sure sounds to me like a lot of people care
> (and will currently be in a state of disappointment)

I'm sorry to report that Getsatisfaction search (as well as the current SUMO search) is notoriously bad about giving false search hits "just because".  So I'd doubt as many as 50 getsatifaction threads were solid hits about nickname autocomplete.  On the current SUMO it is maybe about a dozen. 

> Is there some sort of QA postmortem going on as to how this managed to get
> released in such a poor state?  I don't mean to express (or provoke) anger
> with this point, I just think it's really useful to contemplate how the
> organisation has managed such a cock-up and how not to do it again.

The issues that got us here are pretty well understood by those involved. But probably the biggest hit to the end result (besides lack of personnel and competing priorities) was the lack of automated tests. Automated tests are in effect a spec - they define how the function should behave. Without automated tests, it's a crap shoot to know whether someone's new code functions worse, or better, than the old code. But lack of tests is not the fault of the current developers.

> Would it help if I volunteered to evaluate what is now proposed?

If you wish to help, there are many opportunties to give of your time (and time is what we most need). https://wiki.mozilla.org/Thunderbird#Contributing
As seen in the past every minor change might get someones use-case slightly wrong.
We have usually let patches bake through aurora+beta before esr, but with 38 coming up soonish I think what gets into 31esr should also be stricter.

Based on other reports, there may be something fishy happening regarding popularityindex, but I haven't had time to investigate that yet.
Comment on attachment 8558325 [details] [diff] [review]
Patch for check-in

Review of attachment 8558325 [details] [diff] [review]:
-----------------------------------------------------------------

We should land this for aurora+beta.
Attachment #8558325 - Flags: approval-comm-beta+
Attachment #8558325 - Flags: approval-comm-aurora+
Magnus, thanks for approving this for beta and aurora.

(In reply to Magnus Melin from comment #103)
> As seen in the past every minor change might get someones use-case slightly
> wrong.
> We have usually let patches bake through aurora+beta before esr, but with 38
> coming up soonish I think what gets into 31esr should also be stricter.
> 
> Based on other reports, there may be something fishy happening regarding
> popularityindex, but I haven't had time to investigate that yet.

Well, apart from the fact that popularityIndex is broken by design (because it just counts up forever so doesn't consider recency), the "fishy" thing which happened to popularityIndex is that your patch with new scoring algorithm in Bug 970456 effectively disables it for an unpredictably large number of usecases where search string happens to match the visible results strings in a wordwise manner, more so for matching the beginning of the visible result string, which is typically the display name. So no matter now popular certain results are, they can never get to the top.
No longer blocks: 1121585
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: