1012397 - PostSearchCleanup improvement needed - Address doesn't auto complete if you tab too quickly. NO address filled in.

Status: VERIFIED FIXED

(Toolkit :: Autocomplete, defect)

Description

[Jonathan Kamens]

In current thunderbird trunk, but NOT in 24.5.0, if you type the first name of one of your contacts in the To field of the message compose window and then hit tab quickly, it doesn't complete. In 24.5.0, no matter how quickly you hit tab it completes successfully. The latter behavior is obviously preferred. ;-) This is a regression.

Comment 1

[Magnus Melin [:mkmelin]]

Probably from the toolkit autocomplete switchover, bug 959209?
Tabbing to complete has some problems, but I have a patch that should fix those.

Comment 2

[Magnus Melin [:mkmelin]]

Still there, but you have to be quick enough for autocomplete not to find any matches. 
Autocomplete may be somewhat slower since we now search more fields (bug 529584).

Comment 3

[Wayne Mery (:wsmwk)]

perhaps will be fixed by bug 970456?

Comment 5

[Magnus Melin [:mkmelin]]

Unfortunately I don't think so. I must have forgot to note it here, but tested this and it's from the toolkit autocomplete switchover, bug 959209.

Comment 10

[Nicola Ferralis]

(In reply to Magnus Melin from comment #2)
> Still there, but you have to be quick enough for autocomplete not to find
> any matches. 
> Autocomplete may be somewhat slower since we now search more fields (bug
> 529584).

Are we to assume (and therefore live with) the slower autocomplete to just remain as is, because oft he search within more fields? I would prefer a search to a lower number number of fields but a much faster response. As it is now, it's super-slow, and it affects quite significantly my workflow (i.e. I can't wait for autocomplete to do its thing).

Comment 11

[Beau James]

Since 1048541 was duplicated to this bug, I want to highlight that there is a second part to this problem. If you type "too fast", the address may not autocomplete at all. But it may also autocomplete incorrectly, because it processes only a portion of the characters typed. The "end-of-field" event (TAB or ENTER) gets processed ahead of characters that were typed into that field.

The attachment in bug 1048541 includes screen clips showing both behaviors.

Comment 12

[Thomas D. (:thomas8)]

(In reply to Magnus Melin from comment #2)
> Still there, but you have to be quick enough for autocomplete not to find
> any matches. 
> Autocomplete may be somewhat slower since we now search more fields (bug
> 529584).

Not exactly, we're still searching exactly the same fields as before (and we really can't reduce the number of fields because it's only the most important ones which are absolutely required for name searches). However, {beginsWith "foo bar"} search pattern was replaced with {contains "foo" AND "bar"} in the twin set of bug 529584 and bug 558931 (and let's please always mention them together, just one isn't enough), so we are now comparing separate words against more content in the same fields.

Analysis by Aceman found that just to fire the c++ query, it always takes a noticable minimum of time (almost 2 secs iirc?) regardless of how much data is searched. Furthermore, if you only type a single character, it will search for that single character anywhere in the selected fields which might perform a little less fast and yield many results compared to quickly typing 3 or more characters. The old search algorithm was wrong and incompletely designed, i.e. failed for a lot of everyday scenarios even for searching the beginning of words. I'd much prefer an extra (split) second over missing results, where not finding these or hunting for them with other methods will consume a lot more time... So from that perspective, the new search is more successful and hence can even be said to be faster.

Also, let's not get tricked by personal impressions. I'll only believe actual measurements that mention all the conditions, computer power, database size, data structure, number of matches, length of search words, speed of typing and pauses (yes, we use different code pattern to reduce the first result set when you type on), popularity index etc. There's a lot of factors that influence this.
It's very possible that bug 529584 and bug 558931 are just exposing structural scalability/efficiency problems in the code that went unnoticed so far. The amount of data searched in MB is ridiculously small for most users, so it should really be fast on recent computers. If it's not as fast as we'd hope, we should seek to optimize the runtime speed of the algorithm. UX-wise, the new algorithm is a huge improvement making search much more powerful and efficient. If you quickly type two reasonably unique bits of your contact say with 3 or four letters each, you'll filter out exactly the contact you're looking with much more precision and less hassle than before - don't forget visually parsing and manually picking from long results list also takes time...

Comment 13

[Thomas D. (:thomas8)]

What's really missing is documentation to educate our users about the new search algorithm and how to use it most effectively and efficiently.

Comment 14

[Randy Syring]

What's really missing is the ability to type and tab and have the result there just like it was before.  :o/

I think it's cool that the search works better now.  However, that affects about 10% of my use case.  The other 90% of the time, my result should be the first match based on a single word search (user's first name).  And, 90% of the time now, I tab away before the UI can catch up.

Info you request:

Hardware: latest macbook pro, running ubuntu linux, 16GB memory, 1TB SSD PCIe flash drive, 3Ghz i7
database size: ~300 addresses
data structure: not sure how to answer this
number of matches: doesn't seem to matter, have the problem whether there would be 1 or 10 matches.
length of search words: in the use case I care about, usually only a single word representing a persons first name, so typically somewhere between 4 and 10 characters.
speed of typing: fast, probably 90 WPM+ for the first name of someone I'm used to typing
pauses: none, I want to type the first name, hit tab immediately and have Thunderbird use the first result.  Over time, b/c Thunderbird counts usage and sorts accordingly, this is right 90%+ of the time for me.  I don't personally care if TB takes another .5 secs to finish the search and display the email, I just want it to be there.  Right now, I'm left with whatever letters I typed and no auto-complete.

Comment 15

[Jonathan Kamens]

(In reply to Randy Syring from comment #14)
> What's really missing is the ability to type and tab and have the result
> there just like it was before.  :o/

Yes, yes, yes, a thousand times yes.

I run into this ALL THE FREAKIN' TIME.

I don't know what we were and weren't searching before, or how we were doing the search before. What I do know is that whatever we were searching before, and however we were searching it, WORKED JUST FINE FOR ME, with the added bonus that when I tabbed, I got a match, no matter how fast I tabbed.

The new behavior is just broken. Period, full stop, broken.

Comment 16

[Thomas D. (:thomas8)]

In reply to comment 14, comment 15:

Sorry gentlemen, my bad, comment 12 is somewhat off-topic/peripheral to this bug. So rest assured,
I'm not questioning the existence of this particular bug in any way, and it should be fixed.

Comment 11 looks like a good analysis/starting point to fix it.

Comment 17

[Randy Syring]

(In reply to Thomas D. from comment #16)
> In reply to comment 14, comment 15:
> 
> Sorry gentlemen, my bad, comment 12 is somewhat off-topic/peripheral to this
> bug. So rest assured,
> I'm not questioning the existence of this particular bug in any way, and it
> should be fixed.
> 
> Comment 11 looks like a good analysis/starting point to fix it.

Whew...thanks for clarifying.  :)

Comment 19

[Nicola Ferralis]

My system: Mac Air, 1.gGHz corei7 4GB, running OSX 10.9.4. I have 580 contacts in my Address book. It takes about 2 seconds to autocomplete. Two seconds are not "a split second" but a noticeable amount that significantly affects my workflow. Besides, as stated above, why the new algorithm cannot remember the most used autocomplete entries? I type several emails during my day to the same recipient, and I need to wait still for 2 secs. To me the algorithm should not just be better, but also smarter.

Comment 20

[Thomas D. (:thomas8)]

(In reply to Nicola Ferralis from comment #19)

OT: Nicola, thanks for sharing your user experience. 

> My system: Mac Air, 1.gGHz corei7 4GB, running OSX 10.9.4. I have 580
> contacts in my Address book. It takes about 2 seconds to autocomplete.

What exactly (how many characters) did you type, and how fast (with or without pause)?

> Two
> seconds are not "a split second" but a noticeable amount that significantly
> affects my workflow.

Agreed :) For everyone around here concerned about autocomplete speed: Rest assured; we're fixing it. I wasn't aware that Bug 984875 hasn't landed yet, which comes with an 83% performance gain for autocomplete. So that should bring Nicola's search which now takes 2 seconds down to the split second I mentioned. See Bug 984875 Comment 36.

> Besides, as stated above, why the new algorithm cannot
> remember the most used autocomplete entries?

Well it does, sort of, with a counter called popularityIndex. However, it's a bit old and not designed for present-day expectations, where we really want a proper algorithm of "frecency" (frequency and recency) as in FF. I'm one of the strongest advocates to have "frecency" implemented into TB address autocomplete (Bug 382415); unfortunately, I can't fix it myself so we're looking for another volunteer contributor like you to fix this. Remember TB is developed by the community because Mozilla largely withdrew their funding.

> I type several emails during my
> day to the same recipient, and I need to wait still for 2 secs.

Searching MRU contacts before anything else sounds like an idea worth exploring (while tricky and resource-eating, too); if someone wants to file a bug for that and link to the frecency bug, I'm fine with that.

> To me the
> algorithm should not just be better, but also smarter.

+1 :) As I said, we have several ideas on record, but developer manpower is the main issue...

> I type several emails during my day to the same recipient

Nicola, for that scenario, here's another trick for you:

You want a stable relationship between the searchword you type and that particular result, isn't it? For that very scenario, TB has a feature called "Nickname".

Go into the properties of that "same recipient" which you email frequently, and add a unique value into the nickname field. So if it's your friend Tom, you could add something like "ttt", "Tom#", or ":Tom" etc. (Due to Bug 325458, it's currently not enough to just pick a nickname which is unique out of all nicknames, so you have to make sure there's no other contact in all of your ABs whose properties contain Tom's unique nickname in any of their fields which autocomplete searches. Plus there's a couple of other bugs around nicknames...)

Now and everafter, whenever you type "ttt" somewhat quickly without pause (!), you'll get Tom's contact as a unique match in recipient autocomplete (and if you type fast enough, that result will come faster, too). Please report back if this works for you! :)

Comment 21

[Thomas D. (:thomas8)]

And everybody who'd like to see "frecency" algorithm implemented into TB's recipient autocomplete, so that TB automatically learns to toplists your most used contacts after you've typed those two or three memorable characters, please vote for bug Bug 382415 using bugzilla's vote link... :)

Comment 23

[Beau James]

Speeding up the address completion algorithm is a "good thing", to be sure, but is really not a solution to the fundamental UI behavior issue here. The fundamental issue is that user input events are not being processed in the correct order. The "end-of-field/move-to-next-field" input event is jumping ahead of "character-input-into-field" events. Speeding up the per-input-character processing will at best only mask the problem - sometimes.

In v31.0, a user typing a "move to next field" - TAB or ENTER - "too quickly" into an address field can result in several different behaviors:

(1)All characters typed are in the field, no address completion at all
(2) Address completion has processed only a portion of the characters typed. Remaining characters in the input stream intended for that field are discarded, field is filled with whatever address the algorithm has selected as the most probable match for the characters that were processed. This is almost NEVER the address that the user intended to go into that field.

From a user's point of view, the second behavior is much worse. The first behavior typically results in an invalid email address, which gets detected later e.g. when the message is sent. The second behavior results in a valid but incorrect email address in the distribution list. If not noticed, that may result in the message being delivered to a completely inappropriate recipient!

The behavior that the user expects is that input events are processed in sequence. Consume the "end of field/move to next field" event only after consuming all of the "character input" events in that field.

Comment 24

[Thomas D. (:thomas8)]

Magnus (autocomplete expert), could you look into this?

Judging from the high count of 7 duplicates within about 2 weeks, this issue really affects users, and it's a technical bug in TB because we process events in the wrong order (consuming TAB/confirm before autocomplete parsing is done), per comment 23.

Worse, again per comment 23, this becomes a privacy issue when fast-typed input quickly followed by TAB is only partially processed by autocomplete, resulting in premature resolution of partial search string: Iow, TB changes the intended recipient to another unpredictable recipient from AB, which looks and works all right after completion because it's a valid address. Sending potentially sensitive content to the wrong recipient then becomes a major privacy issue. So this should really block TB31 ESR.

(In reply to Beau James from comment #23)
> Speeding up the address completion algorithm is a "good thing", to be sure,
> but is really not a solution to the fundamental UI behavior issue here. The
> fundamental issue is that user input events are not being processed in the
> correct order. The "end-of-field/move-to-next-field" input event is jumping
> ahead of "character-input-into-field" events. Speeding up the
> per-input-character processing will at best only mask the problem -
> sometimes.

Thanks Beau for that helpful analysis, looks correct to me.
I understand this bug wants to fix just that, the algorithm speed as you say is just a side issue.

> The behavior that the user expects is that input events are processed in
> sequence. Consume the "end of field/move to next field" event only after
> consuming all of the "character input" events in that field.

Of course, that's the direction the patch for this needs to take.
Thanks.

Comment 25

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #24)
> Magnus (autocomplete expert), could you look into this?

Per comment 23 and comment 24, this is a major issue involving privacy too when unintendedly sending to the wrong recipients. Caused by processing TAB/confirm event prematurely/in the wrong order.

Comment 26

[Nicola Ferralis]

(In reply to Thomas D. from comment #20)
> (In reply to Nicola Ferralis from comment #19)
> 
> OT: Nicola, thanks for sharing your user experience. 
> 
> > My system: Mac Air, 1.gGHz corei7 4GB, running OSX 10.9.4. I have 580
> > contacts in my Address book. It takes about 2 seconds to autocomplete.
> 
> What exactly (how many characters) did you type, and how fast (with or
> without pause)?
> 

It's pretty much the same if I type 2 or 5 or more characters. I type as I regularly type any text (i.e. with no interruptions). Basically I write the name or a person (say "Jeff") and after that I wait...

Comment 27

[Nicola Ferralis]

(In reply to Thomas D. from comment #20)
> Nicola, for that scenario, here's another trick for you:
> 
> You want a stable relationship between the searchword you type and that
> particular result, isn't it? For that very scenario, TB has a feature called
> "Nickname".
> 
> Go into the properties of that "same recipient" which you email frequently,
> and add a unique value into the nickname field. So if it's your friend Tom,
> you could add something like "ttt", "Tom#", or ":Tom" etc. (Due to Bug
> 325458, it's currently not enough to just pick a nickname which is unique
> out of all nicknames, so you have to make sure there's no other contact in
> all of your ABs whose properties contain Tom's unique nickname in any of
> their fields which autocomplete searches. Plus there's a couple of other
> bugs around nicknames...)
> 
> Now and everafter, whenever you type "ttt" somewhat quickly without pause
> (!), you'll get Tom's contact as a unique match in recipient autocomplete
> (and if you type fast enough, that result will come faster, too). Please
> report back if this works for you! :)

Thanks, Thomas. This helps a lot. However while not a second or more, it still takes enough time so that when I tab the address is not autocompleted...

Comment 28

[Magnus Melin [:mkmelin]]

Apart from the fact that many people apparently have come to rely on this (the old) behavior I don't think that's very intuitive, and not how auto complete fields work anywhere else. 
So in tb24 i type in one character and hit tab, a moment later it fills in *something*, but in a general case correctness of the filled address would be hit and miss.

The "wrong consumption order" case you mention should not happen, as old searches and results are cleared when input change. Or then there's a bug somewhere...  http://mxr.mozilla.org/comm-central/source/mozilla/toolkit/components/autocomplete/nsAutoCompleteController.cpp#96

Comment 29

[Jonathan Kamens]

Two comments about the old behavior:

1) It was fast enough that it was quite rare to hit tab before being able to see the address being auto-completed.

2) People form "muscle memory" about the people they correspond with frequently. They learn how much of a particular keyword they need to type to get a particular person's email address to auto-complete for them, such that they can do it with a high degree of accuracy even when they don't see the completion happen before they hit tab. Alternatively, they hit tab but they're watching the composition window when they do it, and if it autocompletes wrong, they shift-tab back and fix it. Still faster than the ridiculous several-second delay we get now, coupled with not auto-completing at all if you hit tab too fast.

Comment 30

[Ellis Miller]

(In reply to Magnus Melin from comment #28)
> Apart from the fact that many people apparently have come to rely on this
> (the old) behavior I don't think that's very intuitive, and not how auto
> complete fields work anywhere else. 
> So in tb24 i type in one character and hit tab, a moment later it fills in
> *something*, but in a general case correctness of the filled address would
> be hit and miss.

I agree that a single character is hit or miss, but that's not what most of us are doing. A power user who writes a lot of emails knows Thunderbird's behavior. I've written about 30,000 emails since I switched from Outlook 8 years ago. I type the beginning of the email address and hit tab, glancing up to make sure that it is filled in correctly. This all happens in a split second. Others are saying the exact same thing. See comment #0, comment #14, and comment #15.  

No matter what the auto-complete speed is, I *really* want auto-complete to work even if I tab away before suggestions are listed. This should at least work if there is a single match for the entered text. You can argue that this is similar to a travel booking website filling in the airport after the fact if you type in the airport code correctly.

You can say that the model is wrong and it's not how auto-complete should work. All I know is that for the past eight years, Thunderbird was a tool that just worked without any thought. Since I upgraded to the latest build a week ago, I have hated it many times a day.

Comment 31

[Beau James]

100% agree with comments #29 and #30.

For many years, TB has exhibited the same address auto-complete behavior, across a wide range of hardware (i.e. performance) and OS platforms. That has conditioned users to expect that same behavior and to make use of the behavior, especially "power users".

The behavior changed radically, and for the worse, with the recent release.

Performance aside, filling in an /*INCORRECT*/ email address because only a portion of the user's input actually got used for the address match is wrong.

Comment 32

[walter.rabus]

It is not only the case that Thunderbird is filling slowly, but _not_at_all_ if you tab too quickly. The algorithm choosing the address aside, Thunderbird is not autocompleting at all once you tab out too quickly, although there is user input in the field (I assume it's because once you leave the field the process of searching for addresses is stopped).

And then you have to manually get back into the address-field to fix that.

I appreciate fixing the algorithm, but the performance of the result is unfortunately quite workflow breaking and very inconvenient. The human brain can adapt to a lot, thus making the wrong algorithm a no-issue once the user adapted to what result came with what you typed. However, at the point where you have to wait half a second, the user is completely thrown out of workflow as there is nothing one could adapt to, which makes the new problem much more of an annoyance than the old, wrong algorithm was before.

Hardware:
i5-3320M@2.6Ghz on this machine, enough RAM, 500GB-HDD, ~200 to 300 addresses, I suppose.

Comment 33

[Nicola Ferralis]

A possible workaround to the delay could be to have autocomplete to take its time even after tabbing. Currently, it you tab before autocomplete acts upon, it actually stops. In terms of the workflow, if I tab to the next field and fill that while autocomplete finishes it job, that is a much more reasonable approach. In my case when allowed to finish, autocomplete is accurate (I type at least 4 letters).

Comment 34

[Hungerburg]

The number of chars seems not to matter here. Eg after 6 chars, when there is only one single completion left, an additional TAB will leave "to" as typed and put the mouse into "subject".

Toying around a little, it seems to me, that autocomplete only fires, when I type below a certain speed. I can eg. type 20 chars of a name and still tab off the to field without the address being completed. But as soon as I type more slowly, I get a list a completions, that disappears though, when I continue with my typing.

Reproduce like this:
- type a name till there are only two (or more) completions left up to where they should decide
- type the next char of the second (or third etc.) choice and hit tab quickly
- the first choice will be used, regardless of what was typed befor the tab was hit

Comment 35

[Bill Bach]

I am not quite sure if I have the same issue with v31, or something different.  I currently use auto-complete against my local address database only (i.e. no LDAP).  Further, I use a macro utility (KeyText) to create some Emails to clients.  I have had to deliberately slow down the macro typing speed to deal with slowness in the auto-complete feature in the past, as well as to deal with slowness in the File/Attach function.  Since upgrading to v31, these macros are failing once again.

The problem seems to stem from entering the Email address.  Pressing the first character of the Email address (e.g. "r") now does a lengthy search, and the entire TB window hangs (i.e. Windows shows "Not Responding") for approximately 10 seconds.    The time is consistent, and I've been able to "fix" some of my macros by adding a 10-second pause in it after the first character.  Sadly, not every macro supports this option, as sometimes Email addresses are user-entered fields.

Disabling the auto-complete feature from local address books eliminates the pause, so it is clear from whence this stems -- the problem is with auto-complete.  

Previous comments (e.g. 12) have asked for more details -- this 10-second pause occurs on a Win7-x64 box running 8 cores at 3.2GHz with 24GB of RAM. This system has a Personal Address Book of 5199 entries, a "Customers" address book of 1065 entries, and Collected Addresses of 6697.  Of these, I expect that many will be duplicates.  Matches when I press "R" by itself are 4154, 853, and 5373, respectively.  (Interestingly, I can press "R" to do a lookup in each of these within the Address Book dialog box in under 1 second clock time.  It is unclear why the lookup in the Email window takes 10 seconds.)

Typing faster may be an option, as I can change the macro execution speed on the fly within the macro, and I will definitely try this to see if it helps.  However, the other problem I have is now I try to write an Email to "Ron", and I end up seeing every Email address I've ever sent to the "Itron.com" domain show up in auto-complete.

While I understand the value of substring search in the auto-complete, it would be ideal to have this as an option.  Even better, it would be nice to have a hot-key within the "Write" dialog window to activate/deactivate auto-complete (or activate/deactive substring search).  This would allow me to leave it on for my "usual" Emails, but then the macro processor could disable auto-complete when it generates the message.

Comment 36

[Magnus Melin [:mkmelin]]

That's bug bug 984875 which will go into tb31.1

Comment 37

[Josh Lang]

I'm an email power user (maybe 50+ emails a day). Prior rev's of TBird were fine. Rev 31 is a nightmare. I have over 5000 addresses. After I type the first letter of an address, Thunderbird often just hangs for 20+ seconds (!!). And then invariably returns the wrong address, and I have to start the laborious process over again. It's TERRIBLY UNUSABLE. 

PLEASE PLEASE go back to whatever algorithm you used before v31. Thank you

Win 8 Pro
Quad Core 3.3Ghz
8GB RAM
Fast Motherboard
5000+ addresses

Comment 38

[Nicola Ferralis]

(In reply to Magnus Melin from comment #36)
> That's bug bug 984875 which will go into tb31.1

Has this been merged with nightly builds? Is there a way to test it?

Comment 40

[Nicola Ferralis]

This bug is still pretty much affecting TB 31.1. No difference at all.

Comment 42

[yanlu]

I have a quite large email adress book : 525 "collected adresses" and 15000 "personnal adresses". My colleagues have a less large email adress book but both my thunderbird and my coworker's thunderbird is badly impacted with the new TB31 email autocompletion behaviour.

- it sometimes does not lead to any result and leaves the typed begining of the email as is, without autocompleting it. Ex : when typing 'secret' and TAB, TB leaves 'secret' instead of completing to '[e-mail]'
- it sometimes require to type allmost the whole email before the autocompletion gives a result : we sometime have to type 'secretariat@' so as to get '[e-mail]'
- it sometimes leads to bad results. When typing 'secret' TB autocompletes to '[e-mail]' for some unknown reason (or because it got the same doamine name ????)

Please revert to previous satisfying behaviour or new behaviour !

Comment 43

[Nicola Ferralis]

A related and even more frustrating consequence of this bug: If I type a full email address, and then press enter, another one gets set, if a list of addresses is available for that specific contact. This means that if I type the *full* address and press enter I actually sent the email somewhere else. This is extremely wrong. Why in the world is the autocompletion trying to correct my intentional behavior?

I just don't get it, TB used to work great, and now it's just doesn't work at all as intended.

Comment 44

[Magnus Melin [:mkmelin]]

Nicola: That part is fixed by my patch in bug 1043310.

Comment 45

[Nicola Ferralis]

Magnus, Thanks much! I look forward to testing it. Is it already included in one of the nightlies?

Comment 46

[Magnus Melin [:mkmelin]]

Not yet.

Comment 47

[Brian Hauer]

Possibly related (and possibly distinct) -- if you receive an auto-complete proposal and then change focus using the mouse (rather than pressing TAB), the proposal will not be processed as expected, instead leaving the field with the proposal's literal rendering.

In other words, if I am typing "ab" in an addressing field in order to e-mail "[e-mail]", I will see this as an auto-complete proposal:

"ab >> Alex Baker <[e-mail]>"

Clicking on another address field, the subject, or body, or anything else, will blur the proposal and leave its value as-is.  With the the "ab >>" prefix.

It seems as if some event handler that was originally triggered on blur which either (a) accepted the first auto-complete proposal immediately if none had yet been rendered or (b) accepted the currently proposed auto-complete if one was rendered, is no longer executing on blur.

Comment 48

[Hungerburg]

(In reply to Brian Hauer from comment #47)
> Possibly related (and possibly distinct)
> if I am typing "ab" in an addressing field in order to
> e-mail "[e-mail]", I will see this as an auto-complete proposal:
> 
> "ab >> Alex Baker <[e-mail]>"
> 
> Clicking on another address field, the subject, or body, or anything else,
> will blur the proposal and leave its value as-is.  With the the "ab >>"
> prefix.

This will end up in the "To" field of the mail sent. From the users point, I see this as related: an autocomplete proposal should be acknoledged to become real (MOUSE, TAB or ENTER). Any blurring of the field should leave the field with what was actually typed. No matter what codepaths are in the race. For the sake of comprehensibility.

PS: there are 23 CCs on this issue but only 13 votes - so Vote :) This one is a real nuisance.

Comment 49

[john]

See: Bug 1067681 - Create MetaSearch rank order via Prefs for Address Autocomplete
Several other bugs raised user problems with the means for effecting the autocomplete.
The suggestion within Bug 1067681 would provide a coding backbone for enabling user control over the myriad choices for searches and sorting

See also Bug 1067681 - Create MetaSearch rank order via Prefs for Address Autocomplete
The suggestion there proposes a fix for issues with how the (misnamed) PopularityIndex works (it operates now as an AddressUseCount, and does not address recency & frequency, together also named in BuzillaLand as "frecency...")

Comment 50

[Bill Bach]

I concur that the system is getting worse in 31.1.1.  In a strange twist, I entered in "tori w" to the To and waited for auto-complete.  Sadly, the first hit was "Vareck Walla", followed by "Leta.Wolfinger", and then finally "Tori Williamson".  This was never an issue when a "Begins With" match was used.  If anything, a "begins with" match should take precedence, even if other addresses need to be considered.

Comment 51

[Charles]

(In reply to Bill Bach from comment #50)
> If anything, a "begins with" match should take precedence, even if
> other addresses need to be considered.

+10 on this...

Comment 52

[john]

Correcting My Comment 49 (pasted incorrect bug and title in the 2nd paragraph).   Comment should read:

See: Bug 1067681 - Create MetaSearch rank order via Prefs for Address Autocomplete
Several other bugs raised user problems with the means for effecting the autocomplete.
The suggestion within Bug 1067681 would provide a coding backbone for enabling user control over the myriad choices for searches and sorting.

See also Bug 1058583 - Address Book Popularity Index needs to age
The suggestion there proposes a fix for issues with how the (misnamed) PopularityIndex works (it operates now as an AddressUseCount, and does not address recency & frequency, together also named in BugzillaLand as "frecency...")

======

Charles & Bill:  You both (like I) would probably make good use of the proposed MetaSearch coding backbone and its preferences, found in Bug 1067681 -- vote, there....!

Comment 53

[yanlu]

Please revert to previous behaviour !
Experimental features should stay in devs versions as long as not ready for use.

Comment 55

[Andrew DeFaria]

Geeze I wonder what people would think if instead of the much simpler task of finding a contact, a discussion such as the above would take please @ Google and they auto complete of search terms? Come on guys, this search problem is vastly simpler than Google's and they perform beautifully in comparison.

Comment 56

[Nicola Ferralis]

I am saddened by the fact that unfortunately this new autosearch behavior seems to be considered normal. A typical example of breaking what's working with something that is not. Sorry for the rant, we use heavily TB but because of this we are thinking of migrating to something more reliable.

Comment 57

[Beau James]

The title of the bug report under which this problem is being tracked is a very incomplete description of the problem, thus quite misleading. This bug also causes incorrect autocompletions, easily resulting in email going to the wrong recipient unless the user carefully reviews the address list before sending!

For example, I have a screenshot showing that I have typed "steve t" into the address box, and showing that the default autocompletion is "Steve Sasaki <[e-mail]>". Notice that there is no "t" whatsoever in the default autocompletion. The *second* suggestion on the autocompletion list is the only valid match!

Comment 58

[Beau James]

Attachment: CropperCapture[2].Jpg

Screen clip showing *incorrect* (not incomplete) autocompletion. User typed "steve t" in the address field, default autocompletion is "Steve Sasaki <[e-mail]>" which contains no "t" whatsoever (upper or lower case).

Comment 59

[Magnus Melin [:mkmelin]]

eh, s_T_even does have a t. 
(this issue of ordering is already resolved for trunk in 970456)

Comment 60

[Beau James]

Sorry, my comment was not clear. There is no "t" in the last name or the email address, so "steve t" (a space after 'steve') ought not match "steve sasaki" at all, much less as the default match.

Comment 61

[Josh Lang]

I concur. Today's autocomplete is inferior to previous versions. 

1.) Sometimes TB autocomplete can hang for 20-30 seconds while it "searches" for an address. I have no idea why this happens, but it happens all the time.

2.) The autocomplete does not look at the whole entry when typing, and returns impossibly false results. For instance, when I start typing my son's email address at college (which is, like, a few times a day), a different address shows in the auto-complete. Here's an example. I have two similar addresses in my address book. Let's call them:

[e-mail]
[e-mail]

When I type smith@u

TB returns [e-mail]

I have to type two additional letters (ch) = smith@uch

and then TB autocomplete returns the proper address.

I can't tell you how many times I've sent email to the wrong address because of this recent programming anomaly. 

The old versions of TB-autocomplete would return the correct address when the FIRST UNIQUE LETTER was entered. The current version of TB seems to ignore the last 1 or 2 letters typed, which is frankly just wrong.

Comment 62

[Magnus Melin [:mkmelin]]

(In reply to Beau James from comment #60)
> Sorry, my comment was not clear. There is no "t" in the last name or the
> email address, so "steve t" (a space after 'steve') ought not match "steve
> sasaki" at all, much less as the default match.

The words are split up to match stuff that matches "steve AND t" (since bug 558931). 
The ordering has been fixed in bug 970456. Since bug 1091675 you can quote the string to make it not split up: "steve t" would be different from steve t. (See Target Milestone and flags in each bug for when those landed.)

But let's keep on topic, these have nothing to do with this bug.

Comment 63

[Wayne Mery (:wsmwk)]

To clarify further, this report is only for the use case where NO address is filled in.

For incorrect "completion", annoying slowness and other issues, please see other bug reports.

Comment 65

[Wayne Mery (:wsmwk)]

Comment on attachment 8538035 [details]
CropperCapture[2].Jpg

issue depicted is not this bug

Comment 66

[Wayne Mery (:wsmwk)]

To recap of sort, so we get moving on fixing this ...

I haven't examined code but based on tests of compopse window opening and typing, this seems partly a matter of window/field initialization process:

1a) If mail.compose.max_recycled_windows = 0, I must wait slightly longer before typing an address for it to autocomplete in the same predictability as when mail.compose.max_recycled_windows = 1  (the default).  So there does seem to be performance factor there.
1b) 1b was done on an idle system, so if other Thunderbird threads or other application processes are pulling CPU then this can have a similar effect.

Now, to the more fundamental issue ... whether loss of focus on the address field is the determining factor or whether some overhead of ~2 secs is involved as suggested in comment 12, or a combination.  

2a) All things being equal, auto completion doesn't depend on how many characters I type, but how quickly I tab or hit enter after the last typed character as suggested by others.  I tested an empty profile with only one addressbook entry and no ldap addressbook.  And no difference whether I type 1 character or 5, even if all 5 characters are an EXACT match on the ONLY contact.  
2b) My recollection is old autocomplete simply *forces* the "matched" contact before focus changes to some othe field, even if no autocomplete suggestion is visible at the time that tab key is hit.  

So pure addressbook search speed can't be the cause. I hope it is easy to find whether toolkit autocomplete is handling loss of focus (tab) different from our "old" autocomplete, but I haven't tested. Although bug 245989 comment 2 cites something like this about our old autocomplete.  

OTOH comment 12 states Aceman mention some ~2 seconds to fire the c++ query. Even if it is as short as a second, that MUST be changed. That, or toolkit autocomplete needs to force completion even if loss of focus occurs.

Comment 67

[Wayne Mery (:wsmwk)]

bugs changed in last 2 years that mention enter or tab, and focus - in both autcomplete component and thunderbird compose components: https://bugzilla.mozilla.org/buglist.cgi?v4=auto&f10=CP&f1=short_desc&o7=anywordssubstr&v6=auto&o3=substring&list_id=11806200&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&f8=CP&v11=focus&o11=substring&j2=OR&o1=nowordssubstr&resolution=---&resolution=FIXED&classification=Client%20Software&classification=Components&o6=substring&v7=fill%20complet&f4=component&chfieldto=Now&query_format=advanced&f3=short_desc&chfieldfrom=2y&o4=substring&f2=OP&f11=longdesc&longdesc=tab%20enter&f5=OP&f6=short_desc&component=Autocomplete&component=Autocomplete&component=Composition&component=Composition&component=Message%20Compose%20Window&component=Message%20Compose%20Window&f7=short_desc&product=MailNews%20Core&product=Thunderbird&product=Toolkit&longdesc_type=anywords

Comment 68

[Magnus Melin [:mkmelin]]

Wayne, this bug is that toolkit autocomplete needs some auto filling in PostSearchCleanup, similar to 
http://mxr.mozilla.org/comm-central/source/mozilla/xpfe/components/autocomplete/resources/content/autocomplete.xml#613

Comment 70

[mavwoot]

Just updated to 31.4 and still no luck :(

Are we any closer to a resolution for this functionality as this has been going on for a while for something that seems rather straight forward?

In 31.5, can we please at least get an option within config to revert the autocomplete version back to the old way?

Comment 73

[Jason W Theis]

Thanks for the response and update, Magnus.  I searched for this topic before posting, but did not have any success.

I find it curious that this issue is as old as eight+ months!  I distinctly did not have this issue prior to an update in the early December time frame.  I very much hope you are able to resolve it.  That, along with the strange shading scheme introduced with v34.0, are strong detractors from the quality of the user experience.

Comment 74

[Wayne Mery (:wsmwk)]

(In reply to mavwoot from comment #70)
> ...
> In 31.5, can we please at least get an option within config to revert the
> autocomplete version back to the old way?

Sorry, config option is not a viable solution. The only way forward is to address comment 68.

This bugs me too, but I know it will get fixed. Until then, patience has been my solution. :)

Comment 77

[yanlu]

It looks like things got even worse with 31.4 or maybe 31.3.

I write everyday to some mail partners.
A long time ago, when this search feature had not been broken, the correct mail appeared as soon as i typed the first letter(s).

Then this bug happened, where field is kept empty when not waiting long enough, or showed bad default result.
But before 31.3 or .4 even when the correct result was not the first, it showed, when waiting longer, as 2nd, 3d or 4th or more option in the SELECT menu.

But NOW, the correct email doesnt appear anymore at all in the selectable menu options !!!!!!!

Comment 78

[john]

Since TB20-something, when things starting going awry on autocomplete, I have self trained to type characters which identify the right address.  This self training continued as versions moved on and the responses for the auto-complete changed -- self training that overcomes the consequences of those code changes and problems reported in bugs like Bug 970456

Despite that, in the last couple of months I, too, find myself typing characters and tabbing through, and ending up with just those characters, not blank and not an email address. 

NOTE:  If I wait before tabbing, the correct address -would- show up!  The needed wait seems to depend on how many more suggestions would exist.  Does this impact comment 66 analysis?

It now/still -feels- like there used to be code, executed on field-exit event, which selected "the best" match, but that -that- code disappeared, or someone added code which stops the autocomplete -thinking- process on field exit.

So, Thomas, Wayne and Mangus, why not execute code on field exit which make simple test:  if, by the time the user exits the field, the autocomplete has not made any decision, initiate a temporary thread which chooses "the best" (under whatever current schema) and fill it in anyway.  This would end this bug pretty quick.  Alternatively, shove the -entire- autocomplete code, for each address line, into a separate thread.   

[[ Should this be in a new bug/enhancement, and connected up with depends on and etc...?   Or is this the essence of your comment 66 and comment 68? ]]

I note: the fact that code would make choices, while the user's attention may be focused elsewhere, allows for un-noticed wrong addresses; but this conundrum highlights the -connection- between this bug 1012397 and other bug solutions like bug 970456, and that they all really do depend on each other.  At the same time, it highlights the conundrum that still has the search function steal all TB processing when one types one character and nothing further (type a common vowel, for example, and pause typing briefly) [[ is this another bug report...? ]]

-----------
Also, continue to see also:
Bug 1067681 - Create MetaSearch rank order via Prefs for Address Autocomplete
Bug 1058583 - Address Book Popularity Index needs to age

Comment 79

[Steve Chessin]

Attachment: What I see when I type "steve ch" in the To: line (unfortunately, I can't grab all the suggestions, just the one in the address line.)

Comment 80

[Steve Chessin]

Attachment: What I see if I TAB or ENTER away from that line, and what I used to see if I also clicked in the line

Comment 81

[Steve Chessin]

Attachment: What I see with TB 31.4.0 if I click in the To line. (And if I then Send, I get errors, of course.)

Comment 82

[Steve Chessin]

I added three attachments to illustrate the problem. Clicking in the To: line used to complete the autocomplete function, but now it doesn't. I don't recall with which release this particular symptom showed up.

Comment 83

[Magnus Melin [:mkmelin]]

Those has nothing to do with this bug. And, that was fixed for thunderbird 31.5.0 with bug 1043310.

Comment 84

[Steve Chessin]

(In reply to Magnus Melin from comment #83)
> Those has nothing to do with this bug. And, that was fixed for thunderbird
> 31.5.0 with bug 1043310.

My apologies. I look forward to 31.5.0.

Comment 86

[Christian Moling]

In our company all clients with TB have the same problem. Since .... the version 30/31 (?), so i think.

Comment 88

[yanlu]

So as to recover normal behaviour, can we revert back to TB29 or are there severe security issues with TB29 ?

Comment 89

[Leon]

I've been using TB 24.8.1 for a while now because this issue was driving me nuts.

Here's a list of vulnerabilities fixed in each version: https://www.mozilla.org/en-US/security/known-vulnerabilities/thunderbird/

Most of them come with the message 'In general this flaw cannot be exploited through email in the Thunderbird product because scripting is disabled, but is potentially a risk in browser or browser-like contexts.'

Comment 90

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin from comment #68)
> Wayne, this bug is that toolkit autocomplete needs some auto filling in
> PostSearchCleanup, similar to 
> http://mxr.mozilla.org/comm-central/source/mozilla/xpfe/components/
> autocomplete/resources/content/autocomplete.xml#613

Will this be fixed by bug 1042561?
If not, do we need a new autocomplete bug?

Comment 91

[Magnus Melin [:mkmelin]]

No, that's a different thing. 
I don't think we need a new bug (it's this one no?), and yes it's in toolkit.

Comment 92

[Wayne Mery (:wsmwk)]

needs PostSearchCleanup attention in toolkit per comment 68 via any willing developer.

Comment 93

[:aceman]

Guys, would you be able to look at this autocomplete problem, like you did in bug 1042561? Thanks. It seems this the remaining problem that is on the toolkit side (other bugs are on TB side).

Comment 94

[Marco Bonardo [:mak]]

exactly, why should the controller take care of auto filling in this case, if the search provider itself can issue a default index? Couldn't this be fixed on the consumer side? I feel like we are expecting the controller to do a bit too much, when it should ideally just manage a popup and keyboard interaction.
That said, we can help reviewing a patch or fixing tests, but we don't have the resources to investigate and fix this bug from scratch.

Comment 95

[Kent James (:rkent)]

Aceman, Magnus, what would it take to make some progress on this? Is the needed code clear and doable on c-c?

Comment 96

[Ben Bucksch (:BenB)]

Thanks, Marco, for supporting a patch.

NeilAway, can you fix this inside toolkit, please?

Comment 97

[Thomas D. (:thomas8)]

FTR:

If you do this...

- type a few chars like "jo" to find "[e-mail]", then pause
  -> autocomplete results list pops up, but desired "[e-mail]" happens to be NOT yet toplisted
- type some more characters quickly to make your searchword unique (e.g. add "hnd" quickly to make it "johnd"), and
- be fast to press Tab or Enter immediately after typing (before the the results list disappears)

...you'll run into this bug's NASTIER BIG TWIN:

Instead of autocompleting nothing (which might alarm you at send time so you can fix the issue, but only if there's no correct recipients apart from that incomplete address which is your search word), it will cunningly autocomplete to ANOTHER (undesired, "random") recipient!

Bug 1130858 - Recipient autocomplete suggestion overrides ANY manual address input if quickly entered/pasted and confirmed with Enter/Tab before autocomplete suggestions disappear

Comment 99

[[e-mail]]

Attachment: Possible patch

Comment 100

[[e-mail]]

Comment on attachment 8655521 [details] [diff] [review]
Possible patch

As I was about to say before Bugzilla rudely interrupted me, I think I've managed to fix this completely in XBL, rather than having to touch any C++. I started by adding support for the ignoreBlurWhileSearching property, and preventing the popup from opening if the input isn't focused (as per line 1037 of XPFE code). Finally instead of adding code to PostSearchCleanup I decided that I could use the onSearchComplete callback to do the same job.

Comment 101

[Magnus Melin [:mkmelin]]

Comment on attachment 8655521 [details] [diff] [review]
Possible patch

Review of attachment 8655521 [details] [diff] [review]:
-----------------------------------------------------------------

Unfortunately, with this patch, as a side effect you loose the "confirmation" that's usually done on blur.
Say I enter j, it autocompletes to johannna. Then mouse-click in the body. What we entered is now johanna, not Johanna like it's in my addressbook. (And worse for the cases you get >>> ...)

Comment 102

[[e-mail]]

Ah yes, I forgot to check the search status. Should be a simple fix...

Comment 103

[[e-mail]]

Attachment: Possible patch

Turns out I was simply excluding too much code with that flag.

Comment 104

[Magnus Melin [:mkmelin]]

Comment on attachment 8656561 [details] [diff] [review]
Possible patch

Review of attachment 8656561 [details] [diff] [review]:
-----------------------------------------------------------------

This breaks the case where you have john.f in your ab.

Type, f, and wait for the suggestion: f >> john.f.....
Mouse click msg body. You still have "f >> john.f" showing.

Comment 105

[[e-mail]]

Attachment: Possible patch

Huh, I thought I'd checked >>. No matter, this makes the patch even simpler.

Comment 106

[Magnus Melin [:mkmelin]]

Comment on attachment 8657080 [details] [diff] [review]
Possible patch

Review of attachment 8657080 [details] [diff] [review]:
-----------------------------------------------------------------

Yep, this works!
I wouldn't add the ignoreBlurWhileSearching stuff though since it's really unused.

Comment 107

[[e-mail]]

(In reply to Magnus Melin from comment #106)
> I wouldn't add the ignoreBlurWhileSearching stuff though since it's really
> unused.

I don't know about you but I wouldn't want to break other autocomplete users; message compose autocomplete is the only place that wants this functionality.

Comment 108

[Magnus Melin [:mkmelin]]

Oh yes, I forgot we set that attribute. I guess you could key it of forcecomplete too if you want.

Comment 109

[Marco Bonardo [:mak]]

Comment on attachment 8657080 [details] [diff] [review]
Possible patch

Review of attachment 8657080 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/content/widgets/autocomplete.xml
@@ +219,5 @@
>            else
>              this.removeAttribute("nomatch");
>  
> +          if (!this.focused) {
> +            this.handleEnter();

Can you tell me more about this change?
I'm a little bit worried about possible side effects of this handleEnter call on other consumers.
If it's related to the ignoreBlurWhileSearching behavior, maybe should be conditioned on that?

Comment 110

[[e-mail]]

(In reply to Marco Bonardo from comment #109)
> If it's related to the ignoreBlurWhileSearching behavior, maybe should be
> conditioned on that?

I could do that, just to make it clear that it can only happen when it's true. (Normally the search is cancelled when you blur so it will be focused when it completes.)

Comment 111

[Marco Bonardo [:mak]]

Comment on attachment 8657080 [details] [diff] [review]
Possible patch

Review of attachment 8657080 [details] [diff] [review]:
-----------------------------------------------------------------

Please get a Try run to verify this doesn't raise any suspicious flags.

::: toolkit/content/widgets/autocomplete.xml
@@ +218,5 @@
>              this.setAttribute("nomatch", "true");
>            else
>              this.removeAttribute("nomatch");
>  
> +          if (!this.focused) {

So yes, please condition this also on ignoreBlurWhileSearching. I prefer to restrict the reach of the change, just in case.

Comment 112

[[e-mail]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ac6ad9e34835

Comment 113

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ac6ad9e34835

Comment 114

[Wayne Mery (:wsmwk)]

Kudos. Works great with tab key!  v. on trunk

Return key still has the problem. Unfortunately it was never mentioned here. Please see bug 1151520

Comment 117

[Joey Connick]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #114)
> Kudos. Works great with tab key!  v. on trunk
> 
> Return key still has the problem. Unfortunately it was never mentioned here.
> Please see bug 1151520

Ugh... if it isn't fixed for both tab and enter, I'm not sure that qualifies as fixed.  Is there anyway to adapt this fix to work for enter as well?

Comment 118

[Wayne Mery (:wsmwk)]

(In reply to [e-mail] from comment #117)
> (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment
> #114)
> > Kudos. Works great with tab key!  v. on trunk
> > 
> > Return key still has the problem. Unfortunately it was never mentioned here.
> > Please see bug 1151520
> 
> Ugh... if it isn't fixed for both tab and enter, I'm not sure that qualifies
> as fixed.  Is there anyway to adapt this fix to work for enter as well?

See target milestone of this bug report - this isn't fixed in Thunderbird 38, it will deliver in TB45. Or you may try the 44 beta which also has the fix. http://www.mozilla.org/en-US/thunderbird/channel/

If the beta doesn't help you then see bug 1151520 for the enter key use case.  Commenting here isn't particularly useful

Comment 119

[Joey Connick]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #118)
> (In reply to [e-mail] from comment #117)
> > (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment
> > #114)
> > > Kudos. Works great with tab key!  v. on trunk
> > > 
> > > Return key still has the problem. Unfortunately it was never mentioned here.
> > > Please see bug 1151520
> > 
> > Ugh... if it isn't fixed for both tab and enter, I'm not sure that qualifies
> > as fixed.  Is there anyway to adapt this fix to work for enter as well?
> 
> See target milestone of this bug report - this isn't fixed in Thunderbird
> 38, it will deliver in TB45. Or you may try the 44 beta which also has the
> fix. http://www.mozilla.org/en-US/thunderbird/channel/
> 
> If the beta doesn't help you then see bug 1151520 for the enter key use
> case.  Commenting here isn't particularly useful

Hi, I did try the beta.  When you hit tab, it is fixed (as stated here).  When you hit enter, the problem persists (as you mentioned in an earlier comment, and for a much smaller addressbook size [384] than you mention as problematic in bug 1151520).  The reason I commented here is that one assumes whatever work was done to fix it here could be applicable to the enter key case.  Or maybe it could be as easy as adding a check to this code for "has enter key been pressed?" along with "has tab key been pressed?".  Either way, the two situations seem, at least on their face, very closely related.

Comment 120

[Joey Connick]

In TB 45 beta, I'm noticing that tabbing away from an address field is failing to autocomplete again, IF that field is not the last field of the address section/widget.

In other words, typing quickly and hitting tab will autocomplete if the tab takes you to the next section of the message (the subject field).  Typing quickly and hitting tab will not autocomplete if the tab takes you to another address field--i.e. if you go back in your list of addresses, type quickly, and then hit tab to take you to the next address, the autocomplete fails.

(Just checked--same broken behaviour of not triggering autocomplete if you shift-tab: shift-tab won't autocomplete if the focus moves to another address but will autocomplete if you move OUT of the address list/widget)

Would this warrant a re-opening of this bug or the filing of a new one?  I'm not sure if previous versions exhibited this problem.

Comment 121

[Kent James (:rkent)]

Please open a new bug for tb45 issues, it is too hard to track changes if they involve multiple versions.

Comment 122

[Martin]

see https://bugzilla.mozilla.org/show_bug.cgi?id=1151520#c13
and https://bugzilla.mozilla.org/show_bug.cgi?id=1151520#c22

can this fix be applied for the return key too?

Comment 123

[Joey Connick]

(In reply to Kent James (:rkent) from comment #121)
> Please open a new bug for tb45 issues, it is too hard to track changes if
> they involve multiple versions.

Done: bug 1248486