100645 - [Regression] Non-ASCII auto-complete is not working

Status: VERIFIED FIXED

(MailNews Core :: LDAP Integration, defect)

Description

[ji]

****Obsevered with 09/17 0.9.4 build*****

Auto-complete is not working if the entry has an non-ASCII name.

Steps to reproduce:
1. Set ldap server to the setting below:
   hostname: polyglot.mcom.com
   port: 8000
   base DN: o=ネットスケープ　   (please view this page in Japanese Shift-JIS)
2. Bring up a mail compose window, enter 佐々木, which is a Japanese entry name,
the matching doesn't come up.
3. Bring up another mail compose window, enter sasaki, which is the pronuciation
of the same Japanese entry, the matching will show up.

Comment 1

[ji]

Added intl keyword and nominating for nsbranch, nsenterprise.

Comment 2

[Srilatha Moturi]

ji, do you know since when this is not working.
Adding dmose and leif to the cc list

Comment 3

[marina]

i filed a bug for Mac 97703 a while ago, could that be that this is a dup?

Comment 4

[Katsuhiko Momoi]

I repeat my comments on this problem from the other bug:

As far as I can tell, non-ASCII LDAP auto-complete has been
broken since sometime between 7/2 and 7/23 on the 0.9.3 trunk.
It was working on 0.9.2 branch. 
So a good guess is that this broke on the 0.9.4 branch when
you moved from 0.9.2ec builds to 0.9.4ec builds -- you just inherited
what was already broken sometime ago on the 0.9.3 

Comment 5

[ji]

Yes, confirmed Kat's comments. I can reproduce the problem with
2001-08-01-04-0.9.3 build. Unfortunately there are no builds between 7/2 and
7/23 on sweetlou.

Comment 6

[ji]

QA contact to myself.

Comment 7

[Katsuhiko Momoi]

I have some of these builds from 7/2001. My comments
are based on what builds I have on my machine.

Also ..

The BaseDN does not have to be in non-ASCII to reproduce this bug. So, the
following setup also works for reproducing this problem:

1. Server: polyglot.mcom.com
2. Port: 389
3. Base DN; o=Airius.com

Try Latin 1 names or Japanese names as input:

Hél ..
Franço ..

These will fail with current builds but will suceed
with NS 6.1-rtm/Mozilla 0.9.2.

You will suceed if you just type:

Hel ..
Franco ..

because we match with or without accents, it seems.

To test full non-ASCII matching, start input key 
with non-ASCII characters. This worked in NS 6.1/Moz 0.9.2,
but now fails.

Comment 8

[Jaime Rodriguez, Jr.]

Mscott - Can you take a look at this? Autocomplete not working is pretty bad.

Momoi - Did we ship 6.1 like this? 

Comment 9

[Katsuhiko Momoi]

> Momoi - Did we ship 6.1 like this? 

No. 0.9.2 branch/NS 6.1-rtm worked OK.
The regression occurred on the 0.9.3 banch
between 7/2 and 7/23. Worked OK on a build from
7/2 but did not work on a build from 7/23.

Comment 10

[Katsuhiko Momoi]

>  Worked OK on a build from 7/2 but did not work 
> on a build from 7/23.

I should correct this: I don't have a build from 7/23.
So substitute 7/25 for 7/23 in the above remarks.

Comment 11

[Jaime Rodriguez, Jr.]

Srilatha - Can we take a look at this regression? It degrades the quality of the
release from 6.1.

Comment 12

[Scott MacGregor]

is this ldap only?

Comment 13

[Katsuhiko Momoi]

> is this ldap only?

Yes. If we have non-ASCII entries in PAB, matching
those works with 9/18/2001 Win32 build.

Comment 14

[marina]

i entered bug # 97703 for the failure of autocompletion on Mac, i have some
ascii entries there and if you setup your preferences to use only LDAP server
for autocompletion the ascii entries won't be found either. It doesn't apply to
non-ascii case only. 

Comment 15

[marina]

I am adding here same comments that i posted to bug # 97703, in latest builds
here is what's happening:
Let me clarify what's going on for non-ascii entires: i've got confused and
stated that autocompoletion is not done at all because i was getting
autocompletion for majority of non-ascii names on polyglot. Now i discovered
that those that have non-ascii char on the third (3) position or later will get
autocompleted ( for exemple Clémence [e-mail]), those that have the
non-ascii char on the first or second position won't be found (for ex: bête
pë[e-mail])

Comment 16

[Jaime Rodriguez, Jr.]

Marina/Kat - How serious is this?

Mscott - Do you have any resources to resolve this one by tomorrow?

Comment 17

[marina]

Well if we want to ship 6.2 with the full LDAP functionality then we don't have
it. Kat is saying that autocompletion against polyglot fails for him for all
non-ascii entries but i don't have this experience, for me it happens with
entries that start or have non-ascii char on the second position. So the gravity
would depend on the estimate of those possibilities. It might be that different
LDAP servers ( version, earlier and latest) behave differently, we have to take
this into the account as well. 

Comment 18

[ji]

Since a double-byte entry always starts with non-ASCII char, it fails for all
the double-byte entries. 

Comment 19

[Srilatha Moturi]

I'm looking into the checkins that were done between 7/2 and 7/25.

Comment 20

[Srilatha Moturi]

This is the assertion I am getting when I search for the non-ASCII characters.
###!!! ASSERTION: not a UTF8 string: 'Error', file 
y:\mozilla\string\obsolete\nsString2.cpp, line 1795
###!!! Break: at file y:\mozilla\string\obsolete\nsString2.cpp, line 1795

Here's the stack trace when I get the assertion

NTDLL! 77f7629c()
nsDebug::Assertion(const char * 0x101132c4, const char * 0x100f8264, const char 
* 0x10113298, int 1795) line 290 + 13 bytes
nsDebug::Error(const char * 0x101132c4, const char * 0x10113298, int 1795) line 
458 + 22 bytes
NS_ConvertUTF8toUCS2::Init(const char * 0x02587028, unsigned int 4294967295) 
line 1795 + 20 bytes
NS_ConvertUTF8toUCS2::NS_ConvertUTF8toUCS2(const char * 0x02587028) line 629
nsLDAPService::CreateFilter(nsLDAPService * const 0x0654cbe0, unsigned int 1024, 
const nsAString & {...}, const nsAString & {...}, const nsAString & {...}, const 
nsAString & {...}, const nsAString & {...}, nsAString & {...}) line 940 + 16 
bytes
nsLDAPAutoCompleteSession::StartLDAPSearch() line 679 + 101 bytes
nsLDAPAutoCompleteSession::OnLDAPBind(nsILDAPMessage * 0x0654ba60) line 448
nsLDAPAutoCompleteSession::OnLDAPMessage(nsLDAPAutoCompleteSession * const 
0x05cf0d50, nsILDAPMessage * 0x0654ba60) line 352 + 12 bytes
XPTC_InvokeByIndex(nsISupports * 0x05cf0d50, unsigned int 3, unsigned int 1, 
nsXPTCVariant * 0x0654b900) line 139
EventHandler(PLEvent * 0x0654b940) line 509 + 41 bytes
PL_HandleEvent(PLEvent * 0x0654b940) line 590 + 10 bytes
PL_ProcessPendingEvents(PLEventQueue * 0x00a08950) line 520 + 9 bytes
_md_EventReceiverProc(HWND__ * 0x000a0470, unsigned int 49312, unsigned int 0, 
long 10520912) line 1071 + 9 bytes
USER32! 77e71820()
00a08950()


and this is what causing the assertion
lin 941 in directory/xpcom/src/nsLDAPService.cpp
_retval = NS_ConvertUTF8toUCS2(buffer);

and buffer is declared as char *.
dmose, can you take a look at this.

Comment 21

[Katsuhiko Momoi]

> Kat is saying that autocompletion against polyglot fails for 
> him for all non-ascii entries but i don't have this experience

This is not what I am saying actually. What I am saying is that
if you use a non-ASCII search key word, it will always fail.

As ji mentions this means that all CJK search will fail. You can
do some matching by putting in ASCII characters and get entries
that contain non-ASCII characters elsewhere in the data but this
give you only limited capability for non-ASCII name search.

This is a regression from 0.9.2/NS 6.1 and will be noticed by
those who use this functionality in non-ASCII languages.

Comment 22

[marina]

Kat, you paraphrasing what i've said: if your entry starts with non-ascii chars
( wich includes CJK chars) it'll fail. But if your entry has a non-ascii chars
on a different position then it will be returned. And as a result  we do not
have a full  LDAP functionality ( that what i responded to Jaime, i don't see
where we disagree)

Comment 23

[Srilatha Moturi]

Attachment: patch v1

Comment 24

[Srilatha Moturi]

The above patch should fix this bug. But I still see the problem ji mentioned in 
bug  100608.

Comment 25

[Dan Mosedale (:dmosedale, :dmose)]

r=dmose on that patch

Comment 26

[Jaime Rodriguez, Jr.]

Let's get the SR = and r= in the Patch Status, and bring this one to the PDT today.

Comment 27

[Alec Flett]

Comment on attachment 50105 [details] [diff] [review]
patch v1

that's easy. sr=alecf on that patch

Comment 28

[grega]

Check it in.  Added PDT+.

Comment 29

[Srilatha Moturi]

Fix checked into the branch and trunk

Comment 30

[ji]

Verified with 09/21 branch win32, linux and mac os builds. It's fixed. 

Comment 31

[ji]

Checked MAC OS X build, it's not working. But I think this could be an old 
problem for MAC OS X, not a regression. Unfortunately, there is no good MAC OS 
X 0.9.2 build on sweetlou to confirm this. Sirlatha, what do you think from 
code check? 
CC'ed to Frank. Frank, do we have any converter problem (NS_ConvertUTF8toUCS2) 
on MAC OS X?

Comment 32

[marina]

i posted my comments in the bug report for bug # 97703. And the autocomplete for
Latin-1 and cyrillic works on MacOSX but only after you supply the second char,
in case you enter only the first char the autocomplete is not working.

Comment 33

[marina]

it looks like it is a default behavior: LDAP requires more than one char (2) to
be typed to do matching. The it is working ( as designed) for Latin-1 and
cyrillic on MacOSX

Comment 34

[Dan Mosedale (:dmosedale, :dmose)]

The minimum number of characters required to do a search against a given server
is settable by hidden preference, though:
{directoryServerPrefix}.autoComplete.minStringLength

Comment 35

[Katsuhiko Momoi]

> {directoryServerPrefix}.autoComplete.minStringLength

What is the current default? 
We were able to autocomplete with 1 char in 0.9.2/NS6.1.

Comment 36

[ji]

I used a Japanese name which contains three Ja chas to reproduce the problem on 
Mac OS X. With the same name, auto-complete is working on the other platforms.

Comment 37

[ji]

Logged a seperate bug (bug 101037) for the problem on MAC OS X.

Comment 38

[Frank Tang]

No that code is cross platform . It must be some other reason to make it not 
working. 

Comment 39

[Katsuhiko Momoi]

>The minimum number of characters required to do a search 
> against a given server is settable by hidden preference, 
> though: {directoryServerPrefix}.autoComplete.minStringLength

OK, through some exprimentation, it seems that the length
is set to 2 characters uniformly for all (Unicode) characters.
This accords with what Marina has found out.

I think this is an incorrect implementation. We
should be distinguishing between CJK Ideograph
range characters and other (Unicode) character sets.

I will file a separate bug about this. 

Comment 40

[Katsuhiko Momoi]

The indiscriminate string length problem was filed as 
Bug 101086.

Comment 41

[Srilatha Moturi]

*** Bug 97703 has been marked as a duplicate of this bug. ***

Comment 42

[Dan Mosedale (:dmosedale, :dmose)]

*** Bug 101945 has been marked as a duplicate of this bug. ***

Comment 43

[ji]

Marked it as verified. It's fixed in the trunk as well.