[Regression] Non-ASCII auto-complete is not working



MailNews Core
LDAP Integration
16 years ago
9 years ago


(Reporter: ji, Assigned: Srilatha Moturi)


({intl, regression})

intl, regression

Firefox Tracking Flags

(Not tracked)


(Whiteboard: PDT+)


(1 attachment)



16 years ago
****Obsevered with 09/17 0.9.4 build*****

Auto-complete is not working if the entry has an non-ASCII name.

Steps to reproduce:
1. Set ldap server to the setting below:
   hostname: polyglot.mcom.com
   port: 8000
   base DN: o=ネットスケープ    (please view this page in Japanese Shift-JIS)
2. Bring up a mail compose window, enter 佐々木, which is a Japanese entry name,
the matching doesn't come up.
3. Bring up another mail compose window, enter sasaki, which is the pronuciation
of the same Japanese entry, the matching will show up.

Comment 1

16 years ago
Added intl keyword and nominating for nsbranch, nsenterprise.
Keywords: intl, nsbranch, nsenterprise

Comment 2

16 years ago
ji, do you know since when this is not working.
Adding dmose and leif to the cc list

Comment 3

16 years ago
i filed a bug for Mac 97703 a while ago, could that be that this is a dup?

Comment 4

16 years ago
I repeat my comments on this problem from the other bug:

As far as I can tell, non-ASCII LDAP auto-complete has been
broken since sometime between 7/2 and 7/23 on the 0.9.3 trunk.
It was working on 0.9.2 branch. 
So a good guess is that this broke on the 0.9.4 branch when
you moved from 0.9.2ec builds to 0.9.4ec builds -- you just inherited
what was already broken sometime ago on the 0.9.3 

Comment 5

16 years ago
Yes, confirmed Kat's comments. I can reproduce the problem with
2001-08-01-04-0.9.3 build. Unfortunately there are no builds between 7/2 and
7/23 on sweetlou.

Comment 6

16 years ago
QA contact to myself.
QA Contact: yulian → ji

Comment 7

16 years ago
I have some of these builds from 7/2001. My comments
are based on what builds I have on my machine.

Also ..

The BaseDN does not have to be in non-ASCII to reproduce this bug. So, the
following setup also works for reproducing this problem:

1. Server: polyglot.mcom.com
2. Port: 389
3. Base DN; o=Airius.com

Try Latin 1 names or Japanese names as input:

Hél ..
Franço ..

These will fail with current builds but will suceed
with NS 6.1-rtm/Mozilla 0.9.2.

You will suceed if you just type:

Hel ..
Franco ..

because we match with or without accents, it seems.

To test full non-ASCII matching, start input key 
with non-ASCII characters. This worked in NS 6.1/Moz 0.9.2,
but now fails.

Comment 8

16 years ago
Mscott - Can you take a look at this? Autocomplete not working is pretty bad.

Momoi - Did we ship 6.1 like this? 
QA Contact: ji → yulian

Comment 9

16 years ago
> Momoi - Did we ship 6.1 like this? 

No. 0.9.2 branch/NS 6.1-rtm worked OK.
The regression occurred on the 0.9.3 banch
between 7/2 and 7/23. Worked OK on a build from
7/2 but did not work on a build from 7/23.


16 years ago
QA Contact: yulian → ji

Comment 10

16 years ago
>  Worked OK on a build from 7/2 but did not work 
> on a build from 7/23.

I should correct this: I don't have a build from 7/23.
So substitute 7/25 for 7/23 in the above remarks.

Comment 11

16 years ago
Srilatha - Can we take a look at this regression? It degrades the quality of the
release from 6.1.

Comment 12

16 years ago
is this ldap only?

Comment 13

16 years ago
> is this ldap only?

Yes. If we have non-ASCII entries in PAB, matching
those works with 9/18/2001 Win32 build.

Comment 14

16 years ago
i entered bug # 97703 for the failure of autocompletion on Mac, i have some
ascii entries there and if you setup your preferences to use only LDAP server
for autocompletion the ascii entries won't be found either. It doesn't apply to
non-ascii case only. 

Comment 15

16 years ago
I am adding here same comments that i posted to bug # 97703, in latest builds
here is what's happening:
Let me clarify what's going on for non-ascii entires: i've got confused and
stated that autocompoletion is not done at all because i was getting
autocompletion for majority of non-ascii names on polyglot. Now i discovered
that those that have non-ascii char on the third (3) position or later will get
autocompleted ( for exemple Clémence ...@polyglot.com), those that have the
non-ascii char on the first or second position won't be found (for ex: bête

Comment 16

16 years ago
Marina/Kat - How serious is this?

Mscott - Do you have any resources to resolve this one by tomorrow?
Keywords: regression

Comment 17

16 years ago
Well if we want to ship 6.2 with the full LDAP functionality then we don't have
it. Kat is saying that autocompletion against polyglot fails for him for all
non-ascii entries but i don't have this experience, for me it happens with
entries that start or have non-ascii char on the second position. So the gravity
would depend on the estimate of those possibilities. It might be that different
LDAP servers ( version, earlier and latest) behave differently, we have to take
this into the account as well. 

Comment 18

16 years ago
Since a double-byte entry always starts with non-ASCII char, it fails for all
the double-byte entries. 

Comment 19

16 years ago
I'm looking into the checkins that were done between 7/2 and 7/25.

Comment 20

16 years ago
This is the assertion I am getting when I search for the non-ASCII characters.
###!!! ASSERTION: not a UTF8 string: 'Error', file 
y:\mozilla\string\obsolete\nsString2.cpp, line 1795
###!!! Break: at file y:\mozilla\string\obsolete\nsString2.cpp, line 1795

Here's the stack trace when I get the assertion

NTDLL! 77f7629c()
nsDebug::Assertion(const char * 0x101132c4, const char * 0x100f8264, const char 
* 0x10113298, int 1795) line 290 + 13 bytes
nsDebug::Error(const char * 0x101132c4, const char * 0x10113298, int 1795) line 
458 + 22 bytes
NS_ConvertUTF8toUCS2::Init(const char * 0x02587028, unsigned int 4294967295) 
line 1795 + 20 bytes
NS_ConvertUTF8toUCS2::NS_ConvertUTF8toUCS2(const char * 0x02587028) line 629
nsLDAPService::CreateFilter(nsLDAPService * const 0x0654cbe0, unsigned int 1024, 
const nsAString & {...}, const nsAString & {...}, const nsAString & {...}, const 
nsAString & {...}, const nsAString & {...}, nsAString & {...}) line 940 + 16 
nsLDAPAutoCompleteSession::StartLDAPSearch() line 679 + 101 bytes
nsLDAPAutoCompleteSession::OnLDAPBind(nsILDAPMessage * 0x0654ba60) line 448
nsLDAPAutoCompleteSession::OnLDAPMessage(nsLDAPAutoCompleteSession * const 
0x05cf0d50, nsILDAPMessage * 0x0654ba60) line 352 + 12 bytes
XPTC_InvokeByIndex(nsISupports * 0x05cf0d50, unsigned int 3, unsigned int 1, 
nsXPTCVariant * 0x0654b900) line 139
EventHandler(PLEvent * 0x0654b940) line 509 + 41 bytes
PL_HandleEvent(PLEvent * 0x0654b940) line 590 + 10 bytes
PL_ProcessPendingEvents(PLEventQueue * 0x00a08950) line 520 + 9 bytes
_md_EventReceiverProc(HWND__ * 0x000a0470, unsigned int 49312, unsigned int 0, 
long 10520912) line 1071 + 9 bytes
USER32! 77e71820()

and this is what causing the assertion
lin 941 in directory/xpcom/src/nsLDAPService.cpp
_retval = NS_ConvertUTF8toUCS2(buffer);

and buffer is declared as char *.
dmose, can you take a look at this.

Comment 21

16 years ago
> Kat is saying that autocompletion against polyglot fails for 
> him for all non-ascii entries but i don't have this experience

This is not what I am saying actually. What I am saying is that
if you use a non-ASCII search key word, it will always fail.

As ji mentions this means that all CJK search will fail. You can
do some matching by putting in ASCII characters and get entries
that contain non-ASCII characters elsewhere in the data but this
give you only limited capability for non-ASCII name search.

This is a regression from 0.9.2/NS 6.1 and will be noticed by
those who use this functionality in non-ASCII languages.

Comment 22

16 years ago
Kat, you paraphrasing what i've said: if your entry starts with non-ascii chars
( wich includes CJK chars) it'll fail. But if your entry has a non-ascii chars
on a different position then it will be returned. And as a result  we do not
have a full  LDAP functionality ( that what i responded to Jaime, i don't see
where we disagree)

Comment 23

16 years ago
Created attachment 50105 [details] [diff] [review]
patch v1

Comment 24

16 years ago
The above patch should fix this bug. But I still see the problem ji mentioned in 
bug  100608.
r=dmose on that patch
OS: Windows 2000 → All
Whiteboard: have patch, r=

Comment 26

16 years ago
Let's get the SR = and r= in the Patch Status, and bring this one to the PDT today.


16 years ago
Attachment #50105 - Flags: review+

Comment 27

16 years ago
Comment on attachment 50105 [details] [diff] [review]
patch v1

that's easy. sr=alecf on that patch
Attachment #50105 - Flags: superreview+

Comment 28

16 years ago
Check it in.  Added PDT+.
Whiteboard: have patch, r= → PDT+

Comment 29

16 years ago
Fix checked into the branch and trunk
Last Resolved: 16 years ago
Resolution: --- → FIXED

Comment 30

16 years ago
Verified with 09/21 branch win32, linux and mac os builds. It's fixed. 
Keywords: vtrunk

Comment 31

16 years ago
Checked MAC OS X build, it's not working. But I think this could be an old 
problem for MAC OS X, not a regression. Unfortunately, there is no good MAC OS 
X 0.9.2 build on sweetlou to confirm this. Sirlatha, what do you think from 
code check? 
CC'ed to Frank. Frank, do we have any converter problem (NS_ConvertUTF8toUCS2) 
on MAC OS X?

Comment 32

16 years ago
i posted my comments in the bug report for bug # 97703. And the autocomplete for
Latin-1 and cyrillic works on MacOSX but only after you supply the second char,
in case you enter only the first char the autocomplete is not working.

Comment 33

16 years ago
it looks like it is a default behavior: LDAP requires more than one char (2) to
be typed to do matching. The it is working ( as designed) for Latin-1 and
cyrillic on MacOSX
The minimum number of characters required to do a search against a given server
is settable by hidden preference, though:

Comment 35

16 years ago
> {directoryServerPrefix}.autoComplete.minStringLength

What is the current default? 
We were able to autocomplete with 1 char in 0.9.2/NS6.1.

Comment 36

16 years ago
I used a Japanese name which contains three Ja chas to reproduce the problem on 
Mac OS X. With the same name, auto-complete is working on the other platforms.

Comment 37

16 years ago
Logged a seperate bug (bug 101037) for the problem on MAC OS X.

Comment 38

16 years ago
No that code is cross platform . It must be some other reason to make it not 

Comment 39

16 years ago
>The minimum number of characters required to do a search 
> against a given server is settable by hidden preference, 
> though: {directoryServerPrefix}.autoComplete.minStringLength

OK, through some exprimentation, it seems that the length
is set to 2 characters uniformly for all (Unicode) characters.
This accords with what Marina has found out.

I think this is an incorrect implementation. We
should be distinguishing between CJK Ideograph
range characters and other (Unicode) character sets.

I will file a separate bug about this. 

Comment 40

16 years ago
The indiscriminate string length problem was filed as 
Bug 101086.

Comment 41

16 years ago
*** Bug 97703 has been marked as a duplicate of this bug. ***
*** Bug 101945 has been marked as a duplicate of this bug. ***

Comment 43

16 years ago
Marked it as verified. It's fixed in the trunk as well.
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.