182643 - Palm sync does not correctly handle all ASCII based International languages

Status: VERIFIED FIXED

(MailNews Core Graveyard :: Palm Sync, defect)

Description

[Jari Ahonen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126

I have Some names in my Mozilla addressbook which have characters like ö and ä
in them. These characters are messed up when synced to the Palm. The characters
become two characters so maybe this is some raw UTF-8 (UTF-7?) representation of
them instead of whatever codepage the Palm expects them to be.

Note also bug 2933 which looks pretty much like it's forgotten...



Reproducible: Always

Steps to Reproduce:
1. Install the address book Palm sync.
2. Create an addressbook entry with non-USASCII characters in it.
3. Sync the addressbook to Palm.
4. Check the addressbook entry on Palm.

Actual Results:  
Addressbook entries appear garbled.

Expected Results:  
Addressbook entries should display correctly.


I'm putting severity as Major because a lot of European names have accented
characters in them. Feel free to adjust as needed...

Comment 1

[Rajiv Dayal]

The Palm sync support in Mozilla 1.2 is tested for English (ASCII languages)
only. This bug will be looked into for the 1.3 beta release, marking that as the
target milestone.

Comment 2

[scottputterman]

reassigning to ducarroz.

Comment 3

[Jean-Francois Ducarroz]

Accepting.

Comment 4

[Jean-Francois Ducarroz]

the data we received from the Palm seems to be ISO-Latin-1. But for some reason
we to a conversion to UTF-8 and then think it's still ISO-Latin-1.

Comment 5

[Rajiv Dayal]

dayalrajiv: Hey JF
ducarroz: hi
dayalrajiv: hey man, where do we do the conversion to UTF8 for ISO-Latin-1?
dayalrajiv: in the conduit or Mozilla side code?
ducarroz: somewhere in the conduit, I forget where...
ducarroz: it's when we convert to unicode, we think the internal data is UTF8
when it's ISO-Latin
ducarroz: I want to add a pref to setup the charset/encoding of the palm.
ducarroz: Like that if we found some other palm using a different charset, we
just have to change the pref. Also, I have asked momoi to find me a Japanese Palm
ducarroz: if that exist :-)
dayalrajiv: but where will you have the pref?
ducarroz: that's a good point! The conduit does not have access to Mozilla Prefs
right?
dayalrajiv: that is right
dayalrajiv: and we dont want to
ducarroz: maybe Mozilla will pass it somehow
ducarroz: but first, I'll make it works with ISO Latin
dayalrajiv: basically in Mozilla we already have this pref, right>
dayalrajiv: ?
ducarroz: no, we need a specific pref which tell which encoding Hotsync use?
maybe it's the default system encoding!
dayalrajiv: you cannot be sure
dayalrajiv: because the encoding should be the one that is set in Palm, although
most likely the user will have the same encoding on Palm and desktop
ducarroz: I cannot find any kind of encoding setup in palm. Also I don't have a
clue what will hotsync provide us if we use a palm with a different charset
ducarroz: I am still working in the unknown
dayalrajiv: yep, let me also see if i can find out
ducarroz: I posted the question on the newsgroup pilot.programmer
dayalrajiv: btw u sure you saw we convert to UTF8 in the conduit?
ducarroz: not sure, I need to debug again, I might be confuse. But I know that
the raw data receive was 8bits ISO
dayalrajiv: we do the conversion to UCS2 in the Mozilla support code since that
is what is needed by nsIAbCard
ducarroz: maybe that's where the problem is. We presume the data we get is UTF-8
dayalrajiv: yep that is correct
dayalrajiv: so we need to not use the NS_ConvertUTF8toUCS2 function there

Comment 6

[Jean-Francois Ducarroz]

This bug is about suporting 8Bits ISO-Latin characters and not about support
Unicode language like Japanese. If you want Japanese support, open a new bug...

To fully support ISO-Latin-1, we need to fix the 2 following problems:

1) for some reason, the card data send or received from the Palm is converted to
UTF-8 while the it uses ISO-Latin-1.

2) The name of addressBook is converted from/to the system charset while it's in
fact stored as UTF-8 (see address book code)

I first try to make the data going out the conduit to mozilla UNICODE but I
realized that the CPString class we use is not really friend with unicode. It
will request a lot more works to go that way. Therefore I decided to fix the
bogus conversion we were doing in the Mozilla side code of the PalmSync which
represent only few call.

Comment 7

[Jean-Francois Ducarroz]

Attachment: Win32 API Unicode converter

This is the native Win32 API Unicode converter I wrote in the first try but
that I finally don't use. I pt it here just in case we need it one day...

Comment 8

[Jean-Francois Ducarroz]

Attachment: Proposed fix, v1

Comment 9

[Nils Hammar]

Addition:
Names with any European characters like 'ö' in the Palm will render the entire
name in that field lost (field will be blank) when the data is imported into the
Mozilla address book.

Comment 10

[Rajiv Dayal]

Hi JF,
For attachment (id=108571), are u sure this will work for Japenese and such
languages? Wouldn't we need to do some changes in the Conduit part of code to
make sure it is fully Unicode compliant as we had discussed over AIM? So I guess
it will be a good idea to test using some other languages besides European
(although I am not sure how to find a Japanese palm or configure the emulator to
work for Japenese. :)

Comment 11

[Rajiv Dayal]

Oh sorry JF, I didnt see your comment #6 above. Please ignore the comment #10
above. 

As suggested by you opened another bug for full Unicode support, bug #184297. I
have changed the summary for this bug too.

Comment 12

[Jean-Francois Ducarroz]

Nils, rhis patch fix the problem you mentioned in comment #9. We were loosing
the whole field because the convertion from UTF-8 to Unicode was failing
(because the data wasn't UTF-8)

Comment 13

[Cavin Song]

Comment on attachment 108571 [details] [diff] [review]
Proposed fix, v1

r=cavin.

Comment 14

[(not reading, please use [e-mail] instead)]

Comment on attachment 108571 [details] [diff] [review]
Proposed fix, v1

sr=sspitzer.  can you see if rdayal is available to review this?

Comment 15

[Rajiv Dayal]

r=rdayal.

Comment 16

[Jean-Francois Ducarroz]

Fix checked in

Comment 17

[Ninoschka Baca]

Trunk build 2003-02-12: WinXP
Verified Fixed. Cards and Address Books with special characters are syncing 
(i.e. À, ä, ñ, ö)

Comment 18

[Jens Krinke]

*** Bug 188354 has been marked as a duplicate of this bug. ***