User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126 I have Some names in my Mozilla addressbook which have characters like ö and ä in them. These characters are messed up when synced to the Palm. The characters become two characters so maybe this is some raw UTF-8 (UTF-7?) representation of them instead of whatever codepage the Palm expects them to be. Note also bug 2933 which looks pretty much like it's forgotten... Reproducible: Always Steps to Reproduce: 1. Install the address book Palm sync. 2. Create an addressbook entry with non-USASCII characters in it. 3. Sync the addressbook to Palm. 4. Check the addressbook entry on Palm. Actual Results: Addressbook entries appear garbled. Expected Results: Addressbook entries should display correctly. I'm putting severity as Major because a lot of European names have accented characters in them. Feel free to adjust as needed...
The Palm sync support in Mozilla 1.2 is tested for English (ASCII languages) only. This bug will be looked into for the 1.3 beta release, marking that as the target milestone.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Target Milestone: --- → mozilla1.2beta
reassigning to ducarroz.
Assignee: rdayal → ducarroz
Status: NEW → ASSIGNED
the data we received from the Palm seems to be ISO-Latin-1. But for some reason we to a conversion to UTF-8 and then think it's still ISO-Latin-1.
dayalrajiv: Hey JF ducarroz: hi dayalrajiv: hey man, where do we do the conversion to UTF8 for ISO-Latin-1? dayalrajiv: in the conduit or Mozilla side code? ducarroz: somewhere in the conduit, I forget where... ducarroz: it's when we convert to unicode, we think the internal data is UTF8 when it's ISO-Latin ducarroz: I want to add a pref to setup the charset/encoding of the palm. ducarroz: Like that if we found some other palm using a different charset, we just have to change the pref. Also, I have asked momoi to find me a Japanese Palm ducarroz: if that exist :-) dayalrajiv: but where will you have the pref? ducarroz: that's a good point! The conduit does not have access to Mozilla Prefs right? dayalrajiv: that is right dayalrajiv: and we dont want to ducarroz: maybe Mozilla will pass it somehow ducarroz: but first, I'll make it works with ISO Latin dayalrajiv: basically in Mozilla we already have this pref, right> dayalrajiv: ? ducarroz: no, we need a specific pref which tell which encoding Hotsync use? maybe it's the default system encoding! dayalrajiv: you cannot be sure dayalrajiv: because the encoding should be the one that is set in Palm, although most likely the user will have the same encoding on Palm and desktop ducarroz: I cannot find any kind of encoding setup in palm. Also I don't have a clue what will hotsync provide us if we use a palm with a different charset ducarroz: I am still working in the unknown dayalrajiv: yep, let me also see if i can find out ducarroz: I posted the question on the newsgroup pilot.programmer dayalrajiv: btw u sure you saw we convert to UTF8 in the conduit? ducarroz: not sure, I need to debug again, I might be confuse. But I know that the raw data receive was 8bits ISO dayalrajiv: we do the conversion to UCS2 in the Mozilla support code since that is what is needed by nsIAbCard ducarroz: maybe that's where the problem is. We presume the data we get is UTF-8 dayalrajiv: yep that is correct dayalrajiv: so we need to not use the NS_ConvertUTF8toUCS2 function there
This bug is about suporting 8Bits ISO-Latin characters and not about support Unicode language like Japanese. If you want Japanese support, open a new bug... To fully support ISO-Latin-1, we need to fix the 2 following problems: 1) for some reason, the card data send or received from the Palm is converted to UTF-8 while the it uses ISO-Latin-1. 2) The name of addressBook is converted from/to the system charset while it's in fact stored as UTF-8 (see address book code) I first try to make the data going out the conduit to mozilla UNICODE but I realized that the CPString class we use is not really friend with unicode. It will request a lot more works to go that way. Therefore I decided to fix the bogus conversion we were doing in the Mozilla side code of the PalmSync which represent only few call.
Whiteboard: Have fix
Created attachment 108569 [details] Win32 API Unicode converter This is the native Win32 API Unicode converter I wrote in the first try but that I finally don't use. I pt it here just in case we need it one day...
Attachment #108571 - Flags: review?(rdayal)
Addition: Names with any European characters like 'ö' in the Palm will render the entire name in that field lost (field will be blank) when the data is imported into the Mozilla address book.
Hi JF, For attachment (id=108571), are u sure this will work for Japenese and such languages? Wouldn't we need to do some changes in the Conduit part of code to make sure it is fully Unicode compliant as we had discussed over AIM? So I guess it will be a good idea to test using some other languages besides European (although I am not sure how to find a Japanese palm or configure the emulator to work for Japenese. :)
Oh sorry JF, I didnt see your comment #6 above. Please ignore the comment #10 above. As suggested by you opened another bug for full Unicode support, bug #184297. I have changed the summary for this bug too.
Summary: Palm sync handles international (Non-USASCII) characters incorrectly → Palm sync doesnot correctly handle all ASCII based International languages
Nils, rhis patch fix the problem you mentioned in comment #9. We were loosing the whole field because the convertion from UTF-8 to Unicode was failing (because the data wasn't UTF-8)
Attachment #108571 - Flags: review?(rdayal) → review?(cavin)
Comment on attachment 108571 [details] [diff] [review] Proposed fix, v1 r=cavin.
Attachment #108571 - Flags: review?(cavin) → review+
Attachment #108571 - Flags: superreview?(sspitzer)
16 years ago
Summary: Palm sync doesnot correctly handle all ASCII based International languages → Palm sync does not correctly handle all ASCII based International languages
Comment on attachment 108571 [details] [diff] [review] Proposed fix, v1 sr=sspitzer. can you see if rdayal is available to review this?
Attachment #108571 - Flags: superreview?(sspitzer) → superreview+
Fix checked in
Status: ASSIGNED → RESOLVED
Last Resolved: 16 years ago
Resolution: --- → FIXED
Trunk build 2003-02-12: WinXP Verified Fixed. Cards and Address Books with special characters are syncing (i.e. À, ä, ñ, ö)
Status: RESOLVED → VERIFIED
*** Bug 188354 has been marked as a duplicate of this bug. ***
Component: Palm Sync → Palm Sync
Product: MailNews Core → MailNews Core Graveyard
You need to log in before you can comment on or make changes to this bug.