Palm sync does not correctly handle all ASCII based International languages

VERIFIED FIXED in mozilla1.3beta

Status

MailNews Core Graveyard
Palm Sync
--
major
VERIFIED FIXED
16 years ago
9 years ago

People

(Reporter: Jari Ahonen, Assigned: Jean-Francois Ducarroz)

Tracking

Trunk
mozilla1.3beta
x86
Windows 2000

Details

(Whiteboard: Have fix)

Attachments

(2 attachments)

2.25 KB, text/plain
Details
4.66 KB, patch
Cavin Song
: review+
(not reading, please use seth@sspitzer.org instead)
: superreview+
Details | Diff | Splinter Review
(Reporter)

Description

16 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2) Gecko/20021126

I have Some names in my Mozilla addressbook which have characters like ö and ä
in them. These characters are messed up when synced to the Palm. The characters
become two characters so maybe this is some raw UTF-8 (UTF-7?) representation of
them instead of whatever codepage the Palm expects them to be.

Note also bug 2933 which looks pretty much like it's forgotten...



Reproducible: Always

Steps to Reproduce:
1. Install the address book Palm sync.
2. Create an addressbook entry with non-USASCII characters in it.
3. Sync the addressbook to Palm.
4. Check the addressbook entry on Palm.

Actual Results:  
Addressbook entries appear garbled.

Expected Results:  
Addressbook entries should display correctly.


I'm putting severity as Major because a lot of European names have accented
characters in them. Feel free to adjust as needed...

Comment 1

16 years ago
The Palm sync support in Mozilla 1.2 is tested for English (ASCII languages)
only. This bug will be looked into for the 1.3 beta release, marking that as the
target milestone.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Target Milestone: --- → mozilla1.2beta

Updated

16 years ago
Target Milestone: mozilla1.2beta → mozilla1.3beta

Comment 2

16 years ago
reassigning to ducarroz.
Assignee: rdayal → ducarroz

Updated

16 years ago
Keywords: nsbeta1+
(Assignee)

Comment 3

16 years ago
Accepting.
Status: NEW → ASSIGNED

Updated

16 years ago
QA Contact: meehansqa → nbaca
(Assignee)

Comment 4

16 years ago
the data we received from the Palm seems to be ISO-Latin-1. But for some reason
we to a conversion to UTF-8 and then think it's still ISO-Latin-1.

Comment 5

16 years ago
dayalrajiv: Hey JF
ducarroz: hi
dayalrajiv: hey man, where do we do the conversion to UTF8 for ISO-Latin-1?
dayalrajiv: in the conduit or Mozilla side code?
ducarroz: somewhere in the conduit, I forget where...
ducarroz: it's when we convert to unicode, we think the internal data is UTF8
when it's ISO-Latin
ducarroz: I want to add a pref to setup the charset/encoding of the palm.
ducarroz: Like that if we found some other palm using a different charset, we
just have to change the pref. Also, I have asked momoi to find me a Japanese Palm
ducarroz: if that exist :-)
dayalrajiv: but where will you have the pref?
ducarroz: that's a good point! The conduit does not have access to Mozilla Prefs
right?
dayalrajiv: that is right
dayalrajiv: and we dont want to
ducarroz: maybe Mozilla will pass it somehow
ducarroz: but first, I'll make it works with ISO Latin
dayalrajiv: basically in Mozilla we already have this pref, right>
dayalrajiv: ?
ducarroz: no, we need a specific pref which tell which encoding Hotsync use?
maybe it's the default system encoding!
dayalrajiv: you cannot be sure
dayalrajiv: because the encoding should be the one that is set in Palm, although
most likely the user will have the same encoding on Palm and desktop
ducarroz: I cannot find any kind of encoding setup in palm. Also I don't have a
clue what will hotsync provide us if we use a palm with a different charset
ducarroz: I am still working in the unknown
dayalrajiv: yep, let me also see if i can find out
ducarroz: I posted the question on the newsgroup pilot.programmer
dayalrajiv: btw u sure you saw we convert to UTF8 in the conduit?
ducarroz: not sure, I need to debug again, I might be confuse. But I know that
the raw data receive was 8bits ISO
dayalrajiv: we do the conversion to UCS2 in the Mozilla support code since that
is what is needed by nsIAbCard
ducarroz: maybe that's where the problem is. We presume the data we get is UTF-8
dayalrajiv: yep that is correct
dayalrajiv: so we need to not use the NS_ConvertUTF8toUCS2 function there
(Assignee)

Comment 6

16 years ago
This bug is about suporting 8Bits ISO-Latin characters and not about support
Unicode language like Japanese. If you want Japanese support, open a new bug...

To fully support ISO-Latin-1, we need to fix the 2 following problems:

1) for some reason, the card data send or received from the Palm is converted to
UTF-8 while the it uses ISO-Latin-1.

2) The name of addressBook is converted from/to the system charset while it's in
fact stored as UTF-8 (see address book code)

I first try to make the data going out the conduit to mozilla UNICODE but I
realized that the CPString class we use is not really friend with unicode. It
will request a lot more works to go that way. Therefore I decided to fix the
bogus conversion we were doing in the Mozilla side code of the PalmSync which
represent only few call.
Whiteboard: Have fix
(Assignee)

Comment 7

16 years ago
Created attachment 108569 [details]
Win32 API Unicode converter

This is the native Win32 API Unicode converter I wrote in the first try but
that I finally don't use. I pt it here just in case we need it one day...
(Assignee)

Comment 8

16 years ago
Created attachment 108571 [details] [diff] [review]
Proposed fix, v1
(Assignee)

Updated

16 years ago
Attachment #108571 - Flags: review?(rdayal)

Comment 9

16 years ago
Addition:
Names with any European characters like 'ö' in the Palm will render the entire
name in that field lost (field will be blank) when the data is imported into the
Mozilla address book.

Comment 10

16 years ago
Hi JF,
For attachment (id=108571), are u sure this will work for Japenese and such
languages? Wouldn't we need to do some changes in the Conduit part of code to
make sure it is fully Unicode compliant as we had discussed over AIM? So I guess
it will be a good idea to test using some other languages besides European
(although I am not sure how to find a Japanese palm or configure the emulator to
work for Japenese. :)



Comment 11

16 years ago
Oh sorry JF, I didnt see your comment #6 above. Please ignore the comment #10
above. 

As suggested by you opened another bug for full Unicode support, bug #184297. I
have changed the summary for this bug too.
Summary: Palm sync handles international (Non-USASCII) characters incorrectly → Palm sync doesnot correctly handle all ASCII based International languages
(Assignee)

Comment 12

16 years ago
Nils, rhis patch fix the problem you mentioned in comment #9. We were loosing
the whole field because the convertion from UTF-8 to Unicode was failing
(because the data wasn't UTF-8)
(Assignee)

Updated

16 years ago
Attachment #108571 - Flags: review?(rdayal) → review?(cavin)

Comment 13

16 years ago
Comment on attachment 108571 [details] [diff] [review]
Proposed fix, v1

r=cavin.
Attachment #108571 - Flags: review?(cavin) → review+
(Assignee)

Updated

16 years ago
Attachment #108571 - Flags: superreview?(sspitzer)
Summary: Palm sync doesnot correctly handle all ASCII based International languages → Palm sync does not correctly handle all ASCII based International languages
Comment on attachment 108571 [details] [diff] [review]
Proposed fix, v1

sr=sspitzer.  can you see if rdayal is available to review this?
Attachment #108571 - Flags: superreview?(sspitzer) → superreview+

Comment 15

16 years ago
r=rdayal.
(Assignee)

Comment 16

16 years ago
Fix checked in
Status: ASSIGNED → RESOLVED
Last Resolved: 16 years ago
Resolution: --- → FIXED

Comment 17

16 years ago
Trunk build 2003-02-12: WinXP
Verified Fixed. Cards and Address Books with special characters are syncing 
(i.e. À, ä, ñ, ö)
Status: RESOLVED → VERIFIED

Comment 18

16 years ago
*** Bug 188354 has been marked as a duplicate of this bug. ***
Product: MailNews → Core
Product: Core → MailNews Core

Updated

9 years ago
Component: Palm Sync → Palm Sync
Product: MailNews Core → MailNews Core Graveyard
You need to log in before you can comment on or make changes to this bug.