21743 - 4.x vCard with Japanese chars is garbled.

Status: VERIFIED DUPLICATE of bug 38901

(MailNews Core :: Internationalization, defect, P3)

Description

[ji]

The vCard that is created with 4.x using Japanese chars for the firstname and
lastname is garbled on 5.0.

Steps of repro:
1. Create a vCard using 4.7, enter Japanese chars for the firstname and
   lastname.
2. on 4.7, compose a mail with the vCard as an attachemnt and send it to
   a testing account on 5.0.
3. Use 5.0 to view the mail, the firstname and lastname on the vCard is displyed
   garbled in the message view pane.

Comment 1

[ji]

Attachment: view source for the mail with Japanese vCard

Comment 2

[ji]

Attachment: source for the mail with Japanese vCard

Comment 3

[ji]

Please ignore the first attachment. The second attachment is the source for the mail with Japanese
vCard.

Comment 4

[nhottanscp]

The part has a charset label "iso-2022-j" but the actual Japanese text are
encoded in Shift_JIS with QP. 4.7 generates incorrect vCards for Japanese.
The auto charset detection is not used because the part has a charset label. We
can explicitly set a charset (to Shift_JIS) after we enable charset menu
(override) but the main body cannot be seen because it's usually JIS (not
Shift_JIS).

Comment 5

[Katsuhiko Momoi]

Yes. This has been a known bug in 4.x.
Are we at least decoding QP-encoded strings?

Comment 6

[nhottanscp]

QP decoding is applied according to the attached screen shot.

Comment 7

[nhottanscp]

Adding rhp to cc.

Comment 8

[rhp (gone)]

Hi Guys,
I did some debugging on this one and I think that I addressed a problem that 
could have existed. We didn't seem to be picking up the actual Content-Type 
header from the mime part, so we probably never applied charset conversion. 
I've attached a patch here to address this problem. 

Now, it still won't fix the problem if the part is labeled incorrectly. I'm 
really not sure how we deal with that situation?

Let me know and I will checkin this first fix. 


- rhp

Comment 9

[rhp (gone)]

Attachment: Apply this patch from the same level as your mozilla directory

Comment 10

[Katsuhiko Momoi]

Unfortunately, some of our own Communicator versions send out incorrect charset
info with vCard. I think this is a special situation with Japanese (and possibly Russian)
where the mail charset could be different from the one used for OS (and thus for vCard). 
Either we come up with a way to detect the charset in such a case or we need to have
Charset override take care of it. In 4.x, we use auto-detection and ignore the charset info for the
vCard, it seems. And that is why it probably works. Given the widespread nature of this problem,
I'm not sure if we shouldn't be doing the same thing for vCard in 5.0.

Comment 11

[rhp (gone)]

Well, I've done the code to handle the case of the correctly labled vCard. I'll 
leave it to Naoki to handle the other case.

- rhp

Comment 12

[nhottanscp]

Since the incoming data from 4.x is incorrect, there is no clean way of fixing 
this.
I think we can check the part charset then if "ISO-2022-JP" then
1) apply auto detecion for each field or
2) apply Shift_JIS conversion to the fields which are known to be containing 
incorrect data.

Comment 13

[nhottanscp]

Attachment: patch to apply auto detection if vCard is Japanese

Comment 14

[Katsuhiko Momoi]

QA contact --> ji

Comment 15

[nhottanscp]

Attachment: new patch, more accurate than using auto detection

Comment 16

[nhottanscp]

move to M16

Comment 17

[nhottanscp]

checked in

Comment 18

[nhottanscp]

checked in

Comment 19

[Katsuhiko Momoi]

Looks like there is an unresolved problem with
Phone and Fax numbers.
I'll attach an image for the remaining problem.

Comment 20

[Katsuhiko Momoi]

Attachment: JPN Phone & Fax numbers showing incorrectly.

Comment 21

[nhottanscp]

There is a bug filed specific to phone and fax numbers.

*** This bug has been marked as a duplicate of 38901 ***

Comment 22

[Katsuhiko Momoi]

Thanks! I was looking for that bug to see if it had been fixed.
Looks like we are targeting nsbeta3.
Xianglan, this is yours.

Comment 23

[ji]

Verified it as a dup.

Comment 24

[Pulsebot]

Pushed by ffxbld-merge:
https://hg.mozilla.org/releases/mozilla-beta/rev/5eeded19166b
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21743: Use isNumericAddress to validate Ipv4 and Ipv6 for SDK>=29.