1476527 - Mojibake Unicode(Japanese Kanji/Hiragana etc ) text when Address book import from csv using en-US version of TB

Status: RESOLVED WONTFIX

(Thunderbird :: Address Book, defect)

Description

[Alice0775 White]

Attachment: screenshot, 60.0b10 vs 52.9.1

I find a Bug when I test Bug 1476520.

I wanted to transfer an address book from one profile to the other using a CSV file. So I exported it from TB52 and imported it to Tb60.0b10.

Unicode(Japanese Kanji/Hiragana etc ) will Mojibake, see attachment.

Comment 1

[Alice0775 White]

Attachment: Personal Address Book.csv

sample csv which is exported from Tb 52.9.1

Comment 2

[Alice0775 White]

Regression window:
https://hg.mozilla.org/comm-central/pushloghtml?fromchange=a400770aa52773348935dc6a638ae7caa64eb882&tochange=0b0cba8d70bd35997965afe4c02010c49e402980
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=ce1a86d3b4db161c95d1147676bbed839d7a4732&tochange=d49501f258b105c5e2dcd0a59896ec1ceabf726b

Suspect: 09475dacd05	Jorg K — Bug 1410850 - Port bug 1410288 to mailnews: Convert wstring attributes to AString in nsIPrintProgressParams.idl. rs=bustage-fix

Comment 3

[Jorg K (CEST = GMT+2)]

Thanks Alice, that would be bug 1381762.

Comment 4

[Alice0775 White]

The problem is character encoring of the exported csv file. 
Tb exported the csv file as Shift_JIS on windows.

Auto-detection character encoring of the file seems to fail when import it?

After converting character encoring of the csv file from Shift_JIS to UTF-8, then, importing is successfully performed.

Comment 5

[Jorg K (CEST = GMT+2)]

(In reply to Alice0775 White from comment #4)
> Auto-detection character encoring of the file seems to fail when import it?

Alice thanks as always for you help and your comments. There was never an auto-detection, we used the system charset:
https://hg.mozilla.org/comm-central/rev/0b0cba8d70bd#l1.32
Sadly M-C removed nsIPlatformCharset, so we had to switch to a "cheap" replacement which was already criticised in bug 1381762 comment #6: "mozilla::dom::FallbackEncoding is unlikely to be the right answer".

Masatoshi-san, can you help with a good idea here?

Comment 6

[Masatoshi Kimura [:emk]]

I think importing fails because the TB 60 locale is English.

Comment 7

[Jorg K (CEST = GMT+2)]

Thanks. In this case, try a Japanese version from here:
https://archive.mozilla.org/pub/thunderbird/candidates/60.0-candidates/build2/

Comment 8

[Masatoshi Kimura [:emk]]

TB 60 Japanese imported a Shift_JIS CSV correctly for me. Could you double-check?

Comment 9

[Alice0775 White]

(In reply to Masatoshi Kimura [:emk] from comment #8)
> TB 60 Japanese imported a Shift_JIS CSV correctly for me. Could you
> double-check?

Yes, TB 60.0b10 ja build works as expected.

Comment 10

[Jorg K (CEST = GMT+2)]

So do we close this bug?