62437 - Japanese message filter/folder names can't be migrated correctly

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P2)

Description

[Koike Kazuhiko]

Japanese message filter names are converted as garbage
on installer's profile migration.

On filter rules dialog, Japanese folder names in Action section
are also converted as garbage.

This problem was reported on Bugzilla-jp.
http://bugzilla.mozilla.gr.jp/show_bug.cgi?id=521

Comment 1

[nhottanscp]

cc alecf and sspitzer.
Seth, do we do the OS charset to UTF-8 conversion when migrating filter names?

Comment 2

[Alec Flett]

saving sspitzer from the bugzilla spam, because this isn't a migration issue
we are supposed to do the conversion the first time we open the filter file..
bienvenu probably knows where that is.

adding gayatrib (the new filters owner) and varada (covering for gayatrib right now)
gayatrib may already have a bug on this.

Comment 3

[nhottanscp]

Reassign to varada.

This probably needs Japanese system to reproduce. My machine can be used for
debug/testing.

Comment 4

[nhottanscp]

adding intl keyword

Comment 5

[ji]

Added myself to CC.

Comment 6

[Katsuhiko Momoi]

QA contact to ji.

Comment 7

[msanz]

adding nsbeta1 and jen to the copy list

Comment 8

[scottputterman]

marking nsbeta1+, moving to mozilla0.9 and reassigning to gayatrib.

Comment 9

[scottputterman]

marking nsbeta1- and moving to future milestone.

Comment 10

[Jaime Rodriguez, Jr.]

Jenn - this looks pretty bad froma UE, trying to migrate folder names. would we
allow the US-EN version to ship this way? 

Comment 11

[scottputterman]

reassigning to naving

Comment 12

[Jaime Rodriguez, Jr.]

Marking as nsCatFood, as this look like a migration issue (e.g. Project Goal)

Comment 13

[Jennifer Mulcaster]

Xianglan, 
can you please test to give us a status on this bug. Does it still persist?

Comment 14

[ji]

Jen, sorry for the late reply. I was on vacation.
Yes, it's still happening on 07/06 branch build.
Besides the Japanese keyword in a filter is not migrated.
For example, in a filter, if you put a Japanese word in a criteria, like 
subject contains XXXX (X here stands for a Japanese char)
after migration, XXXX is gone. the heyword field is empty.

Comment 15

[Scott MacGregor]

bringing this back from the future milestone into a more reasonable one for Mach
V work.

Comment 16

[Cindy Roberts]

Marking nsbranch- as it was decided in the August bug triage that we wouldn't
have eenough time in eMojo to fix this.  Let's revisit for MachV.

Comment 17

[Scott MacGregor]

cleaning up nsbranch keywords. 

Comment 18

[Navin Gupta]

Attachment: proposed fix.

Comment 19

[Navin Gupta]

This is the fix for the filter folder non-ascii names for both imap and 
local. For imap convert to utf7 name from unicode. For local convert unicode 
to mod-utf7. 

david, please review. 

Comment 20

[David :Bienvenu]

I looked at our mozilla filters and it looks like we are using the imap mod-utf7
for the filter target name, so it looks like you're doing the right thing here.
r=bienvenu, good job. Please get an sr, though, thanks.

Comment 21

[(not reading, please use [e-mail] instead)]

Comment on attachment 49932 [details] [diff] [review]
proposed fix.

sr=sspitzer

Comment 22

[(not reading, please use [e-mail] instead)]

Comment on attachment 49932 [details] [diff] [review]
proposed fix.

this has r=bienvenu

Comment 23

[Navin Gupta]

fixed on trunk.

Comment 24

[Jaime Rodriguez, Jr.]

Mscott/Navin - Pls bring this one to PDT today.

Comment 25

[ji]

Checked 0920 trunk build. The Japanese filter name is still not migrated
properly. After migration, the Japanese filter name and keyword in criteria are
garbled, and Japanese folder name in the action definition is displayed as raw
utf7 bits. Will attach a screen shot later.

Comment 26

[ji]

Attachment: A screen shot of a migrated Japanese filter.

Comment 27

[Jaime Rodriguez, Jr.]

check it in - PDT+

Comment 28

[ji]

Copied comments from another bug:
Please note that the Japanese folder name in the filter definition is displayed
 as raw utf-7 string after migration. We may need to convert it to utf-8 for 
correct display.

Comment 29

[Navin Gupta]

Attachment: proposed fix, v2

Comment 30

[Navin Gupta]

The fix is to convert to unicode based on the platform charset. 

Comment 31

[David :Bienvenu]

r=bienvenu

Comment 32

[(not reading, please use [e-mail] instead)]

Comment on attachment 50188 [details] [diff] [review]
proposed fix, v2

sr=sspitzer

Comment 33

[Navin Gupta]

fix checked on the trunk. hopefully should be fixed. 

Comment 34

[Navin Gupta]

ji, I think this issue is covered in a separate bug 70246

>Please note that the Japanese folder name in the filter definition is displayed
> as raw utf-7 string after migration. We may need to convert it to utf-8 for 
>correct display.>

Comment 35

[ji]

Verified with 09/21 trunk builds. It's fixed. The Japanese filter name, keyword 
and foldername are migrated correctly.
Navin, bug 70246 doesn't apply to this case, that is about the display problem 
of new created folder on filter window.

Comment 36

[Navin Gupta]

fixed on branch

Comment 37

[ji]

Verified with 09/24 branch builds. It's fixed. Thanks.