145293 - Import in text format (txt,csv,tab): Should support non-ascii display names and email addresses

Status: RESOLVED FIXED

(MailNews Core :: Import, defect)

Description

[Cavin Song]

This is spun off bug 91295 as the code currently only works for ascii display
names and email addresses. Some notes from 91592:

a)  how does your code handle non-ASCII display names?

b) will this code work with IDNs in email addresses?  (does eudora support 
that?) see http://bugzilla.mozilla.org/show_bug.cgi?id=127399.

Comment 1

[ji]

QA contact to myself.
Nominating for nsbeta1 since this used to be working.

Comment 2

[Michael Buckland]

Discussed in mail news bug meeting.  Decided to minus this bug.

Comment 3

[nhottanscp]

cavin, is this a bug for Eudora import?

Comment 4

[Cavin Song]

No, I think this is a problem for all import.

Comment 5

[nhottanscp]

okay, non ASCII e-mail address is not legal at this point I think no commercial
mailer support it, so we can focus on the non ASCII dispaly name problem.

Comment 6

[nhottanscp]

nominate for nsbeta1

Comment 7

[ji]

QA contact to Marina. Please reassgin to appropriate person.

Comment 8

[Simon Montagu :smontagu]

Gregg, please look at this bug too 
-dassi

Comment 9

[Samir Gehani]

Mail triage team: nsbeta1+/adt3

Comment 10

[Peter Kirk]

I received e-mail with the source from line:

From: =?8859_1?B?S2FybGr8cmdlbg==?= Xxxxxxxxx <[e-mail]>

and Mozilla renders that first name as a Chinese character, 翽, which is U+7FFD
meaning "sounds of wings flapping". Well, the arguably input is garbage (it's
nothing like the guy's real name, and I usually get an accurate display of it
although it includes a u with umlaut) so I should expect to see garbage, but
certainly the input seems to be intended as an ISO 8859-1 string which cannot
include Chinese characters.

I am using Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624.

Comment 11

[Wayne Mery (:wsmwk)]

should Bug 163501 dup to or depend on this bug?

Comment 12

[Wayne Mery (:wsmwk)]

Mark and others, 

a) is this really an import issue? or is it garden variety AB?

b) with no dupes or votes it's hard to imagine there aren't other bugs that haven't fixed or dupe parts of this bug.  Might some be
 bug 164121    	
 bug 129407
 bug 207998
And after those, what's not listed that's left to implement?

for the record, the spinoff reference in comment 0 is to end of bug 91295 comment 10.

Comment 13

[Mark Banner (:standard8)]

(In reply to comment #12)
> a) is this really an import issue? or is it garden variety AB?

Not sure, I can't say I've really tried it, but given the lack of comments/other bugs about it, I'd be surprised if its a big problem still.

> b) with no dupes or votes it's hard to imagine there aren't other bugs that
> haven't fixed or dupe parts of this bug.  Might some be
>  bug 164121     
>  bug 129407
>  bug 207998
> And after those, what's not listed that's left to implement?

There's one somewhere about allowing different character sets for import, that's probably the more relevant one in this case. So we could probably close this one now.

Comment 14

[Peter Kirk]

The e-mail mentioned in comment 10 now, in Thunderbird 2.0.0.6, displays with a Unicode unknown character symbol, a question mark in a diamond, rather than the Chinese character. As the input is garbage, that is probably correct behaviour, so at least this part of the bug seems to have been fixed.

Comment 15

[Alex]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100411 Icedove/3.0.4

The message from #10 still unreadable, despite my iconv knows that "8859_1" is valid codeset, and "S2FybGr8cmdlbg==" is decoded properly to "Karljürgen".

Comment 17

[Hiroyuki Ikezoe (:hiro)]

Attachment: Proposed fix

This patch can not be applied without the fix for bug 703175 since both of patches touch nsTextAddress::DetermineDelim.

This patch uses MsgDetectCharsetFromFile which is introduced in bug 703503.

Comment 18

[Hiroyuki Ikezoe (:hiro)]

Attachment: Some tests

Comment 21

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 576876 [details] [diff] [review]
Some tests

I haven't tried running the tests, but the code looks good to me.  Thanks.

Comment 22

[Hiroyuki Ikezoe (:hiro)]

Attachment: Adapt to the latest trunk and some cleanup from the previous patch

Comment 23

[Hiroyuki Ikezoe (:hiro)]

Attachment: debitrotted tests

carrying over review+.

Comment 24

[Hiroyuki Ikezoe (:hiro)]

Attachment: Adapt to the latest trunk and some cleanup from the previous patch

Comment 25

[David :Bienvenu]

Comment on attachment 638259 [details] [diff] [review]
Adapt to the latest trunk and some cleanup from the previous patch

looks reasonable, tests pass, thx, Hiro!

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/c9097070b413
https://hg.mozilla.org/comm-central/rev/e6d59effc2e1

Comment 27

[[e-mail]]

Comment on attachment 638259 [details] [diff] [review]
Adapt to the latest trunk and some cleanup from the previous patch

>-        numQuotes += MsgCountChar(line, '"');
>+        numQuotes += line.CountChar(PRUnichar('"'));
Noooooooooooooooooooooo!