65702 - Can't decode subject if it contains ASCII and Japanese

Status: VERIFIED WORKSFORME

(MailNews Core :: Internationalization, defect, P3)

Description

[Koike Kazuhiko]

If the subject of a message contains encoded ASCII code and Japanese
code, Mozilla cannot decode it.

For example,

$B$"$$$&$($*(B(abcdefghijklmnopqrstuvwxyz)

Mew(Mailer for emacs) encodes this subject as below:

Subject: =?iso-2022-jp?B?GyRCJCIkJCQmJCgkKhsoQihhYmNkZQ==?=
 =?us-ascii?Q?fghijklmnopqrstuvwxyz)?=

Mozilla can't decode it. But NC4.7 can decode it.

Comment 1

[Koike Kazuhiko]

Attachment: Testcase(mail file)

Comment 2

[Frank Tang]

change platform to ALL and mark this as P3 moz0.9.1

Comment 3

[Katsuhiko Momoi]

Change QA contact to ji.

Comment 4

[nhottanscp]

MIME decoder is being rewritten by [e-mail], reassign to him.

Comment 5

[John G. Myers]

My rewrite will indeed handle this.

Comment 6

[John G. Myers]

Fix checked in.

Comment 7

[nhottanscp]

The fix was done by changed the MIME decoder to do a charset conversion and
always returns UTF-8 string. There are a couple of issues appeared by that
implementation.

* By always returning UTF-8, there is no way for the caller to correct
mislabeled charset headers (e.g. ISO-8859-1 labeled Big5, US-ASCII labeled
Shift_JIS).

* There are places which optimizes charset conversions in libmime. By doing
charset conversion inside MIME decoder, we cannot take advantage of them.

The first issue caused a regression of bug 65277. I reopen this bug and propose
a better implementatiuon.
* Do the UTF-8 conversion only if the header contains multiple charsets.
That can be done by pre-parsing the header to check charsets in the header. This
is a litter overhead but avoiding the charset conversion inside the decoder
helps performance gain.

Comment 8

[John G. Myers]

A better approach would be to pass an override charset down to the encoded-word 
decoder.  Converting to UTF-8 only in the multi-charset case requires a 2-pass 
decoder and prevents charset override in the multi-charset case.

What charset conversion optimizations are you talking about?  The only one I 
know of is the one which no-ops conversions between UTF-8 and US-ASCII.

I believe this bug should remain closed fixed and override work be done on bug 
65277.

Comment 9

[nhottanscp]

I think my proposal has minimum impact for the caller (and less chances of 
another regression) because it is basically requesting to back to the old 
behavior. I am not sure if anybody care about overriding multiple charsets case. 
In fact, multiple charset in a header itself is rarely seen.
So the other option could be no support for multiple charset at all then no need 
for the pre-parsing.

Anyway, please try whatever you think it's right to fix the problem but please 
test to prevent another regression.

About the optimization, we cache the charset convertors which saves extra 
createintance and getservice.

Comment 10

[John G. Myers]

This code is still working.  Override work is being done as bug 65277

Comment 11

[ji]

The testing mail that the original reporter attached does't show in the folder.
I'd like to reopen this bug.

Comment 12

[John G. Myers]

I suggest you wait another day.  You may be seeing bug 75390.

Comment 13

[ji]

It doesn't show with previous builds, like 04/02 build either.
I'll wait until tomorrow anyway.

Comment 14

[ji]

The attached testing mail doesn't show up either with today's trunk build
(04/11). Reopened the bug.

Comment 15

[John G. Myers]

Once I added a "From " line to the test case, it worked for me.

Comment 16

[ji]

The original testcase does include the "From:" line.
How did you edit it to get it work?

Comment 17

[John G. Myers]

"From ", not "From:".

Comment 18

[ji]

Yes. it appears correctly. Marked it as verified.