351802 - sends mail with Content-Type charset=ISO-2022-JP although the content is not valid ISO-2022-JP

Status: RESOLVED INCOMPLETE

(MailNews Core :: Internationalization, defect)

Description

[Bruno Haible]

User-Agent:       Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.1 (like Gecko)
Build Identifier: Thunderbird 1.5.0.5 (Windows/20060719)

Thunderbird can send out mails that have a line
  "Content-Type: text/plain; charset=ISO-2022-JP"
in the header although the mail contents is not valid ISO-2022-JP, nor
valid ISO-2022-JP-1, nor valid ISO-2022-JP-2, nor valid ISO-2022-JP-3.

Here is a gzipped, uuencoded mail sent:

begin 600 mail.txt.gz
M'XL(""AC`44``VUA:6PN='AT`(5274_;,!1]KO\"+Y<MD8*&@Y.2Y@-*QT?;
MA:UJU12V:>+!:=S6$)S.<?C8KY]#$5`8FI2'7-_CXW/.O3U97,,U%?1S.>?V
MM+`OEP`]R2%A^B<`QXV\,'()N(2TT)BI2@H\HFH1P?[ZM0/T`P\EGW-!<SPI
MHA7K-9TN["<,.F$YOV&29>\BQFS*-"2+8%9+NZ'\N0G6>OG+"5T[<&TG)+;G
M7VRA1GH/ZWQ@C8I2S?C=%MQRM8!N,IB,@&=`7#=P'.+N!BW4F!7RT<[Z[8.]
M.HOM.H@ZCSJ#QT2<`#Z1D!"P3I/)UFO5+X4Y)+@`2TF^?$=Y#5A)?V4VZ<<X
ML!V-<NSZ:=(DP<[CP6L[5T'0O(M_M[0ASP_V4.-!]_]D#UA9TCG#\8D>YZY'
M2/<P].R0Z%Q>#O:$*A:MDB!O*?T5)>IIYQ$,=(II!4GUI[KB;W?DK&02'\Z9
M4!%,%I7(F$RYS,"Q/5L/$:SO7&3%;;GS8-AW0BTR'G3Q.9,E+T2D@00][<Y:
M8-O_W*>D2B_95+_6[L3)$+MZ[OATU#GJ]._'QZ=G/_-A<AY_2[KIUV[<[K31
M<2&45H<G]TOM6;$[M;/,*1=[,%U063+5?D'SC)94E#-MK2NF1<;%/`(_Y0JA
MR8*7H#^JJ4JE%?+<1AO&D=$T<F-H]LS0_&+2S7WSRN@;X>;'#>NH;IM-,S?7
;VF;??&[C3?P!/Q4>;F&_+A#Z"X9T``O/`P``
`
end

Reminder:
ISO-2022-JP is defined in RFC 1468.
ISO-2022-JP-1 is defined in RFC 2237.
ISO-2022-JP-2 is defined in RFC 1554.
ISO-2022-JP-3 is based on JISX 0213, but is not a valid MIME encoding, since
it's not registered at IANA.

glibc iconv says on this mail:
$ /usr/bin/iconv -f ISO-2022-JP -t UTF-8 < mail.txt > /dev/null 
/usr/bin/iconv: illegal input sequence at position 950
$ /usr/bin/iconv -f ISO-2022-JP-2 -t UTF-8 < mail.txt > /dev/null 
/usr/bin/iconv: illegal input sequence at position 950
$ /usr/bin/iconv -f ISO-2022-JP-3 -t UTF-8 < mail.txt > /dev/null 
/usr/bin/iconv: illegal input sequence at position 950

GNU libiconv 1.11 says on this mail:

$ iconv -f ISO-2022-JP -t UTF-8 < mail.txt > /dev/null 
iconv: (stdin):24:0: cannot convert
$ iconv -f ISO-2022-JP-1 -t UTF-8 < mail.txt > /dev/null 
iconv: (stdin):24:0: cannot convert
$ iconv -f ISO-2022-JP-2 -t UTF-8 < mail.txt > /dev/null 
iconv: (stdin):24:0: cannot convert
$ iconv -f ISO-2022-JP-3 -t UTF-8 < mail.txt > /dev/null 
iconv: (stdin):24:0: cannot convert

Even if this occurs only with some particular user settings, it is a bug
to send out not IETF/RFC conformant mails.


Reproducible: Didn't try

Steps to Reproduce:
Please ask Manabu Suzuki <[e-mail]> for details if you cannot reproduce.


Actual Results:  
A mail with a charset specification that doesn't fit for the body.

Expected Results:  
A mail with a valid charset specification and body (possibly UTF-8).

Comment 1

[Bruno Haible]

Attachment: mail.txt sent by Thunderbird

Comment 2

[Simon Montagu :smontagu]

It looks like this behaviour goes back to bug 71762: we encode cp932 extensions to Shift_JIS in ISO-2022-JP prefaced by the ESC $ B escape sequence, as if they were part of JIS X 0208-1983.

Comment 3

[Simon Montagu :smontagu]

For the record, the characters in question are 
①②③ (U+2460 U+2461 U+2462)
ⅠⅡⅢ (U+2160 U+2161 U+2162)

which we encode as 
1b 24 42 2d 21 2d 22 2d 23 1b 28 42 0a
1b 24 42 2d 35 2d 36 2d 37 1b 28 42 0a

Comment 4

[Bruno Haible]

> we encode cp932 extensions to Shift_JIS in ISO-2022-JP prefaced by the
> ESC $ B escape sequence, as if they were part of JIS X 0208-1983.

This can indeed explain the bug. By doing so Thunderbird creates mails that
receivers that implement the RFC cannot decode.

Comment 5

[Wayne Mery (:wsmwk)]

Reporter, Can you confirm whether this problem is gone, or still exists on a current version of thunderbird?  

We are working to help old bugs move along, so your comment will be helpful.

Comment 6

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to last question. If you feel this change was made in error, please respond to this bug with your reasons why.