457886 - Cannot send ISO-2022-JP email on Tb trunk

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P2)

Description

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

This is blocker bug for JP testers.

ISO-2022-JP encoder may be broken on trunk. If a line ends non-ASCII character, the state should be changed to ASCII. However, the current trunk doesn't append the state change to the end of the line. Therefore, Japanese testers cannot use trunk Tb. We should fix this bug before Tb3b1 release.

Simon, can you take this?

Comment 1

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Ah, I meant that we don't send Japanese email when the *message body* has non-ASCII characters and one or more lines end with non-ASCII character.

Comment 2

[Atsushi Sakai]

Attachment: testcase

Steps to reproduce:
1. Open the testcase with Fx.
2. Save the testcase as text.

Expexted: Saved as ISO-2022-JP
Actual: Saved file is broken.

regression range: 20080604-20080605
Maybe Bug 399257 caused this.

Comment 3

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

adding some mail developers to cc list.

If this bug is not fixed on Tb3b1, Japanese testers cannot test it. I hope that this bug blocks the b1 release, if Simon or someone can fix this bug soon.

Comment 4

[Mark Banner (:standard8)]

(In reply to comment #3)
> adding some mail developers to cc list.
> 
> If this bug is not fixed on Tb3b1, Japanese testers cannot test it. I hope that
> this bug blocks the b1 release, if Simon or someone can fix this bug soon.

If it helps, Tb3b1 is now going to be TB 3 Alpha 3. Not sure if there is anyone we can chase about this. I don't know enough about how character sets work to take this on.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Saved data of test case by "Save Page As/Test Files" on MS Win-XP SP3 :
(A) Saved ISO-2022-JP data(in hexa-decimal) by :
    [App]   Name=Firefox, Version=3.1a1pre, BuildID=2008060403
    [Gecko] MinVersion=1.9.1a1pre, MaxVersion=1.9.1a1pre
    ( [ESC]$B : Escape sequence to double-byte )
    ( [ESC](B : Escape sequence to single-byte )
    ( [CRLF]  : 0x0d0a )
> [ESC]$B                         [ESC](B [CRLF] [ESC]$B                         [ESC](B [CRLF] [CRLF]
> 1B2442 2422 2424 2426 2428 242A 1B2842  0D0A   1B2442 242B 242D 242F 2431 2433 1B2842  0D0A   0D0A

(B) Saved ISO-2022-JP data(in hexa-decimal) by :
    [App]   Name=Firefox, Version=3.1a1pre, BuildID=2008060504
    [Gecko] MinVersion=1.9.1a1pre, MaxVersion=1.9.1a1pre
> 1B2442 2422 2424 2426 2428 242A 000D 000A             242B 242D 242F 2431 2433 000D 000A 000D 000A

As seen in above, following phenomena are observed.
 (1) "Escape sequence to single-byte" is lost, when no 7bit-ascii at line end
 (2) [CRLF](0x0d0a) is changed to 0x000d + 0x000a
 (3) Due to (1), double-byte data in next line is put without escape seq
     to double-byte

This causes cut-off(due to illegal data) of mail body text lines in ISO-2022-JP at "lack of esc to single-byte", upon all Thunderbird trunk builds after upgrade to Gecko 1.9.1a2pre.
>(Regression range of Tb trunk)
> (no problem)
>   Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2pre)
>   Gecko/2008072303 Shredder/3.0a2pre
> (problem started to occur)
>   Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a2pre)
>   Gecko/2008072500 Shredder/3.0a2pre
The "cut-off" means not only "loss of mail body text data" but also "loss of following / last [CRLF]". So it can produce "never ending mail send".
  - Tb sends data before first 000D/000A in above corrupted mail body lines,
    and sends final ".[CRLF]" without preceding [CRLF],
    and waits for OK from SMTP server forever.
  - Because of no "[CRLF].[CRLF]" from Tb, SMTP sever waits for additional data
    and final "[CRLF].[CRLF]" forever.

Fault of converter from Unicode to ISO-2022-JP? Or fault of caller of converter?

Comment 6

[Atsushi Sakai]

Backing out of Bug 399257 fixes this.

Comment 7

[Masatoshi Kimura [:emk]]

Attachment: patch [Checkin: Comment 14]

Bug 399257 fix is valid only for 1-byte charsets.
With this patch, I could save attachment 341129 [details] as text.
This patch also contains automated test case.

Comment 8

[David :Bienvenu]

Thunderbird would really like to have this for our a3 build, but we would like to cut the final build in the next day or so, so anything people can do to drive this fix in, assuming it's good, as soon as possible, would be much appreciated.

Comment 9

[David :Bienvenu]

jshin, if you're reading this, and could review the patch before Simon gets to it, that would be very helpful. The patch looks pretty simple...

Comment 10

[Simon Montagu :smontagu]

I should be able to review this by tomorrow. I just want to confirm that there are no regressions in other encodings.

Comment 11

[Masatoshi Kimura [:emk]]

I'll be offline for a while.
Feel free to modify the patch if you find any problems and want to meet the deadline.
FYI, this patch passed all xpcshell tests.

Comment 12

[Simon Montagu :smontagu]

Comment on attachment 341280 [details] [diff] [review]
patch
[Checkin: Comment 14]

r+moa=smontagu

Comment 13

[David :Bienvenu]

Comment on attachment 341280 [details] [diff] [review]
patch
[Checkin: Comment 14]

this looks reasonable to me.

Comment 14

[Serge Gautherie (:sgautherie)]

Comment on attachment 341280 [details] [diff] [review]
patch
[Checkin: Comment 14]

http://hg.mozilla.org/mozilla-central/rev/d2f88b94140d