41453 - Line folding in plain text body should be done to achieve consistent look on ASCII & JPN mixed text

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 6/2/2000 Win32 build **

I verified the fix for Bug 27062, but this problem
has been left over from that bug.
Given the current handling of format=flowed, the problem 
is very bad. Here's what I see now:

1. If the format=flowed is ON, which is ON by default, then
   if the line is set to break at 72, with a soft break as required
   by format=flowed.

   I checked Communicator and Outlook Express, and neither can
   intelligently break this type of line and they look terrible.
   Only Mozilla which knows how to handle the soft break displays
   these lines correctly. 

2. If format=fixed, there is a hard break at every 72 characters. Now most
   mail window cannot display 72 characters and so the line breaks long
   before 72 and then breaks again at 72.

   Neither Communicator nor Outlook can display these types of lines
   cleanly. Of course, Mozilla has the same problem here.

There is only one defense against this type of problems at
present. That is to always break/fold lines below 32 characters
or 72 bytes.

Comment 1

[Katsuhiko Momoi]

This problem will prevent people from using Mozilla
mail in 2-byte languages. Yes, you can set the character 
count to 36 but then that makes English lines too short.

Let's at least have what Communicator has. It works very well
in this regard, 72 byte setting produces very good results
for most known mailers.
Nominating for nsbeta2 because with this problem, most users
including myself will not use Mozilla mail.

Comment 2

[Katsuhiko Momoi]

I also think that we should make format=fixed the dafault
for CJ at least. Let's not break those mailers which don't understand
the extra space at the end of the format=flowed lines. We are
inserting space there now even for Japanese.

Comment 3

[leger]

Checking with I18n PDT members for call on this.

Comment 4

[leger]

Per msanz, critical to I18n beta2, marking [nsbeta2+].

Comment 5

[nhottanscp]

Accepting, cc akkana, bratell.

Comment 6

[nhottanscp]

Before make a change, I would like to make sure how it should work.
Please set a column width to 36 in the pref (Composing Messages -> Wrap Plain 
text messages at) and try sending CJK messages with that option. Let me know if 
that is the desired behavior for CJK. If that is true then we may be able to 
change the column width depends on the mail charset. Or worst case, we can 
pre-set that column width for the localized version.

Comment 7

[Daniel Bratell]

Could someone explain why the number of bytes on a line is important? What the 
user sees is characters and not bytes so why should we care about how many bytes 
we internally or in the raw mail uses on a line?


The "Space in CJK breaks the text" is bug 26734, and there is a patch there that 
disables format=flowed for certain charsets.

Comment 8

[Katsuhiko Momoi]

Bytes are important because in a 1-byte character takes up
1 unit space while a 2-byte character takes up 2 unit spaces. 
So if you have 72 Japanese characters, then the display space
needed is twice that of 72 English or Danish characters. 
If you have mixed English and Japanese text, you get
an effect like this:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

where X indicates English character line,
but Y indicates Japanese character line.

Do you want to see you mail formatted this way?
I don't!

Comment 9

[Katsuhiko Momoi]

The spacing issue is an additional factor on top of 
the reasons I mentioned above as 1 and 2. This is breaking
all known mailers of Japanese except Mozilla which 
understands format=flowed.

Comment 10

[Daniel Bratell]

It depends on the chracter set used. UTF-8 for instance is multibyte so 'å' is 
represented by two bytes but still doesn't use more than one position. 

We really should come up with a more reliable way to measure space needed to 
display than counting bytes. I mean, layout does it.

Comment 11

[Katsuhiko Momoi]

Daniel, you're right. What I meant by "bytes" is "bytes' 
in the intended native encoding. So if you're sending
Japanese mail, you would have to know that characters in
Unicode CJK range will count as "2" but those in Latin range
will count "1" in that encoding. (Actually, you have to also 
subtract escape sequences to get the correct byte count.)

Comment 12

[nhottanscp]

I wrote to the newsgroup (i18n, mail-news, editor) about this issue.

Comment 13

[Daniel Bratell]

I'm looking at the program that was mentioned in the newsgroups to calculate 
width. Hopefully I will be able to contribute my evaluation of it really soon.

Comment 14

[nhottanscp]

Daniel, do you have an estimate of the work to integrate and use the free code 
in your converter?
I am not sure if the bug should be assigned to me. Do you want me to reassign to 
you?

Comment 15

[Daniel Bratell]

I've implemented use of the unicharwidth program in nsHTMLToTXTSinkStream and it 
looks good performance wise. The time to convert a simple 1000 line latin-1 mail 
increased from 144,1 ms to 157,0 ms which may not be good but it may be the 
price we have to pay. I have not done any tests with CJK texts mainly because I 
don't have know how. That means that I don't really know if it works either.

I will attach the patch (which is a hack just too see what the results would be) 
so that those who knows CJK and easily can compose such mail and then see if the 
results are ok can test it.

If it's good I will clean it up so that i can be checked in.

Comment 16

[Daniel Bratell]

Attachment: Hacky integration of unicharwidth for testing purposes (patch)

Comment 17

[nhottanscp]

I applied the patch and Japanese line are sent out wrapped at 36.
I will ask qa to test this build.
Thanks!

Comment 18

[Daniel Bratell]

Attachment: New cleaner patch with less expensive assertions

Comment 19

[Daniel Bratell]

I've attatched a new patch. It's the same logic as before but with casts to 
remove compiler warnings, assertions ifdeffed DEBUG_wrapping and it removes some 
old debug code that was ifdefed DEBUG_bratell. I've also updated some comments 
in the code.

Comment 20

[nhottanscp]

Attachment: latest diff, r=akkana,nhotta

Comment 21

[nhottanscp]

checked in the patch
clearing nsbeta2
The line wrapping code now understands width of CJK characters, it is still not 
done by number of bytes but it does follow CJK line breaking rule unlike 4.x.

Comment 22

[Daniel Bratell]

I don't think that we should have "wrap by number of bytes" as a goal. What we 
do now is better so I would like to see this closed. WONTFIX or FIXED doesn't 
matter by me.

Comment 23

[nhottanscp]

I think the only remaining issue is whether we should restrict a byte length of 
a line (of outgoing mails), like 80 bytes.
I looked for a spec which restricts line byte length but did not find it so far.

Comment 24

[Katsuhiko Momoi]

OK. Just so that people understand the issues involved in this,
I changed the summary line. I don't really care whether or
not the folding is done by byte. The real issue that I would like to
see lines in Japanese folded at reasonable length and also lines
to fold as we did in 4.x when we have mixed ASCII & Japanese text.

For the 2nd issue, I would like to see the lines like the
following:

XXXXXXXXXXXXXXXXXXX
yy yy yyyy yyy yy y

where X indicates a Japanese character and y indicates an ASCII
character. What you don't want to see is something like:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
yy yy yyyy yyy yy y

That was my original complaint. Did you fix this problem now?

You may not find the exact line folding length specified in
any of the RFC documents. But you should know that RFC 822 and
its successor defines what a reasonable header-folding length
is (65-72 ASXII characters) and in so doing, RFC 822 basically
implies that reasonable body length for implementers would be
about the same lngeth since in some cases the headers and message
body are displayed in the same space by some mail programs.

I think by experience in our earlier version, we know the folding
length of about 72 ASCII characters and 36 Japanese characters
work well for Japanese.

Comment 25

[nhottanscp]

Fixed, please verify.

Comment 26

[Katsuhiko Momoi]

** Checked with 6/27/2000 Win32 build **

With the above build, lines are folded as expected.

multi-byte encodings break the line at half the length
of single-byte encodings.
For Japanese, this means the setting 72 corresponds to 36 JPN
characters and roughly 72 ASCII characters. This produces 
the correct result for format=fixed.
Marking it verified as fixed.