41461 - Japanese is not wrapped in correct position compared with ascii.

Status: VERIFIED FIXED

(MailNews Core :: Composition, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 6/2/2000 Win32 build **

This bug is making it very unpleasant to write in Japanese
under Plain Text Mail Composer.
This problem might extend to other 2-byte languages like
Chinese and Korean. 
Depending on how many characters you choose for 
"wrap lines at" in Pref | Mail & News | Composing Msgs,
if you input Japanese in Mail Composer, each line folds 
automatically at some point before the pref set character length
is reached. This makes composing in Japanese very difficult.

If you input ASCII or Latin 1 characters, you don't see this kind 
of auto-line-folding and each line will keep on going as ong as 
you don't insert a hard break by CR. This behavior is the correct
one also for Japanese. 

This auto-folding is very inconvenient. As long as this bug is 
around, I will not use Mozilla as my every day Japanese mailer.

The pref should be applied to only the msg line length as it is 
sent out, NOT when we are composing! That is the behvaior in Communicator
4.x.

Nominating for dogfood and nsbeta2.

By the way, there is also a big problem when copy/pasting 
non-Latin text. This seems to be related to the current bug
but I will file a separate one for that.

Comment 1

[leger]

Assuming that is wraps at the outgoing width, not window width, not dogfood.
Putting on [nsbeta2+] [will be minus on 6/15] radar.

Comment 2

[scottputterman]

marking M17

Comment 3

[Jean-Francois Ducarroz]

Akkana, is it yours?

Comment 4

[Katsuhiko Momoi]

Akkana, this is related to what we talked about a few days ago.
We (naoki, you, me and anyone else interested) should talk
about the issues surrounding this problem. The disposition of
this bug is clearly related to Bug 41453. If we can solve that problem,
then we can probably alleviate this problem. 

The problem observed in this bug stands out in contrast to what happens
in sending out msgs, which uses "Unicode" character count to 
break lines, while in compostion we seem to be using the width of
Latin charactes matching the pref lenth to set the line folding 
width. This would not be a problem if we send out format=flowed
and if other mailers understand format=flowed. But so far, only
Mozilla seems to know format=flowed. The other problem is 
format=flowed RFC currently has no provision for languages 
which don't use a space as a word break like Japanese. 

I'm thinking that this bug should be about having an 
option not to break lines in Composition window based
on the pref set number. Like 4.x, we should also have
line folding at window width -- however that width is set
by the user.  Bug 41453 and this bug will bring us back to 
Comm 4,x behavior.

I want my messages to "go out" at 36 Japanese characters or less
so that line folding will be compatible with most mailers out there
today. At the same time, I want to be able to compose at any window
widht I choose and see the lines break at the right edge of the window
until I insert a hard break.

Comment 5

[Akkana Peck]

I was assuming that somone in I18n who knows how to test with mixed combinations
of character sets (one-byte, two-byte-single-width, and two-byte-double-width)
would do this.

There's been some good discussion of this on the mailnews and editor newsgroups
in the last few days.

Comment 6

[nhottanscp]

The summary "In Plain text Mail Composer, a line breaks before the window width 
is reached when inputting Japanese". This is not specific to Japanese. English 
also wraps independent from the window width. I assume that this is a feature.

Comment 7

[Akkana Peck]

Yes, it's a feature that lines break at the wrap column setting rather than at
the window width.  Two issues have been raised: 
1. Some users might not want wysiwyg wrapping, might want wrapping to the window
width (yet don't want to use the html composer) and have the mail not wrapped
until it is sent.  This is independant of language, and would be an RFE on the
editor (and could be assigned to me).
2. There's an issue with bytes vs. characters and single-wide vs. double-wide
characters which makes Japanese wrap at the wrong place.  This is a bug which
would need to be fixed by someone knowledgeable about I18n charsets, perhaps
working with someone knowledgeable about layout and/or editor (not sure which,
as I don't really understand the issues) and/or output (I'm not clear whether
there's a problem there or not, and if there is, whether that would be fixed by
fixing the compose-time problems).

I'm not sure which of these issues this bug is, but it doesn't seem like any of
these issues belong to ducarroz.

Comment 8

[nhottanscp]

Is this bug requesting a Japanese line to be wrapped at 36 characters or 72?
Currently, it is not 36 nor 72. The column is set to a width of 72 mono space 
font and a Japanese line is wrapped by the column width.

Comment 9

[Katsuhiko Momoi]

This particular bug was filed based on an initial 
observation that somehow Japanese lines are breaking
before what I thought the line length was.
I assumed that when I set 36 characters, that would break
the Japanese lines at 18 characters. But I observed that
this is not the case. It wrapped at 21 JPN characters.
But upon futher investigation, I realize now that we break at
36 character width by whatever Western font we happen to be using.
And this may or may not correspond to 18 JPN characters. So far
it hasn't, even though I'm using fixed width font for both
Japanese and Western.  

This point is made by Naoki above also.

The width does look correct if you have mixed Latin and Japanese
lines, but for Japanese only lines, I wondered why it was breakig
at a point which was not deducible from any setting I have
done in the preferences, 21 chars, and not 18 chars as I would have 
guessed.

I'm going to send this akkana but would also like Erik's opinion
in addition to Naoki's. 
My question is: is this a reasonable request to see your Japanese
lines wrapped at 36 Japanese monospaced characters if the 
pref count is set to 72 (characters)? I thought if we use 
Japanese font for both JPN and SACII glyphs, this effect might
be achieved.
I'm going to file a new separate RFE about the option to turn on and off the
line wrapping based on pref setting. This will be akkana point #1
above.

Comment 10

[Erik van der Poel]

I chatted with Kat, and I now gather that we are actually trying to do WYSIWYG
mail composition. This is a good thing (for plain text).

One problem is that we are not currently getting what you see. If the pref is
set to 72, we are wrapping at 72 Japanese characters, which is 144 columns on a
standard display. This is a bug in the mail sending code.

The other problem is that the mail compose window is wrapping Japanese text at
whatever width would be required for 72 English characters (if the pref is set
to 72). This would be OK if the Japanese font was exactly twice as wide as the
English font, but on Kat's machine, that wasn't the case. So it seems like we
need to select one or more fonts such that Japanese glyphs are exactly twice as
wide as English ones. In the Windows environment, an easy way to do this is to
use the MS Gothic font, which has both Japanese and English glyphs, and the
Japanese ones are twice as wide as the English ones. I imagine that there is an
easy choice on the Mac too. On Unix, things could get "interesting" (harder).

Comment 11

[Katsuhiko Momoi]

All concerned agree that this is not J-F's bug.
So is this your, akkana?

Comment 12

[Akkana Peck]

No, as I said before, this needs to go to someone who understands I18n issues. 
Naoki or Erik?

Comment 13

[nhottanscp]

Looks like x-western font is always used for the plain text compose.
If I force set Japanese mono space font as x-western (by editing prefs.js) then 
a line wraps at 36 Japanese characters.
I talked to Erik and he is going to take care of this.

Comment 14

[Jim Roskind]

6/15 has passed... and folks are too doomed to handle it.
Changing to nsbeta2-

Comment 15

[Katsuhiko Momoi]

This needs to be re-checked to see if the problem described originally still occurs.
QA contact to ji.

Comment 16

[Frank Tang]

clear out M17

Comment 17

[ji]

Checked 2001-01-09-06-mtrunk win32 build. The problem described originally is still there.
On a plain text mail composer, the line only containing Japanese characters is wrapped at 38 (not 36) characters.
I used MS Gothic for Japanese language. The setting for line wrapping in preferences is 72.

Comment 18

[Frank Tang]

erik resign. reassign all his bug to ftang for now.

Comment 19

[Frank Tang]

line break related problem. Reassign to shanjian

Comment 20

[Frank Tang]

move to moz0.9.3 

Comment 21

[Shanjian Li]

This bug now is becoming a little confusing. I believe the problem now is that Japanese characters are 
wrapped at 38th but ascii are in 72th. But I did not see such problem. Ascii are wrapped at 75th, and 
Japanese chars are at 37th. That is consistent and correct. However, I am not sure where does the program
get this 75. I have a line in my preference:
user_pref("mailnews.wraplength", 60);
I am expecting 60 to be used instead of 75. Am I wrong?

Comment 22

[ji]

Using 06-27-06-0.9.2 win32 build, for an ascii string, it's not wrapped at all
even I specify a value from the preference window.
For a Japanese string, if I specify wraplenth to 72 (default value), it's
wrapped at 74th, If I specify wraplenth to 60, it's wrapped at 63th.

Comment 23

[Shanjian Li]

I did a little bit more investigation, and here is what I found:
If the default mail composing charset is latin1 (which can be set 
in preference/mail/composing), ascii is wrapped at 72th and japanese 
is wrapped 44th. The difference with other observation is understandable. 
That is caused by font used by system, and we are using the actual lenth 
of 72 ascii character to calculate how many japanese character can fit. 
In this case, we probably don't want to do much.

If the default mail composing charset is iso2022-JP, the font choosed will 
render ascii character as exact 1/2 of a japanese character. However, the 
wrapped lenth is not calculated correctly. Japanese chacters is wrapped at 46th 
and ascii characters are wrapped at 91th. This might be the problem we want to 
fix. 

Comment 24

[Shanjian Li]

This is a problem in layout (CSS especially). See my attached test case. If you 
change encoding to 8859-1, ascii looks fine. But if you change it to sjis, the 
wrapped length is incorrect. 

Comment 25

[Shanjian Li]

Attachment: test case to show this is a CSS problem

Comment 26

[Shanjian Li]

The problem is because we are using localeLangGroup in 
nsIRendingContext::SetFont. The langGroup we use should determined by document 
instead of locale. This leads to different font used in calculating wrap length 
(in device unit) and rendering text. 

Comment 27

[Shanjian Li]

Attachment: Patch v1.0

Comment 28

[Shanjian Li]

Add attinasi to cc list. 
attinasi. could you review my fix?

Comment 29

[Shanjian Li]

could not get r/sr in time, push it further.

Comment 30

[Marc Attinasi]

Sorry to be so late - [s]r=attinasi for patch v1.0

Comment 31

[Shanjian Li]

fix checked in. 

Comment 32

[rbs]

+      (const nsStyleVisibility*)frame->GetStyleData(eStyleStruct_Visibility, 
(const nsStyleStruct*&)vis);

should be: frame->GetStyleData(eStyleStruct_Visibility, (const 
nsStyleStruct*&)vis);

that method returns an |nsresult| !

Comment 33

[Shanjian Li]

My bad. I copied the code from another place and put it here without necessary 
cleanup. Thanks rbs for telling me this. I put r=rbs and assumed sr from 
attinasi (hope you don't mind). 

Comment 34

[ji]

Verified with 09/07 builds. It's fixed.