Closed Bug 218823 Opened 21 years ago Closed 2 years ago

RFE: Automatic paragraph direction of plain text (email, IRC messages, {optionally} textareas, etc.)

Categories

(Core :: Layout: Text and Fonts, enhancement)

Product:
Core
Component:
Layout: Text and Fonts
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: bugzillamozilla, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: rtl, Whiteboard: Please read comment #39 before commenting on textareas)

Attachments

(3 files)

Mozilla's current handling of plain text - All LTR
3.82 KB, image/png
Details
Suggested algorithm - context-based direction
4.25 KB, image/png
Details
screenshot- current status with extentions
166.51 KB, image/png
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624

There are many cases where paragraph direction can be assumed according to the
first character.

* In a textarea, if the user starts a paragraph by typing an LTR character (e.g.
an English "A"), it is safe to assume that the general direction of the
paragraph should remain LTR. The same is true with RTL characters (such as a
Hebrew "à") which are usually followed by RTL paragraphs.

* If the first character doesn't have explicit (or "strong") direction, than the
next character that does should set the paragraph direction.

* When a paragraph starts with a character that has different direction than
intended by the author (such as a Hebrew sentence that start with an English
word), there should be an option to manually override Mozilla. Either by
providing a keyboard shortcut (Bug 98160), or via direction buttons in the
toolbar (Bug 96057 and Bug 119857). In the case of plain text, changing the
display of textarea isn't sufficient. Mozilla should insert the relevant control
characters (if supported by the active encoding), so that this text/message
could still display properly at the recipients side.

* This implementation should affect any plain text input widget, as well as the
display of plain text messages (mail, IRC, Gopher and so on). Currently, all
plain text emails that are displayed in Mozilla are rendered left-aligned and
LTR. Implementing my suggestion would fix this problem without requiring any
user intervention.

Reference: At least two existing mail client currently implement a similar
algorithm - KDE KMail and Microsoft Outlook XP.

Prog.

Reproducible: Always

Steps to Reproduce:
Paragraphs #1 and #4 show how context-based automatic direction should work.
Note that the first character sets the direction of the entire paragraph.

Paragraphs #2 and #3 show cases where ambiguous direction can be solved
(manually) through the use of RLM and LRM control characters. The user should
not have to be aware of the underlying workings of such characters, but merely
to use standard direction controls (Ctrl+Shift or Direction buttons). Mozilla
should transparently handle the insertion of these characters in the begining
of the active paragraph, regardless of the current caret position.

Prog.
I'd like to take this, since I did a little preliminary work on it in the past.
As far as I remember, I didn't file a bug on it.
Assignee: mkaply → smontagu
CONFIRMING re Comment #3
Status: UNCONFIRMED → NEW
Ever confirmed: true
My suggested algorithm (hoping that I won't get accused of "blindly following
Unicode" ;-) ) is to follow
http://www.unicode.org/reports/tr9/#The_Paragraph_Level for text we receive. I'm
not yet convinced about adding control characters to text we produce as
suggested in comment 2: I need to consider possible scenarios.
Here are some links to related documents:

RFC 1555 - Hebrew Character Encoding for Internet Messages
http://www.faqs.org/rfcs/rfc1555.html

RFC 1556 - Handling of Bi-directional Texts in MIM
http://www.faqs.org/rfcs/rfc1556.html

Standard ECMA-48 - Control Functions for Coded Character Sets
http://cr.yp.to/bib/1991/ecma-48.pdf

[IS-1904]  (Israeli Standard): Application of Hebrew in mail
I don't have a link, but I'll soon have a copy this document.

Prog.
The first three documents listed in comment #4 are very old (more than 10 
years), thus are quite irrelevant.
The last one (IS-1904) is in Hebrew, so its impact on international developers 
is probably quite limited.
Ooops!!!  In comment #7, I meant "comment #6".  Confused?
shouldn't textarea, being an html element, inherit it's direction from the
document, or from it's own attributes, rather than "decide" independently on
it's direction?
When it is HTML, the spec is quite clear, see 
http://www.w3.org/TR/html401/struct/dirlang.html#h-8.2

But the bug is about plain text.
the bug is about plain text in textarea, among other things. and in a textarea,
IMHO mozilla should not set the direction of the text based on the language
used, but based on the inherited dir attribute.

so i think textareas should be excluded from this bug, and only be handled as
bug 96160 suggests.
There is one very commonly needed situation where implementing this algorithm in
browser textareas would be beneficial: web interfaces for plain text emails.

If the user composes a plain text email that will be rendered using
per-paragraph direction, then the user should see the same result during
composition. Now, I'm not sure how to make this comply with standards, but the
need is definitely there.

Prog.
that was bug 98160, of course.
i can think of many needs for this feature, and many other needs that can be
fulfilled by bending the rules. but that's not why we're here, right?
> but that's not why we're here, right?

Well, in my case, wrong. I'm not into any elusive ideals. I just want to use the
best damn browser in existence and although Mozilla qualifies, it still has room
to improve.

Back to the issue at hand. Possible routes are (1) evolving the standard (not
very likely), (2) providing this functionality as an option/pref or (3) giving
it up altogether. As someone who mostly corresponds using plain text, I'd hate
to see the latter happens.

Prog.
and what if someone wants to start a generaly RTL paragraph with some english
word? i think the most natural solution, at least for text areas, but for
plain-text email messages in Mail & Newsgroups as well, is what is suggested in
bug 98160 - using ctrl+shift, or whatever is common on other platforms, if any.
or perhaps, a combination of the two solution.
(In reply to comment #15)
> and what if someone wants to start a generaly RTL paragraph with some english
> word? i think the most natural solution, at least for text areas, but for
> plain-text email messages in Mail & Newsgroups as well, is what is suggested in
> bug 98160 - using ctrl+shift, or whatever is common on other platforms, if any.
> or perhaps, a combination of the two solution.

I've already suggested a solution for such cases, in comment #2:

> The user should not have to be aware of the underlying workings of such
characters, 
> but merely to use standard direction controls (Ctrl+Shift or Direction
buttons). Mozilla
> should transparently handle the insertion of these characters in the beginning
of the active 
> paragraph, regardless of the current caret position.

Prog.
Blocks: 241587
you are not answering my concern. trying to guess the "right" direction based on
context may result in error, and in user dissatisfaction of the browser's
behaviour. therefore, i think we should stick to the inheritence + user-selected
(via keyboard shortcuts) direction scheme.
(In reply to comment #17)
> you are not answering my concern. 

I already did. See comment #12, but just to reiterate, let's do this over again...

> trying to guess the "right" direction based on
> context may result in error, and in user dissatisfaction of the browser's
> behaviour.

The same error may occur in plain text composition under Mail&News, yet I don't
thing you object to this functionality, right? Users can learn how to make the
best use of automatic paragraph direction, they do on Linux, as well as on
Windows applications such as Babylon Translator.

> therefore, i think we should stick to the inheritence + user-selected
> (via keyboard shortcuts) direction scheme.

Sticking to the old method will leave Mozilla textareas lagging behind the
functionality provided by mail applications, thus effecting many potential users
who use webmail via the browser. With automatic paragraph direction, webmail
users may compose plain text email with paragraphs of different directions and
still see the same layout as an intended recipient who uses a compatible mail
client (such as Outlook XP/2003 and KMail)

Prog.
prog,
a browser is not a mail client, and web mail is far from being the only, or even
the primary, use for textarea.

while in mail the situation is clear - you want to write a message, usually in
one language, on the world wild web, you cannot expect the situation in which
you will use the textarea.

so yes, i don't object to it in mail, but i doubt if it will be of much use in
the browser.
I disagree with much of your comment, but instead of spamming this bug with a
lengthy discussion over webmail usage and trends, let's do this in email or in
one of the forums.

Prog.
Look at line 23 in the screen shot here:
http://linmagazine.co.il/misc/images/screenshot-bluefish-1648.jpg?1148020213

The code is actaully OK, but displayed wrong. How do you propose to overcome
such problems?

The only reason that this seems to be an isolated problem is the fact that I
made sure that the HTML tags are in separate lines.

Unless we have a good way to deal with such cases, we are likely to get into
many messy situatuations which are worse then the current state (with the
directionality bookmarklets).
Your screenshot actually displays the opposite. It looks much more legible than
how a single direction textarea would have displayed the same code. In fact, it
further convinces me that this functionality could be useful not just for
plain-text webmail, but also for web authoring via the browser (which admittedly
is less common).

To make good use of automatic paragraph direction, users will have to know the
rules and how override them if the programs happens to choose the wrong
direction. The basics concepts of both are quite simple:

1. An English character in the beginning of a paragraph is "aligned" to the
left, while a Hebrew character makes the text aligned to the right.
2. To manually right align a paragraph, press the Right Ctrl+Shift. To
left-align a paragraph, press the Left Ctrl+Shift.

These instructions are somewhat simplified and dumbed down, but I'm sure that
most users will not find them difficult to follow.

Prog.
Did you miss my line here?
"The only reason that this seems to be an isolated problem is the fact that I
made sure that the HTML tags are in separate lines."

Don't get confused from the fact that I used a workaround.

This only works well when they are separate- when you have to deal with mixed
text in the same line, you end up with many cases like line 23.

Anyway, how are you planning to deal with cases like line 23 in the screenshot?
Remember- the text is actually ok there, just displayed wrong. Getting it to
display right (manually) produces wrong code.

Try to edit exsisting pages with Bluefish and you can see what I mean. It can be
a real mess. Again- don't get fooled by the fact that I used a workaround in my
screen shot.


(In reply to comment #23)
(In reply to comment #21)
Sorry, Shoshannah Forbes, but your example is not relevant.  We are trying to 
deal with *plain text*, and HTML code certainly does not qualify as such.
Formal languages (like HTML, or Java, ...) are a special case because they 
combine a generally LTR flow (all the syntactic words) with phrases which may 
be in a RTL language.  The orientation of each such phrase should be handled 
independently of the other phrases and of the surrounding LTR syntax.  A 
standard algorithm for plain text, even from Unicode, cannot cope with such 
complexity.
But this is not the problem at hand!
(In reply to comment #24)
> A 
> standard algorithm for plain text, even from Unicode, cannot cope with such 
> complexity.

ýThat is my whole point.


> But this is not the problem at hand!

Why not? Editing a wiki  or a weblog many times ivnvolves code syntax- and they
are done using the browser's textarea. As the bug refferes to textarea's and not
just email, we are going to have to deal with such problems.

And as you wrote- we don't have the tools to do that.

That is why I am opposed to having automatic directionality in textareas.


(In reply to comment #23)
> Did you miss my line here?
> "The only reason that this seems to be an isolated problem is the fact that I
> made sure that the HTML tags are in separate lines."

No I didn't, that's why I suggested to educate users about the benefits and
limitations of this functionality, hence the two rules.

(In reply to comment #24)
> Sorry, Shoshannah Forbes, but your example is not relevant.  We are trying to 
> deal with *plain text*, and HTML code certainly does not qualify as such.

Actually, no one seems to object for implementing this algorithm in plain text
scenarios (such as the mail client). Tsahi and Shosh simply disagree with my
suggestion to also implement this in browser textareas, but if they read comment
#14 again, they'll see that this is only suggested as an optional behavior, not
as the default.

Prog.
Changing the summary to reflect comment #14, option 2:

> Possible routes are (1) evolving the standard (not very likely), (2) providing 
> this functionality as an option/pref or (3) giving it up altogether. As someone 
> who mostly corresponds using plain text, I'd hate to see the latter happens.

Prog.
Summary: RFE: Automatic paragraph direction of plain text (received email, textareas, IRC messages, etc.) → RFE: Automatic paragraph direction of plain text (email, IRC messages, {optionally} textareas, etc.)
Well, it will be ok for email, as long as we don't get confused by quote marks,
and don't brake things like the "quote colors" extenstion (which works great
with moffie's auto-direction extention, which means that this can be done)
Mozilla Thunderbird displaying plain text Hebrew mail with auto-direction using
moofie's extention, and the "quote color" extention.
We should not brake this level of support.
(In reply to comment #25)
I don't want to take side in the discussion between Prog and Shoshannah, 
but... 
Shoshanna sustains that the textarea is no substitute for an HTML editor, and 
she is right.  She infers that there is no point in enhancing it to support 
less ambitious endeavours (e.g. plain text), and the logical relation escapes 
me.
(In reply to comment #30)
> and the logical relation escapes 
> me.

In practice, there are many many cases around the web where a textrea is used as
a "mini html editor". Two common examples are editing weblogs, and editing wikis.

As these are very common uses, I am afraid that in attempt to "support less
ambitious endeavours (e.g. plain text" we are going to brake current widely used
functionality (as I do not see any practical way for mozilla to detect that we
are editing "plain text" vs. text with coded syntax in any given textrea).

Forgot to write (sorry for the spam): these comments are obviosly only valid if
we attempt to implement this in textareas and not just in Email.

If we are only going to implement this in email, then the only concern that I
have is outlined in Comment #29 .
(In reply to comment #29)
> Created an attachment (id=149620)
> screenshot- current status with extentions
> 
> Mozilla Thunderbird displaying plain text Hebrew mail with auto-direction using
> moofie's extention, and the "quote color" extention.
> We should not brake this level of support.

shouldn't the vertical lines to the left of the message be on the right side?
Blocks: 296689
Blocks: Persian
Why is everyone assuming text widgets or plain text messages have paragraphs for
us to play with, to begin with? Think e.g. of one-liner text widgets;

Or suppose I like to type my line
breaks at ends of lines of the same
paragraph. What would you do then?
How will you tell whether the
previous line is a beginning of a
new paragraph or not?

(and this is regardless of how you're determining paragraph direction eventually).

Anyway, do textareas have DOM trees within them? I just inspected this bugzilla
page and it seemed to me like they don't, so what is being suggested w.r.t.
textareas? Creating a DOM subtree, or maybe doing paragraph handling non-DOMishly?

Another point: I'm not sure people would like having paragraphs switching
direction "on their own" as they're writing text. There's a difference between
seeing an incoming message in which you're not the one doing the writing, so
there's just one moment of surprise when it comes up, and composing a message,
when whenever you enter a new paragraph you may be surprised to find the text
switching sides.

Finally, it may be worthwhile considering an implementation of a more limited
feature, regardless of whether direction is auto-set or whether control chars
are used: The ability to set the direction a text widget 'paragraph' manually
_for_viewing_or_composition_purposes_only_ without this affecting the submitted
text (i.e. the same as what happens when you do a Ctrl+Shift+X, but for a single
paragraph rather than the entire text widget, or for the rest of text from this
point on without affecting what you've typed previously).

Ok, that's enough blurb for one comment :-)
I suggest following the HTML specification, i.e. the dir attribute of the 
textarea.
(In reply to comment #36)
> I suggest following the HTML specification, i.e. the dir attribute of the 
> textarea.

Let me elaborate: The display of HTML is governed by the HTML specification, 
http://www.w3.org/TR/html/ and http://www.w3.org/TR/html4/, not by the Unicode 
specification.

This bug isn't a bug.

There are some proposals for CSS3 to support use of direction=auto for
paragraphs, text input, textarea, etc.  So it's good to have such a support
before the standard make us to have.

Another thing is plain-text emails, that should follow unicode standard, not
html/css.  So text widgets (input and view) of mail application needs this feature.
(In reply to rosennej@qsm.co.il, comment #37)
> This bug isn't a bug.

That's right, it's an RFE.

For some reason most of the comments are centered on textareas, so perhaps some
more details are needed to clarify what this RFE is actually about:

1. This RFE is *first and foremost* about auto-direction in the mail and IRC
clients - not about textareas.

2. Textarea implementation is requested as an *option*. The phrasing in the
summary is sufficiently explicit about this ("optionally" anyone?), but I've
also bothered to repeat this fact in the comments more than once. Please read them.

2.1 Textarea with auto-direction is needed by anyone who uses webmail and needs
some visual cue as to how the message will be displayed in auto-direction-aware
mail clients (e.g. Kmail and Outlook). Hotmail and Yahoo alone have more than
200,000,000 users, so it's fair to say that at least a few of them use
plain-text and BiDi and will find such an *option* useful.

Prog.
Whiteboard: Please read comment #39 before commenting on textareas
Depends on Bug 231701 - format=flowed DelSp=yes not supported (RFC 3676)

Because:  Lake of line-break/paragraph-separator detection cause many BiDi
problems, i.e. mail client fails to auto-align paragraph.
Depends on: 231701
*** Bug 296689 has been marked as a duplicate of this bug. ***
Component: Layout: BiDi Hebrew & Arabic → Layout: Text
QA Contact: zach → layout.fonts-and-text
Keywords: rtl

The bug assignee didn't login in Bugzilla in the last 7 months.
:jfkthame, could you have a look please?
For more information, please visit auto_nag documentation.

Assignee: smontagu → nobody
Flags: needinfo?(jfkthame)

I think the original discussion here has been overtaken by events; given that we have HTML dir=auto and the CSS unicode-bidi property (including the plaintext value) available, this should be adequately covered.

Please file new issues for any remaining concerns with the features as currently implemented.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(jfkthame)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: