Closed Bug 229367 Opened 21 years ago Closed 13 years ago

<br> confuses our bidiness (punctuation before <br> at end of line starting with number doesn't follow paragraph directionality)

Categories

(Core :: Layout: Text and Fonts, defect)

Product:
Core
Component:
Layout: Text and Fonts
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: bugzillamozilla, Unassigned)

References

(Blocks 3 open bugs,
URL
)

Details

(Keywords: html5, rtl, testcase)

Attachments

(4 files)

Testcase #1
535 bytes, text/html; charset=windows-1255
Details
another testcase
792 bytes, text/html; charset=windows-1255
Details
Testcase #2 (pre-formatted, no <br> tags)
492 bytes, text/html; charset=windows-1255
Details
some examples of <br> and bidi interaction
6.96 KB, text/html; charset=UTF-8
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5) Gecko/20031007
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5) Gecko/20031007

This happens in lines that end with words of the opposite direction, followed by
lines that start with words of this opposite direction, or followed by lines
that start with numbers.

The linked testcase will make this much easier to understand.


Reproducible: Always

Steps to Reproduce:
1. In an RTL textarea, write a Hebrew word followed by an English one and a full
stop.
2. Start the next line with an English word or a number.

Actual Results:  
The full stop at the end of the first line appears to the right of the English word.


Expected Results:  
The full stop should appear at the visual end of sentence - to the left of the
English word.

Arcane workaround: use RLM (Alt+0254) or LRM (Alt+0253) at the end of lines. See
testcase.

Prog.
Attached file Testcase #1 — 
Attaching the linked testcase...

Prog.
I will look again tomorrow when less jet-lagged, but isn't Mozilla's behaviour
correct according to the Unicode Bidi Algorithm?
I'm not sure about the Unicode BiDi algorithm, but certainly we can't expect
most users to know about control characters, now can we? not when other browsers
handle these situation perfectly - out of the box and without expecting the user
to come up with arcane workarounds.

Prog.
There's definitely a bug here:
   http://www.hixie.ch/tests/adhoc/unicode/bidi/010.html

There's no way our rendering of that testcase is per the algorithm.

See also:
   http://www.hixie.ch/tests/adhoc/unicode/bidi/011.html (similar, works fine)

It would appear the <br> is what is confusing us.

(Original testcase:
   http://oren.gomen.org/mozilla/control_characters/testcase1.html
...)
URL: http://oren.gomen.org/mozilla/control... → http://www.hixie.ch/tests/adhoc/unico...
Keywords: testcase
Summary: Punctuation marks appear in the wrong side of embedded sections with opposite direction → <br> confuses our bidiness (punctuation before <br> at end of line starting with number doesn't follow paragraph directionality)
I think the first question should be "why does [Enter] in a text area add a
<br>?" Is there already a bug report on that?
Daniel: see comment 5.
I don't understand the question here. If the question is "why do we insert
a <br> instead of just a CR", that's because the textarea is really an editor, and
like in all instances of the editor, empty lines need a frame or they won't be
selectable. We solve this problem with <br>. Known old problem... Whoever fixes
this big problem wins my eternal consideration.
as can be seen from the test case (attachment 137938 [details]), a text area is just an
easy way to reproduce this bug. it happenes outside of textareas too, in regular
page text.
In regular page text I consider this as INVALID or Evangelism, because authors
can and should start a new block element rather than using <br>. For example, if
attachment 137938 [details] was a regular page I would just say "use an <ol>".

At least according to HTML [1] and Unicode [2] <br> has the semantics of LINE
SEPARATOR, not CR, and should not cause any kind of paragraph formatting.

[1] http://www.w3.org/TR/html4/struct/text.html#edef-BR
[2] http://www.unicode.org/reports/tr14/index.html#BK
Ok, I fixed my testcase:
   http://www.hixie.ch/tests/adhoc/unicode/bidi/010.html

INVALID?
Status: NEW → RESOLVED
Closed: 21 years ago
Resolution: --- → INVALID
smontagu, even if <br> is just a line seperator, punctuation marks in RTL
segments, if they appear at the end of a LTR word, should be on the left end of
the word.

unless i didn't understand what you mean here.

and how come in ian's test case in comment 10, two different lines are rendered
as identical?
Re comment 10, I have two questions:

(a) Do we want to emulate IE's behaviour as a quirk?
(b) The modified testcase shows a genuine bug which I also encountered when
testing this issue: there is extra space between the two words on the first line
of each pair, which contradicts the UBA's statement [1] that "trailing white
space will appear at the visual end of the line (in the paragraph direction)".
Do we want to keep this open for that issue, or open another report?

[1] http://www.unicode.org/reports/tr9/#L1

Re comment 11: Punctuation between two LTR words should appear at the right edge
of the first word, even in a RTL paragraph. Consider a case where you have
"Profession: Engineer" in a RTL paragraph, and then suppose that you narrow the
margin so that the line wraps just between those two words. Should that make the
colon jump from the right edge to the left edge of the first word?
A half baked idea for a programmatic workaround has occurred to me: what if we
inserted before the <br> in the editor a directional mark in the paragraph
direction (i.e. LRM in left-to-right paragraph and RLM in right-to-left paragraph)?

The following:

   HEBREW english . 123 HEBREW

...becomes:

   HEBREW english . 123 HEBREW
   111111122222222222221111111

...so the "." is LTR and part of the surrounding run of english text.

The reason for this is that the numbers (weak, section 3.3.3:W2) are resolved
before the punctuation (neutral, section 3.3.4:N1). The numbers thus get the
direction LTR (from the english text; you search backwards) and the punctuation
then finds itself between two LTR runs so it becomes LTR too.

A dot at the end:

   HEBREW english .
   1111111222222211

...gets the direction of the paragraph (RTL in this test).

The important point in the test is to realise that the two lines are actually
all part of the same paragraph.

Is there a case we are doing incorrectly?
> (a) Do we want to emulate IE's behaviour as a quirk?

I strongly feel we have enough (or even too many) quirks. If we want to add a
quirk, let's remove another one first. What quirk do you think we should remove?


> (b) The modified testcase shows a genuine bug which I also encountered [...]

That's bug 132561.


> what if we inserted before the <br> in the editor a directional mark in the 
> paragraph direction [...]

The <br> hack in the Editor has caused lots and lots of problems. The correct
fix is to remove it altogether and replace it with a real newline character. I
really don't think we should add even more characters to the mix.
Is this another case where we decide to stick with the UBA and the hell with users?

Simon, even if we do provide a workaround based on control characters, it
doesn't help one bit with existing texts. So not only do users have endure
Mozilla's incompatibility with many (substandard) websites, they have to learn
to live with HyphenMinus at the wrong side of the number, punctuation marks at
the wrong side of the word and the like. Can we really expect anyone to switch?
not when they only get "half the internet" (I don't remember when I heard that
about alternative browsers, but it fits well in this context).

Prog.
>> (b) The modified testcase shows a genuine bug which I also encountered [...]
>That's bug 132561.

I don't think it is, although it overlaps with it in the white-space:normal
case. Let me try a testcase with white-space:pre
So... what do we suggest to users who:

1. Can't input RLM/LRM in their platform of choice (for technical keyboard
layout reasons)
-or-
2. Don't understand the concept of hidden control characters.
-or-
3. Wish to read existing texts as their author envisioned them.

If this behavior is "by design" then we should at least be able offer solutions
to real world problems that are caused by this decision.

Prog.
Blocks: 240501
> In regular page text I consider this as INVALID or Evangelism, because authors
> can and should start a new block element rather than using <br>.

Why? Am I is it improper or disallowed to have a multi-line paragraphs in my HTML?

> For example, if
> attachment 137938 [details] was a regular page I would just say "use an <ol>".

You're right in this case, the testcase is a bad example because it uses <br>'s
as list item separators which is wrong. But this is not the general case!


> 
> At least according to HTML [1] and Unicode [2] <br> has the semantics of LINE
> SEPARATOR, not CR, and should not cause any kind of paragraph formatting.
> 

The consideration of the last punctuation mark in a line within in RTL paragraph
to be RTL is not paragraph formatting, it's line formatting. At least that's
what my intuition says. 

Another point is that regardless of whether some standard says otherwise, this
is plain wrong. Intolerable. It's like with bug 73251: if the standard does not
fit reasonable expectations then the standard must change.

like bug 73251, the way to fix this bug probably goes through the unicode
consortium.
*** Bug 150568 has been marked as a duplicate of this bug. ***
Looks like bug 73251 was fixed (as bug 240943) by going through the Unicode 
Consortium. 
 
I couldn't find references to this problem in Unicode's public mailing list. 
Did anyone propose this change to the Unicode Consortium? 
So far, I gathered this is the contact channel: 
http://www.unicode.org/reporting.html 
Or does the Mozilla Foundation have a better channel for feedback? 
IANAUE (I am not a Unicode expert), but if I read the per-line examples here:

http://www.unicode.org/reports/tr9/#Resolving_Implicit_Levels

correctly, they indicate that the behavior of considering the period at the end
of a line as part of the embedding run continuing upto it (rather than the
paragraph embedding level) - which is the buggy behavior this bug is about - is
not what the UBA requires, but rather the opposite of the requirement. Do
correct me if I'm wrong.
Eyal, as I understand the Bidi algorithm the problem is that even if the <br> is
set to the paragraph embedding level in rule L1, the embedding level of the
punctuation characters has already been set before that in rule N1 to the level
of the text on both sides (if it has the same direction) or to the paragraph
level (if it has different directions).

I have sent mail to the bidi discussion list at unicode.org proposing that the
<br> should be set to the paragraph embedding level at an earlier stage in the
process.
(In reply to comment #25)
> set to the paragraph embedding level in rule L1, the embedding level of the
> punctuation characters has already been set before that in rule N1 to the level
> of the text on both sides (if it has the same direction) or to the paragraph
> level (if it has different directions).

But that's a 'good thing' in our case: suppose the paragraph is RTL and the
first line has a word in English, then a period, than the line break - like in
the test case. Like you said, in rule N1 the level of the period is set to the
paragraph level (since it is part of a sequence of neutrals which begins after
the word in English and ends with the first Hebrew character on the next line),
i.e. its embedding level is set to 1. Great! Now when L2 is applied, the period
should not be reversed - only the word in English need be reveresed.

Or have I missed something?

(In reply to comment #26)
> But that's a 'good thing' in our case: suppose the paragraph is RTL and the
> first line has a word in English, then a period, than the line break - like in
> the test case.

The question is, what is the first directional character after the line break?
If it's RTL, then the reordering is as you describe. But in the test case it's a
number, which was set to LTR in rule W7, so the text on both sides of the period
is LTR, i.e. level 2.
> The question is, what is the first directional character after the line break?
> If it's RTL, then the reordering is as you describe. But in the test case it's a
> number, which was set to LTR in rule W7, so the text on both sides of the period
> is LTR, i.e. level 2.

Yes, you're right. I would think the solution would be to restart level runs at
line breaks; this way W7 would not un-neutralize the period.


In fact, I suggest that we change the UBA implementation to do exactly that. You
know they'll approve this eventually. Just like they did with bug 73251. It's
what the UBA should say.
Blocks: 269759
Now that I have editbugs, I'm reopening the bug. It is certainly not INVALID. I
have made a suggestion in comments #28, #29. If people reject my suggestion, and
any other possible fix, then the status will change to WONTFIX (_please_ don't
do that, though... ).
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Assignee: mkaply → eyalroz
Status: REOPENED → NEW
I'm sorry, I can't take this bug due to lack of time to work on it.
Assignee: eyalroz → mkaply
I will however make some some hopefully useful comments.

It seems there are two alternative ways to modify the UBA implementation. The
first alternatively would be to modify P1 so that text is pre-split into lines
rather than into paragraphs, i.e. instead of 

"P1. Split the text into separate paragraphs. A paragraph separator is kept with
the previous paragraph. Within each paragraph, apply all the other rules of this
algorithm."

We would have

"P1. Split the text into separate lines. A line/paragraph separator is kept with
the previous line. Within each line, apply all the other rules of this algorithm."

This will mean minimal or no changes in nsBidi.* , which assumes P1 is performed
externally.

The second alternative is to consider line breaks to be of type L if the
paragraph embedding level is even, i.e. an LTR paragraph, and of type R if the
paragraph embedding level is odd, i.e. an RTL paragraph. This second alternative
problably means changes to some nsBidi.cpp functions (maybe also to nsBiDi.h
although I'm not too sure), but it has the advantage of allowing BiDi control
chars (e.g. LRO) to work across line breaks.

Finally, we need to consider whether or not the behavior should be different for
'manual' line breaks (e.g. <br>s) and for automatic line breaks due to wrapping.
Attached file another testcase — 
Adding another testcase which does not raise the question of "why don't you
just make it an ordered list" and emphasizes how this bug may occur in 'simple'
text. Note that the two sentences can be said to form a single semantic whole
which should be a single paragraph, despite an author's decision to break the
line at certain points. So "why don't you just make it two paragraphs" is
invalid here, or at least as invalid as it is in any case with <br>'s instead
of paragraph breaks.
See W3C I18N Core WG comments at
http://lists.w3.org/Archives/Public/public-i18n-core/2005JanMar/0065.html

We believe Mozilla is behaving correctly, and that changing the behaviour will
break the more general expectations of behaviour.
Richard, there's no argument that, in that regard (resetting levels at line
breaks), Mozilla behaves by the Unicode Standard whereas Internet Explorer does not.

--

Let's review more realistic cases:

 <p dir="rtl">
  THE ENGLISH MAN SAID "hello,<br>
  my friend" AND SMILED.
 </p>

Mozilla would be right (according to Hebrew grammar), rendering:

  hello," DIAS NAM HSILGNE EHT
       .DELIMS DNA "my friend

Another case (note the order):

 <p dir="rtl">
 I USE mozilla,<br>
 konqueror, opera AND internet explorer.
 </p>

Mozilla would render:

                          mozilla, ESU I
 .internet explorer DNA konqueror, opera

which would look wrong Hebrew-grammar-wise, but it's just making an
already-existing problem more visible -- the comma is part of the Hebrew item
listing and, by the Hebrew grammar rules, it should be:

 [...]
 .internet explorer DNA opera ,konqueror <-- note the comma's position

and if the two lines would be unified, grammar dictates it should render:

 .internet explorer DNA opera ,konqueror ,mozilla ESU I

whereas, in both IE and Mozilla, it'll render:

 .internet explorer DNA mozilla, konqueror, opera ESU I

To achieve the grammatically-correct rendering, in any of the browsers, the user
will have to put in RLMs!

---

So, places where IE has it right due to a linebreak, are places where there'd be
a grammar mistake were it not for the linebreak. Therefore, I take my words back
about needing to change the UBA cause IE's "standard" is better lingually. The
reason to change the UBA is to fit a defacto standard which isn't better than
the UBA's standard.
Excuse the spam, but I'm posting my reply to Richard:

Thank you for your reply Richard,

I still find that your reply is more of a presentation of the current views of
W3C / the Unicode people on the matter.

If this had been an apriori argument about what <br>'s should mean and how they
should be used, I would find your view a perfectly reasonable alternative
(although it would mean there would have to be a way to 'break a line
syntactically' without switching to a new paragraph, which in the current scheme
of things <br> is not intended to do).

However, there is the huge corpus of existing HTML documents with RTL text which
all assume an alternative interpretation of <br> - being not just white space,
but having some semantic significance of a break. True, this in some part due to
MSIE's conventions, but like I argued in my previous e-mails, this assumption
has merit for itself. I know, the 1. xxx 2. xxx is not the best of examples when
considered as HTML, but it is a very common example when plain text is displayed
as HTML. i.e. , if you were to take the following block of text:

-----
Lorem ipsum dolor sit amet, consectetuer
adipiscing elit, sed diam nonummy nibh
euismod tincidunt ut laoreet dolore magna
aliquam erat volutpat.
Ut wisienim ad minim veniam, quis nostrud
exerci tation ullamcorper suscipit lobortis
nisl ut aliquip ex ea commodo consequat.
Duis autem vel eum iriure etc etc
-----

and 'HTML'ify it (this is very common practice, not just in Mozilla and other
web browsers which feed text to their HTML renderer for display and/or edit text
messages as HTML, but also in news sites which store stories as plain text and
use it to prepare their displayed pages) - you would not make every line of text
a paragraph, on one hand, and usually not go to the trouble of running some
intelligent classifier to decide where paragraphs end (e.g. volutpat) and where
they do not (e.g. at consequat). What you would normally do is append <br>'s to
each line. Thus you see text (or html) like

most people in the room don't know the Hebrew word ;DROW<br>
3 people do know it.

or

most people in the room don't know the Hebrew word ;DROW<br>
REHTONA is a better known word.

These are the more representative example - at least from my experience - of the
use of <br>'s, in comparison to your provided example of

1. xxxxxx xxxx English,<br>
and more xxx.

Because one immediately finds one's self asking: Why put a forced line break in
the middle of a phrase? If it's a single phrase, authors prefer to keep it on
the same line; if it is broken by a <br>, the author most likely considers it
not to be an indivisible whole. And if this <br> comes from HTMLizing text, it
is usually less confusing to err by endowing the comma with the 'general'
direction of the text than to err by switching it to the other side of
'English'. The reason is that the first type of error does nonetheless
correspond somewhat to the order of reading (you only need to reinterpret an
'end-of-the-line' punctuation mark as an
'coming-after-the-last-word-in-this-line' punctuation mark) ; while the second
option, in case of an error, makes you pause when first viewing the comma or
period thinking that a new clause is beginning, then to move to the next line
only to find that this is not the case and that 'English' belongs to the
previous clause or sentence after all, and finally to re-read the text while
switching the comma or period back to the end of the line in your head.

I hope this clears up 'where I'm coming from' in requesting this change. This
situation seems to me not entirely unlike the problem we had with minus as a
number separator,

https://bugzilla.mozilla.org/show_bug.cgi?id=73251

, in which the behavior of the minus was modified to accomodate one of its
common uses in RTL text. There too, although MSIE was 'wrong' w.r.t. the Unicode
standard's original scheme, it turned out that the 'wrong' way of laying out the
minus was in general better than the 'right' way. I think we would be
hard-pressed to find more than a handful, if even that, of a people breaking up
their "English, and more" with <br>s, especially within RTL text.

Eyal

PS - As for audio-only browser, the point about <br>s being discarded is sort of
moot, since when reading the HTML or text, the punctuation mark will always come
after the last word if it is written after the last word - the 'switching' is
done only for visual layout.


Ilya, what you've brought up (english words separated with commas in a Hebrew
paragraph) is a different issue, pertaining to the UBA's behavior squarely
within level runs. I think we should not try to link that up with the question
of what happens at <br>'s.
CCing Richard Ishida (who probably missed the last few comments).

Prog.
(In reply to comment #5)
> I think the first question should be "why does [Enter] in a text area add a
> <br>?" Is there already a bug report on that?

For the record, the answer is now "yes": bug 240933.
This bug doesn't happen only when <br> tags are present. Testcase #2 shows that
it also occurs in pre-formatted texts.

Prog.
(In reply to comment #40)
> This bug doesn't happen only when <br> tags are present. Testcase #2 shows that
> it also occurs in pre-formatted texts.

Since this bug is explicitly about <br>, that testcase shows a different bug,
and one which is definitely valid without any changes to the Unicode Bidi
Algorithm. It's mentioned in the comment at
http://lxr.mozilla.org/seamonkey/source/layout/base/nsBidiPresUtils.cpp#486
   // XXX: TODO: Handle preformatted text ('\n')
No longer blocks: 269759
*** Bug 269759 has been marked as a duplicate of this bug. ***
(In reply to comment #43)
I would like to hear people's opinions, especially dbaron's, regarding my comment 36.

I've used BR in both ways, a number of times -- but I didn't have any problems since all my text was LTR.  Probably most of my uses have been for breaks that do have semantics, rather than just a place I'd like to ensure the line be broken.

Given what MSIE does, it sounds like we should probably switch to being compatible with it, since there are a significant number of Websites and applications that depend on the MSIE behavior.
Blocks: 137995
Assignee: mozilla → nobody
QA Contact: zach → layout.bidi
Not going to block 1.8.1 for this, but we'd be happy to consider a patch that's baked on the trunk.
Flags: blocking1.8.1? → blocking1.8.1-
This is a recent discussion on the matter on #devs@irc.mozilla.org :

http://pastebin.mozilla.org/3187
That was a temporary post to pastebin. Resubmitted as http://pastebin.mozilla.org/3190
Blocks: 376359
Component: Layout: BiDi Hebrew & Arabic → Layout: Text
QA Contact: layout.bidi → layout.fonts-and-text
Keywords: rtl
Simon: are you aware of the latest status of this bug?  (Feeling kind of lost due to the number of comments here.)
Attachment #137938 - Attachment mime type: text/html → text/html; charset=windows-1255
Attachment #174368 - Attachment mime type: text/html → text/html; charset=windows-1255
Attachment #190274 - Attachment mime type: text/html → text/html; charset=windows-1255
HTML5 is apparently going to require that we change our behavior to fix this; it's also tested in the CSS 2.1 test suite, in the fifth test of:
http://test.csswg.org/suites/css2.1/20101001/xhtml1/bidi-breaking-002.xht
(I think tests 3 and 4 are invalid.)
Blocks: html5bidi
Depends on: 263359
Fixed in bug 263359
Status: NEW → RESOLVED
Closed: 21 years ago → 13 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: