47154 - Line breaks disappear when entities present in text (newlines are not converted to spaces) (Newline characters do not translate to whitespace) ([TEXT] Certain characters make Mozilla run words together)

Status: VERIFIED FIXED

(Core :: Internationalization, defect, P3)

Description

[Hixie (not reading bugmail)]

Line breaks disappear to nothingness if we have a "—" character raw in the
paragraph's text. That is character 151, which of course is invalid in
ISO-8859-1 and HTML. <snip snide comments about suck.com's web design>

TO REPRODUCE:
   See attached test case or visit the remarkably innaccurate and now infamous
   suck.com article: http://suck.com/daily/2000/07/31/daily.html

ACTUAL RESULTS:
   Some words mash into each other. This happens at line breaks when character
   code 151 (0227, 0x97) is somewhere in that element.

EXPECTED RESULTS:
   Well duh. We should not lose whitespace just because the paragraph contains
   some mystical undefined character.

TESTED ON:
   Windows 2000 running commercial bits 6.0.17.2000072920.

Note: This bug is proof that some good _did_ come from that article. And yes, 
the total useful content of that article boils down to a 2048 byte testcase...

Comment 1

[Hixie (not reading bugmail)]

Attachment: Testcase showing problem along with several cases that work.

Comment 2

[Tomer Brisker]

Also exists on Linux, changing OS to all.

Comment 3

[Hixie (not reading bugmail)]

Another test case:
   http://www.eleves.ens.fr:8080/home/madore/.test/test-spaces.html
...which uses the — entity instead of a raw 151 character, and is thus
valid. Eek. Another testcase is coming up, showing this problem with α.

This is therefore a VERY serious issue. It will, as far as I can tell, show up
on EVERY page that includes entities, including many top100. It turns out that
there are many dups of this. They will be marked shortly.

Comment 4

[Hixie (not reading bugmail)]

Attachment: Updated testcase, uses valid entities to show problem.

Comment 5

[Hixie (not reading bugmail)]

*** Bug 44519 has been marked as a duplicate of this bug. ***

Comment 6

[Hixie (not reading bugmail)]

From the about-to-be-duped bug 45367:

------- Additional Comments From [e-mail] 2000-07-27 18:14 -------
Endorsing for nsbeta3 stopper as this is a correctness of rendering bug on
top100 sites that renders text unreadable
(unlessyoudon'tmindreadingEnglishwithoutspaces).

Comment 7

[Hixie (not reading bugmail)]

*** Bug 45367 has been marked as a duplicate of this bug. ***

Comment 8

[tenthumbs]

This seems to depend on the entity. It works on a 2000803 build if I change
&alpha to &pound. Perhaps only entities outside the current charset cause
problems.

Comment 9

[Blake Ross]

er.  slightly more embarrassing is the fact that this happens on netscape's own
NS6 marketing page (http://home.netscape.com/browsers/6/datasheet/index.html).
Ian, maybe you could tell them how to fix it up...?

Comment 10

[ekrock's old account (dead)]

chrisn -- Apparently a regression is showing up on some of our marketing pages. 
FYI. Hopefully this will become [nsbeta3+] but be aware we may have site 
readability problems during nsbeta2.

Comment 11

[Eugene Savitsky]

I have a some most popular russian sites, where I see this problem. I think 
there will be many critics not fixing such a problem for second beta 
(espessially in Russia where NS PR1 was a disaster). :(

Actually should be nsbeta2+...

When it can't be fixed for nsbeta2, this should be in relnotes written with 
big-big font (and in russian too ;))) ).

Comment 12

[Tom Mraz]

As I noted in the bug 44519, it isn't only issue with russian sites but with all
international sites as well. It should be fixed as soon as possible!

Comment 13

[Eugene Savitsky]

But look at attachment in 47895. Is it cyrillic only?

Comment 14

[Hixie (not reading bugmail)]

*** Bug 47495 has been marked as a duplicate of this bug. ***

Comment 15

[Hixie (not reading bugmail)]

*** Bug 48452 has been marked as a duplicate of this bug. ***

Comment 16

[harishd]

Document Loaded:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html lang="en">
<head>
   <title>Mozilla Tests: line breaks and perfectly valid entities</title>
</head>
<body>
<p>first^M
second &alpha;</p>

</body>
</html>

------------------------

Content model #1:

docshell=00F0A800
html@02614068 refcount=3<
  head@026163C8 refcount=2<
    title@021F25B8 refcount=2<
      Text@021F6880 refcount=2<Mozilla Tests: line breaks and perfectly valid en
tities>
    >
  >
  body@021F12E8 refcount=3<
    p@021A9A58 refcount=3<
      Text@021A9750 refcount=3<first\n\nsecond \u03b1>
    >
  >
>

Content model #2 ( after changing &alpha to &k ):

docshell=00F0A800
html@021AD928 refcount=3<
  head@021AD838 refcount=2<
    title@021DA0D8 refcount=2<
      Text@021DA060 refcount=2<Mozilla Tests: line breaks and perfectly valid en
tities>
    >
  >
  body@021DE838 refcount=3<
    p@02625E78 refcount=3<
      Text@02625C30 refcount=3<first\n\nsecond &k;>
    >
  >
>

Note: In both the cases ( refer the content model ) the line breaks are 
preserved by the parser.  

This is definitely a layout bug.  Reassigning bug to waterson.

Comment 17

[harishd]

Attachment: Further reduced test case ( from which the above content model was derived )

Comment 18

[Chris Waterson]

The problem is that nsJISx4501LineBreaker doesn't think that ASCII 0x0a ('\n') 
is a space:

http://lxr.mozilla.org/seamonkey/source/intl/lwbrk/src/nsJISx4501LineBreaker.cpp
#211

It looks like this was changed by shanjian in r1.25:

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&ro
ot=/cvsroot&subdir=mozilla/intl/lwbrk/src&command=DIFF_FRAMESET&file=nsJISx4501L
ineBreaker.cpp&rev2=1.25&rev1=1.24

Comment 19

[Chris Waterson]

Attachment: proposed fix

Comment 20

[Chris Waterson]

The fix is to treat ASCII 0x0a and 0x0d ('\n' and '\r', respectively) as
whitespace characters. shaijin/ftang, does this seem right?

Comment 21

[Chris Waterson]

Attachment: better fix; avoid re-evaluation of macro argument

Comment 22

[Hixie (not reading bugmail)]

*** Bug 48234 has been marked as a duplicate of this bug. ***

Comment 23

[Frank Tang]

Is this a dup of 46239 ?

Comment 24

[Chris Waterson]

fix checked in

Comment 25

[Katsuhiko Momoi]

CC'ed myself on this.

Comment 26

[Gervase Markham [:gerv]]

*** Bug 48167 has been marked as a duplicate of this bug. ***

Comment 27

[Sean Richardson]

*** Bug 49492 has been marked as a duplicate of this bug. ***

Comment 28

[Chris Waterson]

*** Bug 48164 has been marked as a duplicate of this bug. ***

Comment 29

[Hixie (not reading bugmail)]

*** Bug 49666 has been marked as a duplicate of this bug. ***

Comment 30

[Hixie (not reading bugmail)]

*** Bug 49812 has been marked as a duplicate of this bug. ***

Comment 31

[Hixie (not reading bugmail)]

*** Bug 48828 has been marked as a duplicate of this bug. ***

Comment 32

[Jan Carpenter]

verified:
2000-09-05-08-M18 : win32
2000-09-05-08-M18 : linux
2000-09-05-08-M18 : mac

Comment 33

[Hixie (not reading bugmail)]

*** Bug 51553 has been marked as a duplicate of this bug. ***

Comment 34

[Loco]

*** Bug 49389 has been marked as a duplicate of this bug. ***

Comment 35

[Fabian Guisset]

*** Bug 54877 has been marked as a duplicate of this bug. ***