385776 - make entity handling match IE7 / HTML5

Status: RESOLVED FIXED

(Core :: DOM: HTML Parser, defect)

Description

[Simon Pieters [:zcorpan]]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a6pre) Gecko/20070620 Minefield/3.0a6pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a6pre) Gecko/20070620 Minefield/3.0a6pre

Semi-colonless entities in content aren't supported (001). ⟨ and ⟩ don't return the right characters (per HTML5) (001..003). ™ is not supported (001..003). U+002D (-), U+002E (.), and U+005F (_) don't terminate entities in attributes (004).

Reproducible: Always

Steps to Reproduce:
1. Run the test cases 001..004
2.
3.



Relevant spec: http://www.whatwg.org/specs/web-apps/current-work/multipage/section-tokenisation.html#tokenising

Comment 1

[Simon Pieters [:zcorpan]]

Attachment: Entities in content

Comment 2

[Simon Pieters [:zcorpan]]

Attachment: 002: Entities in attributes

Comment 3

[Simon Pieters [:zcorpan]]

Attachment: 003: Entities in attributes

Comment 4

[Simon Pieters [:zcorpan]]

Attachment: 004: Characters that terminate an entity in attributes

Comment 5

[Boris Zbarsky [:bzbarsky]]

Note to whoever decides to fix this: don't screw up view-source of XML.

Comment 6

[Daniel Gutson]

(In reply to comment #1)
> Created an attachment (id=269684) [details]
> Entities in content

Hi, I'm fixing bug 427240 which tries to coveer this tetstcase (entities in content). Please take a look to comment #15 https://bugzilla.mozilla.org/show_bug.cgi?id=427240#c15

Are lang and rang codes wrong here? And, should I add TRADE in uppercase too?

Please let me know.

Thanks!

  Daniel.

Comment 7

[Daniel Gutson]

(In reply to comment #5)
> Note to whoever decides to fix this: don't screw up view-source of XML.

Hello Boris, could you please attach a test case, so I ensure I don't screw up anything?

Thanks!
  Daniel.

Comment 8

[Boris Zbarsky [:bzbarsky]]

When viewing source, the entire entity name should be an error:

  data:text/xml,<root>&ampersand</root>

When viewing source, the "amp" part should be an error:

  data:text/xml,<root>&amp ersand</root>

There might be other situations that need testing too.  I don't recall all of them offhand, but the point is that the HTML tokenizer is used for XML view-source, so any changes to it need to be carefully thought about in that context.

Comment 9

[Daniel Gutson]

Sorry Boris, I didn't understand your example :(   (please note I'm new here).
I understand that the same parser is used both for HTML and XML.

I beg you to create a minimalistic XML with your examples above, so I can open it with FF before and after my fix, and check that it is parsed in the same way.

Thanks!

  Daniel.

Comment 10

[Daniel Gutson]

With the newly added patch of bug 427240, all testcases of this bug report pass (except for lang and rang). I'm waiting for a response about this. If we agree on that, then this bug can be closed as well.
  Daniel.

Comment 11

[Daniel Gutson]

Attachment: output of testcase #2 with patch 427240

Hi, this is how my patched FF renders the output of testcase #2.
As you can see, the left angle and right angle brackets are properly rendered with the existing values. So, PLEASE CONFIRM that I should change them, since once I do it, they will be no longer properly rendered.
(unless I should change them in the rendering engine as well).

Thanks!
  Daniel.

Comment 12

[Simon Pieters [:zcorpan]]

(I think that U+3008 and U+3009 don't properly render for you because you don't have any glyphs for them in your fonts.)

Please note that HTML 5 has changed wrt ⟨ and ⟩ since this bug was reported:

http://www.whatwg.org/specs/web-apps/current-work/multipage/named.html#named
http://html5.org/tools/web-apps-tracker?from=1285&to=1286

lang; 	U+027E8
rang; 	U+027E9

Comment 13

[Henri Sivonen (:hsivonen)]

(In reply to comment #12)
> (I think that U+3008 and U+3009 don't properly render for you because you don't
> have any glyphs for them in your fonts.)
> 
> Please note that HTML 5 has changed wrt ⟨ and ⟩ since this bug was
> reported:
> 
> http://www.whatwg.org/specs/web-apps/current-work/multipage/named.html#named
> http://html5.org/tools/web-apps-tracker?from=1285&to=1286
> 
> lang;     U+027E8
> rang;     U+027E9

Fixed in the HTML5 parsing repo with this spec change. However, the default font for U+027E8 and U+027E9 looks terrible on Mac.

Comment 14

[Henri Sivonen (:hsivonen)]

Attachment: Screenshot of HTML5-compliant rendering

Comment 15

[Henri Sivonen (:hsivonen)]

The ugliness is bug 431559.