40179 - doctype autodetection of text/html too aggressive

Status: RESOLVED INVALID

(Core :: DOM: HTML Parser, defect, P3)

Description

[Sean Richardson]

Currently, the doctype autodetection code appears to be calling any file 
"text/html" if it contains a string that looks like an SGML-ish tag ( <foo> ) 
near the top of the file, whether or not the tag is a valid HTML start-tag. 

This problem is currently made very apparent by bug 40060, where files with 
text/plain metadata are nonetheless being autodetected, but there are enough 
README files and the like on ftp servers and local filesystems for this to be 
an issue after 40060 is fixed.

This causes any file for which Mozilla does not have any content type 
metadata, that also quotes HTML, to be rendered as HTML despite being properly
text/plain. I'd guess that autodetection for XML, and XHTML, and any other
SGML applications would rely on the DOCTYPE declaration, so those files
should be out of this problem domain.

Suggested:
For any file without content-type metadata specified:

1. Check first buffer for DOCTYPE
   HTML (any): --> text/html
   Other: --> go to appropriate handler
   None: --> continue
2. Check for an <HTML> start-tag
   Yes: --> text/html
   No: --> continue
3. Check for a <TITLE> start-tag
   Yes: --> text/html
   No: --> continue trying to detect other content types

Following this algorithm would still mean that files with no metadata to say 
that they are text/plain would be handled as text/html if they contain <TITLE> 
or <HTML>near the top (see bug 29049), but that is still much better than the 
current behaviour.

IMO, a file meant to be HTML that contains no <TITLE> element (IIRC, has always 
been mandatory), no <HTML> element, and no DOCTYPE, that also has neither a
.htm, .html, or .shtml extension nor HTTP content-type metadata does not deserve
to be recognized as text/html -- and likely is not anyway. See the testcase for
bug 40096 for an example of text that quotes, but is not, HTML.

Comment 1

[rickg]

The text/plain issue is fixed in my tree. I'm not sure if there are any other 
issues here. 

Comment 2

[Sean Richardson]

Attachment: short file beginning with <foo> tag; none/none mime type

Comment 3

[rickg]

Oh, and this: an html 3.x/4.x(transitional) file simply do not need to have 
<html>, <title> or <body>. Using their presence in choosing a doctype is 
misguided.

Comment 4

[Sean Richardson]

Attachment: Simulated beginning of mailing list archive file

Comment 5

[Sean Richardson]

The issue is that there are many other uses of paired angle brackets besides
HTML, many in text files that need to be displayed with whitespace preserved
to be readily human-parse-able. 

As an example of real-life usage, imagine a mailing list archive file at
ftp://ftp.some.org/pub/some-list/archive/may2000 beginning thus:
  From: [e-mail]
  Date: 1 May 2000 18:50:29 -0000
  To: List Member <[e-mail]>
  Subject: [List!]  IT'S A FLOOR WAX!  IT'S A DESSERT TOPPING! 

  Message....

If this is treated as text/html, the email address disappears, and the messages
become hard to delimit, let alone read.

Comment 6

[Sean Richardson]

Regarding the advisability of using HTML elements to distinguish HTML text 
from other text that may use angle brackets:

HTML specs prior to 2.0 recommend but do not require a <TITLE> element:
http://www.w3.org/History/19921103-hypertext/hypertext/WWW/MarkUp/Tags.html
http://www.w3.org/MarkUp/HTMLPlus/htmlplus_50.html
http://www.w3.org/MarkUp/draft-ietf-iiir-html-01.txt

From HTML 2.0 forward, the <TITLE> element has always been required:

Quoting from the HTML 2.0 spec, http://www.ics.uci.edu/pub/ietf/html/rfc1866.txt
  5.2.1. Title: TITLE
     Every HTML document must contain a <TITLE> element.

Quoting from the HTML 3.0 non-spec, http://www.w3.org/MarkUp/html3/dochead.html,
under "TITLE":
  Every HTML document must contain a TITLE element. 

Quoting from the HTML 3.2 spec, http://www.w3.org/TR/REC-html32.html,
under The Structure of HTML documents:
  Every HTML 3.2 document must also include the descriptive title element. 
  A minimal HTML 3.2 document thus looks like: 

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
  <TITLE>A study of population dynamics</TITLE>

Quoting from the HTML 4.01 spec, section 7.4.2, The TITLE element
http://www.w3.org/TR/REC-html40/struct/global.html#h-7.4.2 --
  Every HTML document must have a TITLE element in the HEAD section.

Quoting from the HTML 4.01 loose DTD http://www.w3.org/TR/REC-html40/loose.dtd
  <!-- The TITLE element is not considered part of the flow of text.
         It should be displayed, for example as the page header or
         window title. Exactly one title is required per document.
      -->
  <!ELEMENT TITLE - - (#PCDATA) -(%head.misc;) -- document title -->
  <!ATTLIST TITLE %i18n>

Under this proposal, the presence or absence of the <TITLE> attribute is moot 
if there is a DOCTYPE or <HTML> element, or if there is a content type header 
or a .html extension; only ancient or invalid HTML documents could possibly 
slip through all five layers of that sieve, and they must be truly rare.

Comment 7

[rickg]

The doctype analysis code uses the mimetype, doctype (if present) and other 
hints from the networking library to choose a DTD. The bug fix concerning text 
files will land shortly.

Comment 8

[bsharma]

updated qa contact.