Closed Bug 55264 Opened 24 years ago Closed 23 years ago

[DOCTYPE] Documents with unknown DOCTYPE should be displayed in strict mode

Categories

(Core :: DOM: HTML Parser, defect, P1)

Product:
Core
Component:
DOM: HTML Parser
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla0.9.5

People

(Reporter: ekrock, Assigned: dbaron)

References

Details

Attachments

(4 files)

perl script to find doctypes of sites trawled from dmoz
3.27 KB, text/plain
Details
preliminary patch
46.51 KB, patch
Details | Diff | Splinter Review
much improved patch
56.07 KB, patch
harishd
: review+
vidur
: superreview+
Details | Diff | Splinter Review
the new code within the above patch (easier to read than the diff)
14.76 KB, text/plain
Details 
Proposal for DOCTYPE handling hived off from bug 42525; see that bug for 
discussion and background.

(I currently have no position on this proposal; I'm opening this bug report so 
discussion can take place within this report.)

*** This bug has been marked as a duplicate of 55263 ***
Status: NEW → RESOLVED
Closed: 24 years ago
Resolution: --- → DUPLICATE
Oops... I should have read more closely.  Sorry.  I *thought* I read the titles 
and they were the same.
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Reassigned to Rickg/Parser. Nominating for rtm, like bug 42525.
Assignee: clayton → rickg
Status: REOPENED → NEW
Component: Layout → Parser
Keywords: rtm
OS: Windows NT → All
Hardware: PC → All
Are there any existing, currently "unknown" doctypes? Are there any real-world
documents that use such an existing, currently "unknown" doctype? Or is this a
pure forward compatibility issue?
This is a forward compatibility issue in the hypothetical cases that W3C
publishes HTML 4.02 with some minor corrections or that ISO comes up with a yet
unknown doctype.

As for existing "unknown" doctypes, one such doctype is the DTD Metrius
Presentational doctype declaration used at Motorola's site for example. It is
debatable, what an HTML browser should do with such a doctype.

How common is it for page authors or page editing software to accidentally 
introduce a typo in the DOCTYPE when editing markup? If it's common, we could 
create a real problem for product usability by applying strict layout to every 
page on the web with a corrupted DOCTYPE statement.

Another concern: have we looked at the DOCTYPE produced by every known authoring 
tool? We already know that Netscape Composer output an invalid DOCTYPE and had 
to build in awareness of that; I'm concerned about the risk that we may overlook 
bad DOCTYPEs output by other tools.

Finally, could someone use a spider to scan the web and generate a list of every 
unique DOCTYPE it found for HTML? *That* would be the way to make sure we 
weren't going to cause a lot of problems with this. Marking qawanted for this.
Keywords: qawanted
Marking [DOCTYPE] for easy searching.
Summary: Documents with unknown DOCTYPE should be displayed in strict mode → [DOCTYPE] Documents with unknown DOCTYPE should be displayed in strict mode
[rtm-]. The proposal may well have been meritorious, but unfortunately, no one
inside or outside of Netscape was able to do sufficient analysis of the
potential of this bug to cause regressions in time for us to commit this and
implement and accept a patch in time for RTM. Marking Future. Some will argue
that if we don't do this for RTM, we can't do it ever, but actually that's not
necessarily true. Depending upon the speed of market uptake of Netscape 6, the
length of time before the first point release, and further analysis of how many
existing web pages actually use unknown DOCTYPEs and would be affected by a
change either way, it may still make sense to consider this for the first point
release.
Whiteboard: [rtm-]
QA Contact: petersen → janc
I just wrote a perl script that would traverse dmoz.org and identify the
DOCTYPEs on each site. My computer isn't in a situation to run this over the
whole of dmoz, but I'll attach the script so that if someone wants to run it
over a larger subset of the web, they can do so (It should run on any system
that can support perl and wget).

The results for the first 100 sites (alphabetically by category on dmoz) are as
follows:

7
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
4
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
1
<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 3.2//EN'>
1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">
85
NO DOCTYPE

Since only 15 of these have doctypes at all, this is clearly not a large enough
sample, but the script can keep going indefinitely...

Suggested tweaks to the script if anyone wants to enhance it:

- Some sort of parallelizability (eg run 100 threads, or at least open 100 http
connections)
- The ability to save its status before terminating and then resume afterwards
 from the same place
- Pick the META GENERATOR tag out of the file if one exists, so that we can
identify what popular editors are putting in the doctype
- Some kind of crawling from the sites themselves (perhaps to a depth of 2 or
3), rather than only going to the single page linked from dmoz. The current
process doesn't even attempt to find the actual frames in a frameset, let alone
linked pages.
- Although wget knows that the file is text/html, the script doesn't retrieve
this information. If dmoz links to a plaintext file that has <!doctype in the
middle somewhere, you'll get a potentially very strange result...

If I get a chance to address any of these issues, I'll post updated versions of
the script in this bug. In the meantime, if anyone else wants to work on it (or
just to run it for a larger number of sites), go for it.

(attachment to follow...)

I'd just like to add something (I've already posted this as a bug which was
marked "invalid", and as a comment on another thread).

The subject of this bug is one of the things I fell are more lacking to Mozilla.
There *must* be a way to enable standards-compliant rendering with a
transitional doctype. Having a "quirks mode" must be of some use to many people
(not for me, though), but there must be a way to go around it without having to
go to strict mode.

Why this? Take for example one vastly used HTML authoring tool: Dreamweaver.
Pages generated by it have (questionably) transitional content, and they look
very good when shown in IE. But, if you look at them with Mozilla, you get the
same effect as NS4, which is not as good as IE's.

One example: tables with colored borders (both outline and inside the table).
Dreamweaver does this by setting a cell spacing higher than 0, and setting a
background color to the table. IE shows them great (even without a DOCTYPE
attached), because as far as I can tell it is not using a "quirks mode". But
looking in Mozilla with quirks mode it goes back to the same behaviour NS4 had:
showing the spacing between the cells as a white background instead of the table
background, ruining the effect.

Using a strict DTD makes Mozilla show them correctly, but it breaks almost the
whole page as Dreamweaver uses transitional syntax.

I'm not a huge fan of Dreamweaver myself (actually I hate it most of the time),
but I'm not a web designer (nor work with it, I do web programming), but the
designers here use it most of the time. And I can say that because of this
results are much better with IE than with Mozilla.

If anyone is interested in seeing the example in action, follow:
http://www.geocities.com/mvanzin/mozilla/table-strict.htm
http://www.geocities.com/mvanzin/mozilla/table-loose.htm

Same HTML, diferent DTD's (both with the DTD URL), and different results in
Mozilla. I'm not sure about if my intentions in the second table are correct,
but the first one (case I described above) should render equally in both cases
(by equally I mean equal to the strict mode one).

Hope this adds some food for thought.
(BTW, I'm adding myself to the CC list.)
This bug is not about transitional doctypes. This is about unknown doctypes.
Let's keep the bug focused.

(It is possible to activate the standards layout mode for transitional
documents. See: http://www.hut.fi/u/hsivonen/doctype.html)
Oops, sorry (the original bug, 42525, was about transitional. I think I did not
check the summary when going to this new thread).

BTW, thanks for the link. (I think that 4.0 transitional should also fire
standars mode, but...)
The issue about table backgrounds is, I think, covered by bug 4510.
Hmm, maybe I can add something to the discussion the. :-)

I'll try to base my comments on my previous comment (using the fact that many 
people use WYSIWYG HTML tools today, and they are mainly focused on producing 
nice output in IE, while mantaining an accetable result in NS4).

I'd say that using strict mode for unknown doctypes would be a very bad idea in 
this case. If you go around, you will see very few pages using strict layout, 
and when they do they generally use the correct doctype declaration. The same 
cannot be said about the much more common case of transitional doctypes. Most of 
the pages use transitional syntax, and many of them do not contain a doctype 
declaration.

That would leave us with three options: transitional with standards mode, 
transitional with quirks mode and old HTML 3 mode.

First one is the best in my opinion. I changed the doctype in documents at work 
today (using Henri's tip) and the results were better, but with minor quirks I 
will be watching more closely tomorrow.

Second one should be ok also, but then we fallback to my rants above. :-) It 
would not be in my preference to use this mode, but...

Third one is out of question I think. By doing so we were going to ignore CSS 
mostly, and results would be horrible.

This is a pretty tricky topic, but I think that it should, at least at this 
time, be resolved based on today's trends on HTML authoring. A nice idea would 
be (I think this was already suggested in some way): make one of them the 
default, and create an invisible preference (only editable going to the prefs 
file) to change it. That way, Mozilla can easily adapt to any decision made, 
and when people decide to change it then no problems would arise.
Remember that, by definition, "unknown" doctypes *excludes* all doctypes for the
most popular authoring tools (because, as described in this bug, we would have
to search out and identify what authoring tools use before we could fix this,
and then they become "known").

The aim here is, I think, that we should do something like the following:

1) Identify the doctypes that are widely used on the net, including *at least*
the ones used by popular authoring tools.
2) Decide what to do with these popular doctypes on a case-by-case basis, and
encode that knowledge (the majority of these will probably require quirks-mode
enabled, with a few exceptions such as XHTML, all STRICT variations and 4.0
Transitional with URL).
3) Treat doctypes that are *still* unknown as if they were strict.

The rationale here is that we need to render the web as it is now (for
compatibility) but we also need to render HTML5, XHTML 2, SOMEFUTUREML 7.6 etc
which we don't know about yet. The ones we *do* know about, we can decide what
to do with, but the ones we *don't* are ones that haven't been invented yet...
and those will *certainly* require strict handling.

I agree on the point that future (thus yet unknown) DTD's should be treated as 
strict mode. But I think that doing such a move *now* would break more things 
than fix them. At least until we get more standards compliance from everyone 
(browsers and authoring tools).

bug 55916 has an example of such a behaviour. If a new version of a popular tool 
is released and then includes a new DTD declaration, what will be the result? 
Maybe it will still output transitional syntax for better backward 
compatibility, but the DTD could be declared in a way that fools the doctype 
parser so it would think it was better to use strict parsing... and we would be 
breaking the rendering again.


Nominating for mozilla 0.9.  I think we need to fix this for mozilla 1.0 and we
need to get a bit of testing in before that happens.  I am willing to fix this.
Keywords: rtm → mozilla0.9
updated qa contact.
QA Contact: janc → bsharma
Because of bug 55916, I propose we WONTFIX this bug. If we want to encourage 
standards support, we should be encouraging XHTML, and we already do all of 
text/xml in standards mode.
Keywords: qawanted
Whiteboard: [rtm-] → WONTFIX?
That's a silly reason.  We should just add HotMeTaL's DOCTYPE to our list of
quirks doctypes.  This is crucial for future-compatibility, since AFAICT XHTML
won't be able to be sent as text/xml for a long time in the future since there
are still non-supporting browsers around today.
As I see it it's six of one and half a dozen of the other. We're going to get
as many people writing new text/html pages with DOCTYPEs we don't recognise 
and wanting strict layout as we are people publishing old pages with silly 
DOCTYPEs with typos or other weird things and expecting a compatible rendering.

XHTML2 is not going to be backwards compatible with XHTML1 or HTML4, and the
latest version of XHTML, 1.1, is already treated in strict mode:

   http://www.bath.ac.uk/%7Epy8ieh/cgi/compat-test.pl?DOCTYPE=%3C%
21DOCTYPE+html+PUBLIC+%22-%2F%2FW3C%2F%2FDTD+XHTML+1.1%2F%2FEN%22++%22http%3A%
2F%2Fwww.w3.org%2FTR%2Fxhtml11%2FDTD%2Fxhtml11.dtd%22%3E&MODE=full

So the only likely possible forward compatability problem is already covered.
I've said this before, but I really think we should do some type of
doctype-crawl of the web to find out what's actually out there. Perhaps we could
ask the ODP/dmoz people, or google, or altavista, or somebody with a big
database of web pages whether they have any exhaustive lists of the DOCTYPEs
found on pages in their catalogs.

Then we can look at pretty much all existing doctypes on a case by case basis -
and hard-code this knowledge for these doctypes - making it much easier to say
"if nobody's EVER used it before, then it's almost guaranteed to want standard
rendering".
> We're going to get as many people writing new text/html pages with DOCTYPEs
> we don't recognise and wanting strict layout as we are people publishing old
> pages with silly DOCTYPEs with typos or other weird things and expecting a
> compatible rendering.

I disagree. If there's an error in your page as fundamental as a bad DOCTYPE, 
then all bets are (or should be) off as to how a browser will render it. I'm 
with David on this one -- forward compatibility is more important than backward 
compatibility.
Keywords: mozilla0.9 → mozilla0.9.2
> If there's an error in your page as fundamental as a bad DOCTYPE, 
then all bets are (or should be) off as to how a browser will render it. 

I agree compleatly.
However do notice 1 important thing about the doctype if this will ever be a
"fix" for this bug.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 ...etc
is the exact same as
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 ...etc
since html markup is case insensitive.

Thus the casing on the word "HTML" (and only HTML) is not relevant and should
thus not yield an invalid doctype.

For "proof" compair with the corresponding XHTML (which is case sensitive)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 ...
At least this is how I have intrepreted the difference between high & low case
in HTML/XHTML declarations.
I'm taking this bug, P1/critical (for standards support)/0.9.5.  See my comments
on bug 60511.
Assignee: rickg → dbaron
Severity: normal → critical
Priority: P3 → P1
Target Milestone: --- → mozilla0.9.5
Status: NEW → ASSIGNED
I still need to go through the code that I removed a little more carefully,
since the code I added is an updated version of the old patch I had on bug
44340, and the code I am replacing may have changed more that I noticed since then.

I also need to do a good bit of testing...
Whiteboard: WONTFIX?
I filed bug 98218, which exists with or without my patch.  I'll attach an
much-improved patch shortly (although it still uses obsolete string code).
Oops, I just noticed the bad formatting in ParsePS, and fixed it in my tree.
what impact will this bug have to those thousands of websites out there having 
no doctype specified. What are the consequences in rendering and the used 
appearance of websites to customers?

no impact -- no DTD will be done in quirks mode just as now. This bug is only about
*unknown* DTDs, not missing DTDs.
David: I like your changes a lot. The only thing that I didn't like is inlining
DetermineHTMLParseMode(). What's the reason behind it?
The reason I made it |inline| was that it's only used once -- this may as well
all get compiled into one big function (it should be slightly more efficient
that way), but I'd rather not *think* about it as one big function.  Of course,
changing |inline| to |static| would probably be only a negligible slowdown
(assuming the compiler is even capable of inlining the function), and it doesn't
really matter to me.
In the past Metrius has used an FPI like this: "-//Metrius//DTD Metrius
Presentational//EN" on pages of their clients. I suggest including it on the
list of quirky doctypes. Otherwise, bug 22274 will occur on Motorola's site.

BTW, in DetermineHTMLParseMode() there's a call like this
aBuffer.InsertWithConversion(
        "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\n",
        0);
What's the purpose of that one? Is it used when Editor creates a new doc?
I updated my proposal at
  http://www.people.fas.harvard.edu/~dbaron/mozilla/doctypes
to reflect one bit of current practice -- that we *are* using strict
mode for HTML 4.01 transitional and frameset doctypes when a system
identifier is present.  (I also fixed some escaping errors in some of
the links.)

I also updated both the proposal and the code in my tree (a
one line change, removing the line noting that doctype -- not worth
posting a new patch) when I discovered that the old code treated
"-//IETF//DTD HTML i18n//EN" as a strict-mode public ID, and we haven't
had any problems caused by that.

Finally, I updated both the proposal and the code for the Metrius
doctype mentioned above (as eQuirks, not eQuirks3).

I verified that I was not changing the behavior on any of the tests
listed in that page (other than the one mentioned above that I changed)
where I was not expecting to change the behavior.  The only changes
were:

 * on the 2d, 3d, and 4th items in the strict mode list (system
   identifier only, neither system nor public identifier, and internal
   subset), which I think are safe changes.

 * on the public ID "-//SoftQuad Software//DTD HoTMetaL PRO
   6.0::19990601::extensions to HTML 4.0//EN" in the quirks list, which
   is bug 55916

So I think the patch is tested well enough that it's ready for checkin
early in a milestone.  I suspect we'll get a few reports of obscure
doctypes that my search-for-doctypes missed.  I plan to post to
n.p.m.layout and n.p.m.seamonkey, and also email some Netscape tech
evangelism folks so they know to be aware of the change.

So I think the patch is ready for review.  It's just the patch attached
above, with the one formatting change in ParsePS, and the one doctype
declaration mentioned above removed from the list of quirky doctypes,
and the Metrius one mentioned above added (as eQuirks).  I expect we'll
have to add a few more public IDs to the list in the coming weeks, but
that's why I want to check it in early in the milestone cycle.

(To respond to Henri Sivonen's comment:  I'm not sure what that
InsertWithConversion is there for, but it was there and I didn't want to
remove it for fear I'd break something.)

Comment on attachment 48191 [details] [diff] [review]
much improved patch

Change inline to static. With that r=harishd
Attachment #48191 - Flags: review+
harishd: Why do you think it is better for that function to be static rather than
inline? I'm trying to learn the "tricks" of the trade, and can't quite understand
this particular request. Thanks in advance for any explanation! :-)
Not to speak for Harish, but generally 'static' is used to prevent a method from
being exported from a module. It is useful, often necessary, if you want to make
sure that you don't end up with several different global functions with the same
name clashing at link time. 'inline' will likewise prevent the method from being
exported, so I think inline is fine here.
inline functions, especially large functions, can introduce code bloat which in
turn can cause negative performance. I would therefore perfer inlining smaller
functions. On the other hand the inline keyword does not force the complier to
inline a function ( I think ). It leaves the discretion to the compiler. I,
personally, don't prefer guessing compilers' actions :-)
But dbaron pointed out that the function is only called once, which means
inlining it is zero bloat (how many times can you duplicate something to produce
a single call to it?)

The only potential disadvantage is if somebody else doesn't realize the overhead
and makes another call to the same function. Perhaps that could be avoided by
arranging somehow to make sure the code isn't visible anywhere else, or by
having a sufficiently big comment everywhere it's used saying CHANGE INLINE TO
STATIC ON THE DECLARATION OF THIS IF YOU DUPLICATE THIS CODE OR USE IT ANYWHERE
ELSE.

Hopefully a good compiler would inline any function that's only called once and
isn't visible elsewhere anyway, right?
Comment on attachment 48191 [details] [diff] [review]
much improved patch

A question: Does ParsePS do the right thing if we don't see the end of a comment in the buffer passed in? It doesn't return kNotFound and my impression is that we wind up not skipping the comment.

If you are doing the right thing, sr=vidur.
Attachment #48191 - Flags: superreview+
Yes, what it does is not skip the comment, which will then lead to a parsing
error when we look for whatever should have followed it and find "--" instead. 
The idea of ParsePS is to consume as much as possible while matching the PS
production, but not to consume something that would be a partial match.  Then it
will fall back to quirks mode since it can't parse the whole DOCTYPE declaration
out of the initial buffer with which it is called -- that's the disadvantage of
doing the mode detection in an initial first pass rather than while consuming
the page (which I think would be better, and it wouldn't be too late).  Doing
the DOCTYPE parsing in the main pass requires more changes to the parser than I
want to make now.
Fix checked in 2001-09-08 11:37 PDT.
Status: ASSIGNED → RESOLVED
Closed: 24 years ago → 23 years ago
Resolution: --- → FIXED
QA Contact: bsharma → moied
*** Bug 91038 has been marked as a duplicate of this bug. ***
verified
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: