485587 - Add a Tokenize method to the nsITokenizer interface

Status: RESOLVED WONTFIX

(Core :: DOM: HTML Parser, enhancement)

Description

[Ben Newman (:bnewman) (:benjamn)]

The token consumption loop should be moved into the tokenizer(s) rather than living in nsParser.cpp.  ConsumeToken is the wrong abstraction to expose, not least because it's called too frequently to be worth proxying across threads.

Comment 1

[Ben Newman (:bnewman) (:benjamn)]

Attachment: reorganizing nsParser::Tokenize to ease extraction of token consumption loop

Planning to extract the part delimited by ASCII scissors (---8<---).

Comment 2

[Ben Newman (:bnewman) (:benjamn)]

Attachment: implementation with nsTokenizer adapter class

Turns out the implementation for nsExpatDriver (implements nsITokenizer—I know, it's crazy) and nsHTMLTokenizer were identical, so I've moved the default Tokenize implementation into a common base class, nsTokenizer.

Comment 3

[Ben Newman (:bnewman) (:benjamn)]

Come to think of it, this pretty much obsoletes WillTokenize and DidTokenize.

Comment 4

[Ben Newman (:bnewman) (:benjamn)]

(In reply to comment #3)
> Come to think of it, this pretty much obsoletes WillTokenize and DidTokenize.

Or not.  Tokenize might be called repeatedly.

Comment 5

[Ben Newman (:bnewman) (:benjamn)]

Long since mooted by HTML5 parser work.