1472031 - Miscellaneous tokenizing improvements

Status: RESOLVED FIXED

(Core :: JavaScript Engine, enhancement)

Description

[Jeff Walden [:Waldo]]

Nothing strictly necessary for UTF-8 parsing, but things that will to one degree or another contribute to getting there.

Comment 1

[Jeff Walden [:Waldo]]

Attachment: Introduce a drainCharBufferIntoAtom function to clear charBuffer and create an atom from the chars that were in it

Technically the "drain" aspect of this is not necessary, but it seems preferable to drain immediately than to hold onto unbounded memory we're never going to use.  There are other places that could stand this "drain" treatment, but one thing at a time, as I run across them.

Comment 2

[Jeff Walden [:Waldo]]

Attachment: Rename atomizeChars to atomizeSourceChars to clarify it's only for atomizing directly from source text

Comment 3

[Jeff Walden [:Waldo]]

Attachment: Define TokenStreamAnyChars::updateFlagsForEOL inline in the header, for simplicity

Comment 4

[Jeff Walden [:Waldo]]

Attachment: Implement a SourceUnits::remaining() function exposing how many code units are left

Comment 5

[Jeff Walden [:Waldo]]

Attachment: Avoid copying potential directives in TokenStreamSpecific::getDirective

Small improvement, nice to not copy stuff (or impose on the compiler to figure out how not to copy, maybe, if we were really lucky before).

Comment 6

[Jeff Walden [:Waldo]]

Attachment: Add SourceUnits::matchHexDigits and remove SourceUnits::peekCodeUnits used for the same ends (but that also performed a needless copy)

Fewer lines of code *and* less copying.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b64aec5cf563
Introduce a drainCharBufferIntoAtom function to clear charBuffer and create an atom from the chars that were in it.  r=arai
https://hg.mozilla.org/integration/mozilla-inbound/rev/08cf6e4cccce
Rename atomizeChars to atomizeSourceChars to clarify it's only for atomizing directly from source text.  r=arai
https://hg.mozilla.org/integration/mozilla-inbound/rev/481871dcc488
Define TokenStreamAnyChars::updateFlagsForEOL inline in the header, for simplicity.  r=arai
https://hg.mozilla.org/integration/mozilla-inbound/rev/d857b06f51ca
Implement a SourceUnits::remaining() function exposing how many code units are left.  r=arai
https://hg.mozilla.org/integration/mozilla-inbound/rev/dad92f9be9b7
Avoid copying potential directives in TokenStreamSpecific::getDirective.  r=arai
https://hg.mozilla.org/integration/mozilla-inbound/rev/ba24991f7f4d
Add SourceUnits::matchHexDigits and remove SourceUnits::peekCodeUnits used for the same ends (but that also performed a needless copy).  r=arai

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/b64aec5cf563
https://hg.mozilla.org/mozilla-central/rev/08cf6e4cccce
https://hg.mozilla.org/mozilla-central/rev/481871dcc488
https://hg.mozilla.org/mozilla-central/rev/d857b06f51ca
https://hg.mozilla.org/mozilla-central/rev/dad92f9be9b7
https://hg.mozilla.org/mozilla-central/rev/ba24991f7f4d