1347861 - Consider storing hints during syntax parse

Status: NEW

(Core :: JavaScript Engine, enhancement, P5)

Description

[David Teller [:Yoric] - still alive but not very active]

While the syntax parser doesn't build an AST, it needs to resolve all the nasty bits of the JS syntax, including:
- finding out whether a `/` is an operator, the start of a comment, a regexp (tokenizer);
- finding out where a string/string template ends (tokenizer);
- finding out where a comment ends (tokenizer);
- extrapolating missing semicolons (parser);
- finding out whether a `{` opens a block or an object literal (parser);
- finding out where a nested function ends (parser).

The notion of "nasty bits" is very handwavy at this stage, but let's say it's related to lookahead.

When we later perform full parsing, we need to re-resolve these same nasty bits. No measure has been done at this stage, but I suspect that at least some of these nasty bits can be expensive. Now, at the cost of some memory, we could store hints in constant time (there are several ways to do this), in the tokenizer and/or in the parser, then reuse these hints (also in basically constant time) when we full parse.

Comment 1

[Jason Orendorff [:jorendorff]]

Not that anyone asked, but...

> - finding out whether a `/` is an operator, the start of a comment, a regexp (tokenizer);
> - extrapolating missing semicolons (parser);
> - finding out whether a `{` opens a block or an object literal (parser);

These are already pretty fast. It's hard to imagine caching information will make them faster, because the decision is already so quick.

Automatic semicolon insertion, for instance, happens when you call matchOrInsertSemicolon and the next token isn't a semicolon or isn't on the same line. I hate ASI, but it's not exactly hard to do.

> - finding out where a string/string template ends (tokenizer);
> - finding out where a comment ends (tokenizer);
> - finding out where a nested function ends (parser).

In these cases, teleporting to the end when reparsing might be a nice speedup.