432185 - Implement the definer

Status: VERIFIED WONTFIX

(Tamarin Graveyard :: Self-hosting compiler (ESC), defect, P2)

Description

[Lars T Hansen]

Currently there is no definer phase to resolve namespace bindings and type names, ESC punts by making the AVM lookup types and by faking namespaces.  We need to implement a proper definer that performs the correct resolutions.  This probably involves implementing a sequence-of-files compilation model where the result of one file compilation forms the background for the next file.

Big job.

Comment 1

[Lars T Hansen]

Attachment: Intermediate patch - work in progress

This patch gets rid of string lexemes and switches the compiler to pass around token structures instead.  The token structures carry hash codes so the underlying strings don't need to be hashed more than once per token.  The token structures could also carry other data if necessary, but the hash code will pay for itself also when the token gets incorporated into names in the Ast (which will be stored in hashed environment ribs).  Anyhow the current change by itself speeds up total recompile time by about 10%.

Comment 2

[Lars T Hansen]

(In reply to comment #1)
> Created an attachment (id=324472) [details]
> Intermediate patch - work in progress

BTW that's a clean patch against rev 581.

Comment 3

[Lars T Hansen]

Attachment: JSON decoder, sketches

Attached is a JSON decoder that uses the ES4 lexer to do all the heavy lifting.  I'm attaching it because I expect it will be put to use in the definer.  The definer will need to know about definitions in "previous" files.  The way to do this is always to compile all the files together, but to generate digests of each file, containing information about top-level definitions in those files.  The digests are then saved in JSON format, and reused during the next compile if the source file has not changed.  The tar file contains some sketches for that.

Comment 4

[Lars T Hansen]

Attachment: Intermediate patch

This patch (relative to changeset 588) has the following:

  - strings replaced by interned token structures ("symbols") throughout, these carry a hash code to avoid rehashing strings all over the place
  - definer split out as a separate file, with a somewhat cleaner API and somewhat cleaner internals
  - some sketches for how the definer will work
  - preliminary optimizations, notably in Parse::hasName
  - sundry minor fixes, including Unix line endings

Though not a complete definer by any means, it seems worthwhile to push it now in order to forking the source too much.  Something like what's in this patch will eventually be needed.

To compile from source, comment out define.es.abc from esc.sh the first time around.

Comment 5

[Lars T Hansen]

Comment on attachment 325738 [details] [diff] [review]
Intermediate patch

Pushed the intermediate patch.