484527 - alternative parser-based syntax engine for JS, XML, HTML, CSS, PHP and mixed

Status: RESOLVED FIXED

(Skywriter Graveyard :: Editor, enhancement, P3)

Description

[Roberto Saccon]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
Build Identifier: changeset:   825:5835e048f0a1

The engine is a port from CodeMirror (http://marijn.haverbeke.nl/codemirror/ - BSD-licensed) and is based on a streaming, interruptible / resumable parser.

The engine can be commandline-enabled / disabled via: set syntax2 on / off

The enigne uses Malte Ubis worker facade approach to run in a background thread if webworker or gears are available otherwise it still runs  asynchronous (via settimer)

The parsers for the different languages have currently different state of maturity:

JS: in very good shape (but not as exact as narcissus)
XML, CSS: rather simple
HTML: also detects inner languages such as CSS and JS
PHP: does also realtime syntax-error highlighting

The paint routine in the editor has been slightly modified so that the engines can be switched.

This is a very initial release, and performance is not good yet because of the following reason:
the editor repaints for every cursor blink, but the  engine does NOT cache the syntax results, so it does too much unnecessary work. 

Next steps (depending on feedback, of course):
------------------------------------
- port to th textarea and fix the performance issues 
- consolidate colorschemes with the default syntax engine
- more consolidation: run default and this engine in worker facade ????
- more languages: Django (halfway done), Python (maybe by the CodeMirror author), Ruby (who ???)
- better name than "syntax2"
- extend with fancy things like syntax-aware auto-indentation, realtime preview template rendering, ... 


Reproducible: Always

Comment 1

[Roberto Saccon]

Attachment: alternative syntax engine

Comment 2

[Roberto Saccon]

Attachment: same patch, but straight from patchfolder

original attachment was created with qdiff -U 8, against tip. This is the same patch, but straight from the patch folder, just in case applying gives trouble (see recent conversation with danzat: on irc)

Comment 3

[Dion Almaer]

Roberto,

This was fun to play with!

A few comments:

- On Safari 4 the colors were bleached out and everything looked wacky.
- If you move about quickly, you definitely feel the painting is wacky too
- Would love to see this as: set syntaxengine codemirror (instead of syntax2) as you mention. Then we can have: set syntaxengine codemirror|simple|another
- Would love to use your work as a basis for working out how we can plugin syntax engines. You do this in deleting and applying "this.syntaxModel". It would be nice to make this a formal API where you can just setSyntaxEngine()

Cheers,

Dion

Comment 4

[Roberto Saccon]

Dion, 

I am taking  out "syntax2" and introducing the setting "syntaxengine" as you suggested, but now, because the other setting related to syntax is called "syntax" (and can have value: "auto" or the actual language), I think we should also rename the setting  "syntax" to "language". What do you think ?

about the safari coloring issue:
Unfortunately I am expecting that this is a serious issue, we are loosing canvas context because of the asynchronous nature of our callback for painting text. It might just be coincidence that this problem does not occur on FF. I have ported the syntaxengine already to new thx texrarea (really fast there), were the  problem seems to be even worse, but I am working with Dany on a solution.

Comment 5

[Dion Almaer]

Roberto,

- I like changing syntax to language
- Losing ctx is interesting. Maybe fixcanvas can do some checking or something

Cheers,

Dion

Comment 6

[Roberto Saccon]

Dion,

I did some heavy surgery and synced the patch with tip. Here the changes in detail:

- changed "syntax" to "language" at all places I found
- got rid of syntax2, now it is on commnandline: set syntaxengine ( simple | codemirror )
- refacotored bespin.syntax directory structure
- enabled caching and implemented cache invalidation, now it is muuuuch faster !!!
- added a syntaxengine resolver, for simple switch between engines

Issues:
- the colors in safari still come out wrong, started to investigate with Dany about the possibility that we are loosing canvas context.
- color schemas

Some whitespace cleanup made it unintentionally  into that patch, and don't know how to take that out again

Comment 7

[Roberto Saccon]

Attachment: updated patch

Comment 8

[Dion Almaer]

Roberto,

This looks awesome! I am getting two small issues with the diff. A settings.js one which is trivial, and a large one on editor.js.

Would you mind doing a diff on what you have to rev: 217437c352da and check out editor.js to make sure that it is happy? A recent change in editor is a "highlightline" setting.

If you can give me one AOK, then I will get it patched in ASAP.

Comment 9

[Roberto Saccon]

Dion,

I updated the patch, diff is now against changeset:   863:217437c352da
also fixed some minor bugs

recently detected issue with FF3.1b / FB 1.4b / Mac OS X (and this is probably NOT caused by the syntax engine, but prevents the syntax engine from delivering useful output):

the paint routine in editor.js gets into a state where the following variables have the following values:

this.editor.model.getRowCount() = 1
this.firstVisibleRow = 0 
lastLineToRender = 0

this obviously does not any real document, which should have been loaded

I managed to reproduce this behavior by simple refreshing the browser, and killing the browser brings things into normal state again.


So what's next:
- More languages: I plan to add django and write some mini tutorial so others can step in
- Consolidate simple / codemirror engines (for example CSS: there is good stuff in simple I could reuse in codemirror)
- The real cool stuff: auto indention of selected blocks, reflecting the lexical structure 
- highlighting syntax errors (e.g. different background color)
- and of course track down that color shift issue in safari

Comment 10

[Roberto Saccon]

Attachment: updated patch against changeset: 863:217437c352da

Comment 11

[Dion Almaer]

Roberto,

Fantastic. I have tweaked this a little to deal with settings not being set yet etc.... but it committed in changeset d2676ad166e5

Love the "what's next" items too. Thanks!

Comment 12

[Gordon P. Hemsley [:GPHemsley]]

Comment on attachment 369604 [details] [diff] [review]
updated patch against changeset: 863:217437c352da

Apologies for the bug spam. I've marked all of these patches as such, as well as marking the old ones as obsolete.

Robert, please remember to do this in the future. It helps keep things streamlined.

Comment 13

[Gordon P. Hemsley [:GPHemsley]]

Also, this changeset appears to have created bug 485492. Things are going really wonky—not to mention that syntax highlighting doesn't work.

Now, this is may or may not be related to Dion's removal of the old syntax engine. I'm not sure.

Comment 14

[Roberto Saccon]

bugfix patch against d2676ad166e5

- serveral minor corrections, mostly adding settings-checks
- updated singleeditor.html to work again (with simple engine)
- the nasty FF bug turned out to be a FF / FB issued non-related to bespin

Comment 15

[Roberto Saccon]

Attachment: bugfix against d2676ad166e5

Comment 16

[Roberto Saccon]

updated bugfix  patch against d2676ad166e5

- set command is now working again
- codemirror syntaxengine is now dynamically loaded via dojo.require

Comment 17

[Roberto Saccon]

Attachment: updated patch against d2676ad166e5

Comment 18

[Roberto Saccon]

Attachment: updated patch against d2676ad166e5

bugfix for cache-invalidating multiline chunk-inserts

Comment 19

[Dion Almaer]

Ok, changeset ed73681b0eb4 has this with a few more tweaks (changing all syntax -> language, guarding settings a touch more etc).

Let's open a new bug for the Safari 4 codemirror issue where the colors aren't applied correctly.

Comment 20

[Gordon P. Hemsley [:GPHemsley]]

This is a mass migration from Mozilla Labs :: Bespin to Bespin :: Editor.