alternative parser-based syntax engine for JS, XML, HTML, CSS, PHP and mixed

RESOLVED FIXED in Future

Status

Skywriter
Editor
P3
enhancement
RESOLVED FIXED
9 years ago
8 years ago

People

(Reporter: Roberto Saccon, Unassigned)

Tracking

unspecified
Future
x86
Mac OS X

Details

Attachments

(1 attachment, 6 obsolete attachments)

(Reporter)

Description

9 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
Build Identifier: changeset:   825:5835e048f0a1

The engine is a port from CodeMirror (http://marijn.haverbeke.nl/codemirror/ - BSD-licensed) and is based on a streaming, interruptible / resumable parser.

The engine can be commandline-enabled / disabled via: set syntax2 on / off

The enigne uses Malte Ubis worker facade approach to run in a background thread if webworker or gears are available otherwise it still runs  asynchronous (via settimer)

The parsers for the different languages have currently different state of maturity:

JS: in very good shape (but not as exact as narcissus)
XML, CSS: rather simple
HTML: also detects inner languages such as CSS and JS
PHP: does also realtime syntax-error highlighting

The paint routine in the editor has been slightly modified so that the engines can be switched.

This is a very initial release, and performance is not good yet because of the following reason:
the editor repaints for every cursor blink, but the  engine does NOT cache the syntax results, so it does too much unnecessary work. 

Next steps (depending on feedback, of course):
------------------------------------
- port to th textarea and fix the performance issues 
- consolidate colorschemes with the default syntax engine
- more consolidation: run default and this engine in worker facade ????
- more languages: Django (halfway done), Python (maybe by the CodeMirror author), Ruby (who ???)
- better name than "syntax2"
- extend with fancy things like syntax-aware auto-indentation, realtime preview template rendering, ... 


Reproducible: Always
(Reporter)

Comment 1

9 years ago
Created attachment 368657 [details] [diff] [review]
alternative syntax engine
(Reporter)

Comment 2

9 years ago
Created attachment 368694 [details] [diff] [review]
same patch, but straight from patchfolder

original attachment was created with qdiff -U 8, against tip. This is the same patch, but straight from the patch folder, just in case applying gives trouble (see recent conversation with danzat: on irc)

Comment 3

9 years ago
Roberto,

This was fun to play with!

A few comments:

- On Safari 4 the colors were bleached out and everything looked wacky.
- If you move about quickly, you definitely feel the painting is wacky too
- Would love to see this as: set syntaxengine codemirror (instead of syntax2) as you mention. Then we can have: set syntaxengine codemirror|simple|another
- Would love to use your work as a basis for working out how we can plugin syntax engines. You do this in deleting and applying "this.syntaxModel". It would be nice to make this a formal API where you can just setSyntaxEngine()

Cheers,

Dion
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(Reporter)

Comment 4

9 years ago
Dion, 

I am taking  out "syntax2" and introducing the setting "syntaxengine" as you suggested, but now, because the other setting related to syntax is called "syntax" (and can have value: "auto" or the actual language), I think we should also rename the setting  "syntax" to "language". What do you think ?

about the safari coloring issue:
Unfortunately I am expecting that this is a serious issue, we are loosing canvas context because of the asynchronous nature of our callback for painting text. It might just be coincidence that this problem does not occur on FF. I have ported the syntaxengine already to new thx texrarea (really fast there), were the  problem seems to be even worse, but I am working with Dany on a solution.

Comment 5

9 years ago
Roberto,

- I like changing syntax to language
- Losing ctx is interesting. Maybe fixcanvas can do some checking or something

Cheers,

Dion
(Reporter)

Comment 6

8 years ago
Dion,

I did some heavy surgery and synced the patch with tip. Here the changes in detail:

- changed "syntax" to "language" at all places I found
- got rid of syntax2, now it is on commnandline: set syntaxengine ( simple | codemirror )
- refacotored bespin.syntax directory structure
- enabled caching and implemented cache invalidation, now it is muuuuch faster !!!
- added a syntaxengine resolver, for simple switch between engines

Issues:
- the colors in safari still come out wrong, started to investigate with Dany about the possibility that we are loosing canvas context.
- color schemas

Some whitespace cleanup made it unintentionally  into that patch, and don't know how to take that out again
(Reporter)

Comment 7

8 years ago
Created attachment 369479 [details] [diff] [review]
updated patch

Comment 8

8 years ago
Roberto,

This looks awesome! I am getting two small issues with the diff. A settings.js one which is trivial, and a large one on editor.js.

Would you mind doing a diff on what you have to rev: 217437c352da and check out editor.js to make sure that it is happy? A recent change in editor is a "highlightline" setting.

If you can give me one AOK, then I will get it patched in ASAP.
(Reporter)

Comment 9

8 years ago
Dion,

I updated the patch, diff is now against changeset:   863:217437c352da
also fixed some minor bugs

recently detected issue with FF3.1b / FB 1.4b / Mac OS X (and this is probably NOT caused by the syntax engine, but prevents the syntax engine from delivering useful output):

the paint routine in editor.js gets into a state where the following variables have the following values:

this.editor.model.getRowCount() = 1
this.firstVisibleRow = 0 
lastLineToRender = 0

this obviously does not any real document, which should have been loaded

I managed to reproduce this behavior by simple refreshing the browser, and killing the browser brings things into normal state again.


So what's next:
- More languages: I plan to add django and write some mini tutorial so others can step in
- Consolidate simple / codemirror engines (for example CSS: there is good stuff in simple I could reuse in codemirror)
- The real cool stuff: auto indention of selected blocks, reflecting the lexical structure 
- highlighting syntax errors (e.g. different background color)
- and of course track down that color shift issue in safari
(Reporter)

Comment 10

8 years ago
Created attachment 369604 [details] [diff] [review]
updated patch against changeset: 863:217437c352da

Comment 11

8 years ago
Roberto,

Fantastic. I have tweaked this a little to deal with settings not being set yet etc.... but it committed in changeset d2676ad166e5

Love the "what's next" items too. Thanks!
Priority: -- → P3
Whiteboard: editor
Target Milestone: -- → Future
Attachment #368657 - Attachment is obsolete: true
Attachment #368657 - Attachment is patch: true
Attachment #368657 - Attachment mime type: application/octet-stream → text/plain
Attachment #368694 - Attachment is obsolete: true
Attachment #368694 - Attachment is patch: true
Attachment #368694 - Attachment mime type: application/octet-stream → text/plain
Attachment #369479 - Attachment is obsolete: true
Attachment #369479 - Attachment is patch: true
Attachment #369479 - Attachment mime type: application/octet-stream → text/plain
Comment on attachment 369604 [details] [diff] [review]
updated patch against changeset: 863:217437c352da

Apologies for the bug spam. I've marked all of these patches as such, as well as marking the old ones as obsolete.

Robert, please remember to do this in the future. It helps keep things streamlined.
Attachment #369604 - Attachment is patch: true
Attachment #369604 - Attachment mime type: application/octet-stream → text/plain
Also, this changeset appears to have created bug 485492. Things are going really wonky—not to mention that syntax highlighting doesn't work.

Now, this is may or may not be related to Dion's removal of the old syntax engine. I'm not sure.
(Reporter)

Comment 14

8 years ago
bugfix patch against d2676ad166e5

- serveral minor corrections, mostly adding settings-checks
- updated singleeditor.html to work again (with simple engine)
- the nasty FF bug turned out to be a FF / FB issued non-related to bespin
(Reporter)

Comment 15

8 years ago
Created attachment 369861 [details] [diff] [review]
bugfix against d2676ad166e5
Attachment #369861 - Attachment is patch: true
Attachment #369861 - Attachment mime type: application/octet-stream → text/plain
(Reporter)

Comment 16

8 years ago
updated bugfix  patch against d2676ad166e5

- set command is now working again
- codemirror syntaxengine is now dynamically loaded via dojo.require
(Reporter)

Comment 17

8 years ago
Created attachment 369862 [details] [diff] [review]
updated patch against d2676ad166e5
Attachment #369604 - Attachment is obsolete: true
Attachment #369861 - Attachment is obsolete: true
(Reporter)

Comment 18

8 years ago
Created attachment 369863 [details] [diff] [review]
updated patch against d2676ad166e5

bugfix for cache-invalidating multiline chunk-inserts
Attachment #369862 - Attachment is obsolete: true

Comment 19

8 years ago
Ok, changeset ed73681b0eb4 has this with a few more tweaks (changing all syntax -> language, guarding settings a touch more etc).

Let's open a new bug for the Safari 4 codemirror issue where the colors aren't applied correctly.
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
This is a mass migration from Mozilla Labs :: Bespin to Bespin :: Editor.
Component: Bespin → Editor
Product: Mozilla Labs → Bespin
QA Contact: bespin → editor
Whiteboard: editor
You need to log in before you can comment on or make changes to this bug.