191668 - Tokenizer can fail on <!-

Status: VERIFIED FIXED

(Rhino Graveyard :: Core, defect)

Description

[Igor Bukanov]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20021003
Build Identifier: Rhino CVS from 2003-02-02

Rhino tokenizer violates single character lookahead restriction of
omj.LineBuffer when implementing recognition of HTML comments <!-- . In
particular, LineBuffer assumes that calls to LineBuffer.match will not unread
after successful match, but code to recognize <!-- does exactly that as <! and
<!- are valid sequence of JavaScript operators. 

The following attached test case calls eval on "if (0<!-0) ++j;" prefixed by
increasing sequence of spaces to make sure that at some point <!- occurs on
internal buffer boundary when LineBuffer.unread can not work.



Reproducible: Always

Steps to Reproduce:
Run the test by Rhino shell
Actual Results:  
Exception in thread "main" java.lang.RuntimeException: FAILED ASSERTION
        at org.mozilla.javascript.Context.codeBug(Context.java:2150)
        at org.mozilla.javascript.LineBuffer.unread(LineBuffer.java:116)
        at org.mozilla.javascript.TokenStream.getToken(TokenStream.java:1131)


Expected Results:  
The test should print OK

Comment 1

[Igor Bukanov]

Attachment: Test case

Comment 2

[Igor Bukanov]

As the bug presents in 1.5R3, its fix should wait until 1.5R4 is released. 

Comment 3

[Igor Bukanov]

Attachment: Fix: use special buffer for unreading of several chars and move the logic of recognizing line ends and format chars to TokenStream to follow SM more closely. LineBuffer is kept only for buferization.

Comment 4

[Phil Schwartau]

Testcase added to JS testsuite:

      mozilla/js/tests/js1_5/Regress/regress-191668.js

Currently passing in SpiderMonkey, but failing in Rhino as
Igor described above -

Comment 5

[Igor Bukanov]

Attachment: Patch update

This is a patch update that integrates LineBuffer into TokenStream and teaches
it to read tokens from String, not only java.io.Reader to avoid constructing
StringReader/TokenStream buffer when evaluating scripts from strings.

Comment 6

[Igor Bukanov]

I commited the patch

Comment 7

[Phil Schwartau]

Verified FIXED.

The above testcase now passes in the Rhino shell. I tested
optimization levels -1, 0, 1, and 9.

Comment 8

[Igor Bukanov]

Targeting as resolved against 1.5R5

Comment 9

[Igor Bukanov]

Fixing wrong milestone assignment