1194022 - Assertion failure: next.type != TOK_DIV && next.type != TOK_REGEXP (next token requires contextual specifier to be parsed unambiguously), at frontend/TokenStream.h

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

function f()
    yield
    /x/

asserts js debug shell on m-c changeset 7649ffe28b67 with --fuzzing-safe --no-threads --no-ion at Assertion failure: next.type != TOK_DIV && next.type != TOK_REGEXP (next token requires contextual specifier to be parsed unambiguously), at frontend/TokenStream.h.

Configure options:

CC="clang -Qunused-arguments" CXX="clang++ -Qunused-arguments" AR=ar AUTOCONF=/usr/local/Cellar/autoconf213/2.13/bin/autoconf213 sh /Users/skywalker/trees/mozilla-central/js/src/configure --target=x86_64-apple-darwin12.5.0 --enable-debug --enable-nspr-build --enable-more-deterministic --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests

python -u ~/funfuzz/js/compileShell.py -b "--enable-debug --enable-more-deterministic --enable-nspr-build" -r 7649ffe28b67

=== Treeherder Build Bisection Results by autoBisect ===

The "good" changeset has the timestamp "20150812050310" and the hash "2c3c8a927e3fcc3c7d63c2ecf5ff9a189d1cbd96".
The "bad" changeset has the timestamp "20150812054307" and the hash "a8493abd3c62b77e4741da181239af5549d84f80".

Likely regression window: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=2c3c8a927e3fcc3c7d63c2ecf5ff9a189d1cbd96&tochange=a8493abd3c62b77e4741da181239af5549d84f80

Setting needinfo? from :arai as per the regression window.

In the meantime, trying to get a better bisection.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: stack

(lldb) bt 5
* thread #1: tid = 0x1f21aa, 0x000000010004caa4 js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::TokenStream::addModifierException(this=<unavailable>, modifierException=<unavailable>) + 372 at TokenStream.h:437, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x000000010004caa4 js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::TokenStream::addModifierException(this=<unavailable>, modifierException=<unavailable>) + 372 at TokenStream.h:437
    frame #1: 0x000000010004ea7a js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::Parser<js::frontend::FullParseHandler>::yieldExpression(this=0x00007fff5fbfe658, inHandling=<unavailable>) + 554 at Parser.cpp:5771
    frame #2: 0x0000000100049851 js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::Parser<js::frontend::FullParseHandler>::assignExpr(this=<unavailable>, inHandling=InAllowed, yieldHandling=<unavailable>, invoked=<unavailable>) + 705 at Parser.cpp:6699
    frame #3: 0x0000000100048e46 js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::Parser<js::frontend::FullParseHandler>::functionBody(this=0x00007fff5fbfe658, inHandling=InAllowed, yieldHandling=YieldIsName, kind=Statement, type=ExpressionBody) + 150 at Parser.cpp:1046
    frame #4: 0x000000010004a538 js-dbg-64-dm-nsprBuild-darwin-7649ffe28b67`js::frontend::Parser<js::frontend::FullParseHandler>::functionArgsAndBodyGeneric(this=0x00007fff5fbfe658, inHandling=InAllowed, yieldHandling=YieldIsName, pn=0x00000001028ae020, fun=<unavailable>, kind=Statement) + 472 at Parser.cpp:2694
(lldb)

Comment 2

[Tooru Fujisawa [:arai]]

thank you for finding it out!
it should be bug 1089045.
I'll try fixing it shortly

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

autoBisect shows this is probably related to the following changeset:

The first bad revision is:
changeset:   https://hg.mozilla.org/mozilla-central/rev/05f838caf076
user:        Tooru Fujisawa
date:        Fri Aug 07 04:11:59 2015 +0900
summary:     Bug 1089045 - Part 1: Supply consistent modifiers to TokenStream. r=Waldo

Yes, you're right! :)

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

This is triggering often due to its simplicity, so setting as [fuzzblocker].

Comment 5

[Tooru Fujisawa [:arai]]

this is the most exceptional case with modifier.

the TOK_REGEXP token is gotten with Operand by yieldExpression at first, and with None by MatchOrInsertSemicolon, and finally with Operand by unaryExpr in next statement.  We know that there ASI will happen and the token will be finally gotten with Operand, while getting the token in yieldExpression, as it gets TOK_EOL.

Simply we should add another type of exception for EOL after yield, which allows TOK_REGEXP but TOK_DIV.
I'll post patch shortly

Comment 6

[Tooru Fujisawa [:arai]]

try is running
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=4767135bbbe4

Comment 7

[Tooru Fujisawa [:arai]]

again:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=18e1ee96f1d7

Comment 8

[Tooru Fujisawa [:arai]]

Attachment: Add another exception for a token after yield expression without operand and followed by EOL.

Added NoneIsOperandYieldEOL exception, which is specified when yield expression has no operand and is followed by EOL or EOF.  Then, if we handle that with dedicated exception, we don't have to use bitset, but just the ModifierException enum itself, as there's no mo more mixed state.

Now, addModifierException can be called for a token with no exception, or a token after yield without operand.  In latter case, modifierException should be OperandIsNone, which is for ASI after the yield  expression.  in that case, token's exception is not updated, since the token is already gotten with Operand, and OperandIsNone has no meaning.

This doesn't happen with break or continue, because MatchOrInsertSemicolon is called with Operand for them.

Then, I found that comments for NoneIsOperand and OperandIsNone were swapped in last commit, fixed in this patch, sorry.

Comment 9

[Jeff Walden [:Waldo]]

Comment on attachment 8647314 [details] [diff] [review]
Add another exception for a token after yield expression without operand and followed by EOL.

Review of attachment 8647314 [details] [diff] [review]:
-----------------------------------------------------------------

I don't see anything wrong here, but my mind is fuzzing over the more I look at this.  If this stuff weren't finding bugs I would rip it out in a heartbeat as basically inscrutable.  :-(  Alas.

::: js/src/frontend/TokenStream.h
@@ +458,5 @@
> +                           "next token requires contextual specifier to be parsed unambiguously");
> +
> +            // Do not update modifierException.
> +            return;
> +        }

Blank line after }.

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a466d341647f

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a466d341647f