1292635 - [e10s] Can't complete the test262 suite

Status: NEW

(Core :: JavaScript Engine, defect, P3)

Description

[Alexandre Folle de Menezes]

I am trying to run the test262 suite hosted here:
http://v8.github.io/test262/website/default.html

I am using Firefox Developer Edition (v49.0a2), which cannot complete the test. halfway through the test it stops painting some areas of the tab running the test (they become black), and later the browser stops responding completely.

Comment 1

[Alexandre Folle de Menezes]

To see the painting issue, minimize the window running the test and let it run for a while.  When the window is restored, there are a lot of black areas.  Moving the cursor around the page will make some elements to appear, but not all.

Comment 2

[Alexandre Folle de Menezes]

I have now tried this with v50.0a2, and still have the same issue.  Actually, instead of hanging the browser it now says "A web page is slowing down your browser.  What you would like to do?".  Maybe this happens with 49.0a2 too, I left it running overnight and never saw it showing this before it stopped responding.

Comment 3

[Alexandre Folle de Menezes]

And now I am running with the current stable version (v48.0) and everything seems fine.
What can be going on:
- Some issue with electrolisys, since it is off in the stable version but on in the Dev Edition;
- Some issue with the 64bit version (I am running 32bit for stable and 64bit for Dev Edition);
- Some other issue introduced in v49;

Can I do something else to help diagnosing this?

Comment 4

[Alexandre Folle de Menezes]

I have now disabled Electrolysis on v50.0, and it is running fine.  So I am assuming this is an Electrolysis issue. Marking it as such.

Comment 5

[Tom S [:evilpie]]

There are definitely tests that will just hang in the shell, for example built-ins/Array/prototype/slice/S15.4.4.10_A3_T1

Comment 6

[Guilherme Lima]

Nightly completed locked up on the "result-get-matched-err" test. It made the content process use 3GB (maximum on my Win7 machine) and use 50% of the CPU total. Had to kill the process.

Comment 7

[Alexandre Folle de Menezes]

This is still happening with 63.0a1.  I believe this is a major stability issue, in a high visibility test, which may hurt Firefox in a future review or browser comparison.

Comment 8

[André Bargull [:anba]]

I think at this point the issue should be closed as WONT FIX, because the test262 version at <http://v8.github.io/test262/website/default.html> is too old to work with current browsers (it's test262 from 2016-03-23). Both, Firefox and Chrome, hang at 10% for me, which is a good indication at least one of the tests expects some behaviour which is no longer compliant with the current ECMAScript spec.

Comment 9

[Alexandre Folle de Menezes]

I ran the tests with Chrome and Firefox, and both hang at the same place as you said:

Firefox 61: Tests To run: 16436 | Total tests ran: 1679 | Pass: 1617 | Fail: 62 | Failed to load: 0
Firefox 63: Tests To run: 16436 | Total tests ran: 1679 | Pass: 1617 | Fail: 62 | Failed to load: 0
Chrome 68:  Tests To run: 16436 | Total tests ran: 1679 | Pass: 1617 | Fail: 25 | Failed to load: 0

I even tried to turn Electrolisys off with <browser.tabs.remote.autostart=false>, which used to fix the issue on Firefox 50 (see comment 4), but I get the same exact result, regardless of its value.

But I disagree with you, I don't think a browser hang is a reasonable result, even if the test is buggy.

I also ran the test with IE and Edge, and both are able to finish it:

IE 11:   Tests To run: 16436 | Total tests ran: 16436 | Pass: 12465 | Fail: 3971 | Failed to load: 0 
Edge 16: Tests To run: 16436 | Total tests ran: 16436 | Pass: 15665 | Fail: 771  | Failed to load: 0

Comment 10

[André Bargull [:anba]]

I haven't investigated which test (or tests) let the browser hang, or more correctly just take a long time to complete, but there have been tests in test262 which tested the wrap around behaviour of, for example ToUint32, in some spec algorithms. Later spec revisions then changed ToUint32 to ToLength, so the input value is no longer wrapped around, but instead clapped at 2^53-1. And if test262 tests now basically had `ToUint32(4294967296)` and expected the value is wrapped to 0, but newer specs now use `ToUint32(4294967296)`, it's inevitable that tests need longer for completion in implementations which implement the ToLength call. 

ToUin32: https://tc39.github.io/ecma262/#sec-touint32
ToLength: https://tc39.github.io/ecma262/#sec-tolength

Comment 11

[Timea Cernea [:tbabos][inactive]]

Hi Alexandre,
Are the tests still failing on the latest Firefox version?

Comment 12

[Alexandre Folle de Menezes]

Yes, I tested today's nightly and it hangs in the same spot as previous versions:

Tests To run: 16436 | Total tests ran: 1679 | Pass: 1607 | Fail: 72 | Failed to load: 0

But now I get a warning that the script is making the page to stop responding and allows me to stop it. If the test is indeed buggy, I believe this is the appropriate behavior.

Comment 13

[Timea Cernea [:tbabos][inactive]]

Thanks, Alexandre for checking it out. I'll update the severity to go with current severity rankings.