Closed Bug 1512989 Opened Last year Closed Last year

jstests in the browser are broken, failures are not reported

Categories

(Core :: JavaScript Engine, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla66
Tracking Status
firefox66 --- fixed

People

(Reporter: jandem, Assigned: anba)

References

Details

Attachments

(2 files, 2 obsolete files)

Bug 1510819 sort of exposed this because J1 on OS X starts hitting the 1h timeout (intermittently). Looking at the logs there are many failures that should result in orange but don't:

[task 2018-12-10T14:23:43.892Z] 14:23:43     INFO - REFTEST TEST-START | file:///builds/worker/workspace/build/tests/jsreftest/tests/jsreftest.html?test=non262/regress/regress-1476383-calloc-exc.js
[task 2018-12-10T14:23:43.893Z] 14:23:43     INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/jsreftest/tests/jsreftest.html?test=non262/regress/regress-1476383-calloc-exc.js | 1959 / 11236 (17%)
[task 2018-12-10T14:23:43.944Z] 14:23:43     INFO - TEST-INFO |  FAILED!  ReferenceError: quit is not defined
[task 2018-12-10T14:23:43.945Z] 14:23:43     INFO - JavaScript error: file:///builds/worker/workspace/build/tests/jsreftest/tests/non262/regress/regress-1476383-calloc-exc.js, line 3: ReferenceError: quit is not defined
[task 2018-12-10T14:23:43.970Z] 14:23:43     INFO - REFTEST TEST-PASS | file:///builds/worker/workspace/build/tests/jsreftest/tests/jsreftest.html?test=non262/regress/regress-1476383-calloc-exc.js | (LOAD ONLY)
[task 2018-12-10T14:23:43.970Z] 14:23:43     INFO - REFTEST TEST-END | file:///builds/worker/workspace/build/tests/jsreftest/tests/jsreftest.html?test=non262/regress/regress-1476383-calloc-exc.js
It may be interesting to bisect this to see when it started.
My first guess is that this is intentional -- somewhere there's a log-parsing script with a long list of regular expressions, and `/quit is not defined/` is on that list.

It should be fixed, though.
(In reply to Jason Orendorff [:jorendorff] from comment #2)
> My first guess is that this is intentional -- somewhere there's a
> log-parsing script with a long list of regular expressions, and `/quit is
> not defined/` is on that list.

I don't think this is true. The failure in bug 1510819 is a similar issue that's being swept under the rug - that job just turns orange because there's a timeout.
So both here and in bug 1510819, we get

INFO - REFTEST TEST-PASS | [path] | (LOAD ONLY)

and I found a comment that claims this means "test without a reference (just test that it does not assert, crash, hang, or leak)". The failing tests throw an exception, but don't seem to be doing any of the things that are expected to make it fail, according to the comment.

Possibly these tests should be "script" rather than "load" tests; I managed to trace the determination back as far as https://searchfox.org/mozilla-central/rev/fd62b95c187a40b328d9e7fd9d848833a6942b57/layout/tools/reftest/reftest.jsm#739
(In reply to :Ms2ger (he/him; ⌚ UTC+1/+2) from comment #4)
> Possibly these tests should be "script" rather than "load" tests; 

^ This!

Starting with bug 1392391, test entries for TYPE_SCRIPT are marked as TYPE_LOAD. [1]

[1] https://searchfox.org/mozilla-central/rev/adcc169dcf58c2e45ba65c4ed5661d666fc3ac74/layout/tools/reftest/manifest.jsm#299
Changes ReadManifest to propagate the correct test type.
Assignee: nobody → andrebargull
Status: NEW → ASSIGNED
Attachment #9030511 - Flags: review?(ahal)
Fix various tests to work correctly in browser environments. 

Changes:

js/src/tests/Makefile.in
- The "whatwg" directory wasn't packed for the test environment.

js/src/tests/jstests.list
js/src/tests/non262/lexical-environment/block-scoped-functions-annex-b-with.js
- ECMA-262 defines that global var-bindings are non-configurable, but in the browser they're apparently configurable.

js/src/tests/non262/Date/browser.js
js/src/tests/non262/String/ropes.js
- Export various testing functions to the global scope.

js/src/tests/non262/ReadableStream/subclassing.js
js/src/tests/non262/generators/yield-star-throw-htmldda.js
js/src/tests/non262/regress/regress-1456512-greyreadbarrier.js
js/src/tests/non262/regress/regress-1456512.js
js/src/tests/non262/regress/regress-1463421.js
js/src/tests/non262/regress/regress-1476383-calloc-exc.js
- Shell-only functions are called in these tests.

js/src/tests/non262/String/replace-rope-empty.js
- The "isRope" testing function doesn't seem to work correctly in a browser environment.
- I guess the function is in a different compartment, so when the argument string is passed to this different compartment, it gets flattened...?

js/src/tests/test262-update.py
js/src/tests/test262/*
- dynamic import is only available in the shell
- regenerated the tests to contain the correct skip annotation

js/src/tests/ecma_6/Date/parse-from-tostring-methods.js
- Test file wasn't moved into the correct directory when the patch was rebased.
Attachment #9030517 - Flags: review?(jorendorff)
anba++
Comment on attachment 9030511 [details] [diff] [review]
bug1512989-part1-pass-test-type.patch

Review of attachment 9030511 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry about that :(, we need better test coverage for the harness. I'm going to see if there's an easy way to make sure these tests fail if they have the wrong type, but feel free to go ahead and land (assuming there haven't been regressions in the meantime).
Attachment #9030511 - Flags: review?(ahal) → review+
Comment on attachment 9030517 [details] [diff] [review]
bug1512989-part2-update-tests.patch

Review of attachment 9030517 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you so much for doing this.

::: js/src/tests/non262/generators/yield-star-throw-htmldda.js
@@ +1,1 @@
> +// |reftest| skip-if(!xulRuntime.shell) -- needs createIsHTMLDDA

This horrible functionality is for the browser's benefit, so let's make the test run in the browser:

if (this.createIsHTMLDDA === undefined) {
  this.createIsHTMLDDA = () => document.all;
}

::: js/src/tests/non262/regress/regress-1476383-calloc-exc.js
@@ +3,4 @@
>  if (!this.oomTest) {
>      this.reportCompare && reportCompare(true, true);
>      quit(0);
>  }

Please consider `skip-if(!this.oomTest)` and deleting lines 3-6.
Attachment #9030517 - Flags: review?(jorendorff) → review+
See Also: → 1513577
Updated part 2 per review comments, carrying r+.
Attachment #9030517 - Attachment is obsolete: true
Attachment #9030846 - Flags: review+
(In reply to Jason Orendorff [:jorendorff] from comment #10)
> ::: js/src/tests/non262/generators/yield-star-throw-htmldda.js
> @@ +1,1 @@
> > +// |reftest| skip-if(!xulRuntime.shell) -- needs createIsHTMLDDA
> 
> This horrible functionality is for the browser's benefit, so let's make the
> test run in the browser:
> 
> if (this.createIsHTMLDDA === undefined) {
>   this.createIsHTMLDDA = () => document.all;
> }

I saw that there are more tests using dynamic checks to see if |createIsHTMLDDA| is present, so I've instead added "createIsHTMLDDA" to js/src/tests/browser.js to enable us to run these on the browser, too.
Blocks: 1513577
See Also: 1513577
Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/59a6d8169f80
Part 1: Pass correct test type. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/4c79e8192f9f
Part 2: Fix browser jstests failures. r=jorendorff
Keywords: checkin-needed
Hmm, based on the time zone offsets in the error messages it looks like the "OS X 10.10" and "Android 4.3" both run with the time zone "America/Los_Angeles" instead of "PST8PDT". And "Android 7.0 x86" runs with "Europe/London" instead of "PST8PDT".
Two updates:

js/src/tests/non262/Date/15.9.4.2.js
- Handle the case when the time zone offset in January 1970 is different than the time zone offset in January 2009.


js/src/tests/non262/Date/parse-from-tostring-methods.js
- Handle the case when the time zone offset contains seconds (or milliseconds).
- The time zone offset for America/Los_Angeles is -7:52:58 until 1883 Nov 18 12:07:02 (cf. https://en.wikipedia.org/wiki/Local_Mean_Time)
- This also means the requirements for Date.parse in <https://tc39.github.io/ecma262/#sec-date.parse> need to be updated, because |x.valueOf()| may be different than |Date.parse(x.toString())| when the time zone string in |x.toString()| omits seconds offset values.
Attachment #9030846 - Attachment is obsolete: true
Flags: needinfo?(andrebargull)
Attachment #9031151 - Flags: review+
Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/94904f2a0a97
Part 1: Pass correct test type. r=ahal
https://hg.mozilla.org/integration/mozilla-inbound/rev/099ba012de27
Part 2: Fix browser jstests failures. r=jorendorff
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/94904f2a0a97
https://hg.mozilla.org/mozilla-central/rev/099ba012de27
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Depends on: 1514212
Duplicate of this bug: 1510819
You need to log in before you can comment on or make changes to this bug.