1636939 - WPT results in errorsummary.log can contain empty groups

Status: NEW

(Firefox Build System :: Task Configuration, defect, P3)

Description

[Marco Castelluccio [:marco]]

Recently, I started seeing test_result lines in Mochitest errorsummary.log files which contain an empty group.

I think it is a regression, as I have code relying on "group" not to be empty which has never failed for a long time.

E.g. in https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=52cf30bf74473e78c0baf8c4d6055910f67a0a37&searchStr=test-linux1804-64-shippable%2Fopt-mochitest-plain-e10s-1&selectedTaskRun=V_lcR6-nRaaOMyciFzg7Jw-0:

{"action": "test_groups", "line": 2, "groups": ["browser/extensions/formautofill/test/mochitest/creditCard/mochitest.ini", "testing/mochitest/tests/Harness_sanity/mochitest.ini", "dom/tests/mochitest/storageevent/mochitest.ini", "dom/tests/mochitest/sessionstorage/mochitest.ini", "browser/components/payments/test/mochitest/mochitest.ini", "image/test/mochitest/mochitest.ini", "dom/vr/test/mochitest/mochitest.ini", "docshell/test/iframesandbox/mochitest.ini", "dom/indexedDB/test/mochitest-intl-api.ini", "dom/credentialmanagement/tests/mochitest/mochitest.ini", "browser/components/payments/test/mochitest/formautofill/mochitest.ini", "layout/forms/test/mochitest.ini", "dom/payments/test/mochitest.ini", "dom/tests/mochitest/ajax/offline/mochitest.ini", "dom/file/tests/mochitest.ini", "netwerk/test/mochitests/mochitest.ini", "toolkit/components/windowcreator/test/mochitest.ini", "dom/canvas/test/crossorigin/mochitest.ini", "modules/libjar/test/mochitest/mochitest.ini", "dom/xhr/tests/mochitest.ini", "testing/mochitest/baselinecoverage/plain/mochitest.ini", "dom/tests/mochitest/keyhandling/mochitest.ini", "widget/tests/mochitest.ini", "intl/uconv/tests/mochitest.ini", "dom/indexedDB/test/mochitest.ini", "netwerk/cookie/test/mochitest/mochitest.ini", "dom/security/test/general/mochitest.ini", "dom/events/test/pointerevents/mochitest.ini", "dom/tests/mochitest/dom-level1-core/mochitest.ini", "parser/htmlparser/tests/mochitest/mochitest.ini", "dom/security/test/https-only/mochitest.ini", "dom/tests/mochitest/orientation/mochitest.ini", "caps/tests/mochitest/mochitest.ini", "security/manager/ssl/tests/mochitest/stricttransportsecurity/mochitest.ini", "dom/localstorage/test/mochitest.ini"]}
{"status": "FAIL", "group": "dom/indexedDB/test/mochitest.ini", "line": 4007, "subtest": "Test timed out.", "action": "test_result", "test": "dom/indexedDB/test/test_readonly_transactions.html", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:150:18\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "dom/indexedDB/test/mochitest.ini", "line": 4018, "subtest": "Test timed out.", "action": "test_result", "test": "dom/indexedDB/test/test_readwriteflush_disabled.html", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:150:18\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "dom/indexedDB/test/mochitest.ini", "line": 4029, "subtest": "Test timed out.", "action": "test_result", "test": "dom/indexedDB/test/test_remove_index.html", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:150:18\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "dom/indexedDB/test/mochitest.ini", "line": 4040, "subtest": "Test timed out.", "action": "test_result", "test": "dom/indexedDB/test/test_remove_objectStore.html", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:150:18\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "", "line": 4042, "subtest": "4 test timeouts, giving up.", "action": "test_result", "test": "(SimpleTest/TestRunner.js)", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:158:20\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "", "line": 4044, "subtest": "Skipping 26 remaining tests.", "action": "test_result", "test": "(SimpleTest/TestRunner.js)", "message": null, "stack": "    SimpleTest.ok@SimpleTest/SimpleTest.js:299:16\n    reportError@SimpleTest/TestRunner.js:128:22\n    TestRunner._checkForHangs@SimpleTest/TestRunner.js:163:20\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    setTimeout handler*TestRunner._checkForHangs@SimpleTest/TestRunner.js:184:15\n    TestRunner.runTests/<@SimpleTest/TestRunner.js:420:16\n    promise callback*TestRunner.runTests@SimpleTest/TestRunner.js:407:48\n    RunSet.runtests@SimpleTest/setup.js:218:14\n    RunSet.runall@SimpleTest/setup.js:197:12\n    hookupTests@SimpleTest/setup.js:294:12\nparseTestManifest@http://mochi.test:8888/manifestLibrary.js:46:13\ngetTestManifest/req.onload@http://mochi.test:8888/manifestLibrary.js:59:28\nEventHandlerNonNull*getTestManifest@http://mochi.test:8888/manifestLibrary.js:55:3\n    hookup@SimpleTest/setup.js:270:20\nEventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&cleanupCrashes=true:11:1\n", "expected": "PASS"}
{"status": "FAIL", "group": "", "line": 4047, "subtest": "/tests/dom/indexedDB/test/test_remove_objectStore.html", "action": "test_result", "test": "(SimpleTest/TestRunner.js)", "message": "finished in a non-clean fashion, probably because it didn't call SimpleTest.finish()", "stack": "{u'loaded_test_url': u'/tests/dom/indexedDB/test/test_remove_objectStore.html'}", "expected": "PASS"}
{"status": "ERROR", "group": "", "line": 4048, "subtest": null, "action": "test_result", "test": "(SimpleTest/TestRunner.js)", "message": "Finished in 329989ms", "stack": "{u'runtime': 329989}", "expected": "OK"}

Comment 1

[Marco Castelluccio [:marco]]

Same in https://firefoxci.taskcluster-artifacts.net/RYmCJO0fQsOor5jwtCCUzw/0/public/test_info//wpt_errorsummary.log:

{"status": "FAIL", "group": "", "line": 79772, "subtest": "Call foo", "action": "test_result", "test": "/html/cross-origin-opener-policy/access-reporting/opener-accessed_openee-coop-ro.https.html", "message": "assert_not_equals: got disallowed value \"timeout\"", "stack": "@https://web-platform.test:8443/html/cross-origin-opener-policy/access-reporting/opener-accessed_openee-coop-ro.https.html:55:22\n", "expected": "TIMEOUT"}
{"status": "TIMEOUT", "group": "", "line": 79773, "subtest": "Call location", "action": "test_result", "test": "/html/cross-origin-opener-policy/access-reporting/opener-accessed_openee-coop-ro.https.html", "message": "Test timed out", "stack": null, "expected": "NOTRUN"}

(as a side note, in this task "suite_end" was not emitted)

Comment 2

[Marco Castelluccio [:marco]]

WPT is probably bug 1646761.

Comment 3

[Andrew Halberstadt [:ahal]]

Marco, do you know if this is still an issue? Or should we close it out and re-open if it surfaces again.

Comment 4

[Marco Castelluccio [:marco]]

Yes this is still happening.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Marco, I believe I have fixed most if not all of these cases, can you comment if there are remaining cases?

Comment 6

[Marco Castelluccio [:marco]]

I'm still seeing this, but only in WPT tasks.

For example, in https://firefox-ci-tc.services.mozilla.com/tasks/Hhw-OLgAT7a3ciUegdPDFg/runs/0/logs/public/test_info/wpt_errorsummary.log:

{"groups": ["testharness"], "action": "test_groups", "line": 30}
{"test": "/css/cssom-view/scroll-behavior-main-frame-root.html", "subtest": "Main frame with auto scroll-behavior ; scroll() with smooth behavior", "group": "", "status": "PASS", "expected": "FAIL", "message": null, "stack": null, "known_intermittent": ["PASS"], "action": "test_result", "line": 10924}
...
{"group": "", "status": "ERROR", "duration": null, "action": "group_result", "line": 233100}

In particular, the last line is:
{"group": "", "status": "ERROR", "duration": null, "action": "group_result", "line": 233100}

Likely related to bug 1859529.