1860006 - errorsummary Manifest is marked as "OK" even if it failed

Status: NEW

(Firefox Build System :: Task Configuration, defect, P3)

Description

[Marco Castelluccio [:marco]]

In task O0hmOq6rQjGAwdjP_ePtqA, which is a test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-5, the errorsummary file is:

{"groups": ["dom/abort/tests/mochitest.toml", "dom/base/test/mochitest.toml", "dom/cache/test/mochitest/mochitest.toml", "dom/credentialmanagement/tests/mochitest/mochitest.toml", "dom/fetch/tests/mochitest.toml", "dom/fs/test/mochitest/mochitest.toml", "dom/indexedDB/test/mochitest-intl-api.ini", "dom/l10n/tests/mochitest/mochitest.toml", "dom/messagechannel/tests/mochitest.toml", "dom/security/sanitizer/tests/mochitest/mochitest.toml", "dom/security/test/https-only/mochitest.toml", "dom/serviceworkers/test/mochitest-dFPI.ini:dom/serviceworkers/test/mochitest-common.ini", "dom/tests/mochitest/ajax/scriptaculous/mochitest.ini", "dom/tests/mochitest/dom-level1-core/mochitest.ini", "dom/tests/mochitest/general/mochitest.ini", "dom/tests/mochitest/pointerlock/mochitest.ini", "dom/tests/mochitest/whatwg/mochitest.ini", "dom/worklet/tests/mochitest.ini", "editor/composer/test/mochitest.toml", "image/test/mochitest/mochitest.toml", "js/xpconnect/tests/mochitest/mochitest.toml", "layout/style/test/mochitest.toml", "mobile/android/components/extensions/test/mochitest/mochitest.toml", "security/manager/ssl/tests/mochitest/mixedcontent/mochitest.toml", "toolkit/components/extensions/test/mochitest/mochitest.toml:toolkit/components/extensions/test/mochitest/mochitest-common.toml", "toolkit/components/extensions/test/mochitest/mochitest.toml", "toolkit/content/tests/widgets/mochitest.toml", "widget/tests/mochitest.toml"], "action": "test_groups", "line": 487}
{"level": "ERROR", "message": "TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_lock_orientation_with_pending_fullscreen.html logged result after SimpleTest.finish(): Fullscreen request is canceled.", "action": "log", "line": 1584}
{"group": "dom/base/test/mochitest.toml", "status": "OK", "duration": 534206, "action": "group_result", "line": 6236}
{"group": "dom/cache/test/mochitest/mochitest.toml", "status": "OK", "duration": 617225, "action": "group_result", "line": 6236}
{"group": "dom/security/test/https-only/mochitest.toml", "status": "OK", "duration": 792274, "action": "group_result", "line": 6236}
{"group": "dom/serviceworkers/test/mochitest-dFPI.ini:dom/serviceworkers/test/mochitest-common.ini", "status": "OK", "duration": 894896, "action": "group_result", "line": 6236}
{"group": "dom/tests/mochitest/general/mochitest.ini", "status": "OK", "duration": 1211598, "action": "group_result", "line": 6236}
{"group": "dom/tests/mochitest/pointerlock/mochitest.ini", "status": "OK", "duration": 1230180, "action": "group_result", "line": 6236}
{"group": "dom/tests/mochitest/whatwg/mochitest.ini", "status": "OK", "duration": 1258137, "action": "group_result", "line": 6236}
{"group": "editor/composer/test/mochitest.toml", "status": "OK", "duration": 1308326, "action": "group_result", "line": 6236}
{"group": "image/test/mochitest/mochitest.toml", "status": "OK", "duration": 1515197, "action": "group_result", "line": 6236}
{"group": "js/xpconnect/tests/mochitest/mochitest.toml", "status": "OK", "duration": 1557928, "action": "group_result", "line": 6236}
{"group": "layout/style/test/mochitest.toml", "status": "OK", "duration": 1969892, "action": "group_result", "line": 6236}
{"group": "mobile/android/components/extensions/test/mochitest/mochitest.toml", "status": "OK", "duration": 2016575, "action": "group_result", "line": 6236}
{"group": "security/manager/ssl/tests/mochitest/mixedcontent/mochitest.toml", "status": "OK", "duration": 2075894, "action": "group_result", "line": 6236}
{"group": "toolkit/components/extensions/test/mochitest/mochitest.toml:toolkit/components/extensions/test/mochitest/mochitest-common.toml", "status": "OK", "duration": 2363771, "action": "group_result", "line": 6236}
{"group": "toolkit/content/tests/widgets/mochitest.toml", "status": "OK", "duration": 2390652, "action": "group_result", "line": 6236}
{"group": "widget/tests/mochitest.toml", "status": "OK", "duration": 2408076, "action": "group_result", "line": 6236}
{"group": "dom/abort/tests/mochitest.toml", "status": "OK", "duration": 2431, "action": "group_result", "line": 6236}
{"group": "dom/credentialmanagement/tests/mochitest/mochitest.toml", "status": "OK", "duration": 1764, "action": "group_result", "line": 6236}
{"group": "dom/fetch/tests/mochitest.toml", "status": "OK", "duration": 2021, "action": "group_result", "line": 6236}
{"group": "dom/fs/test/mochitest/mochitest.toml", "status": "OK", "duration": 22510, "action": "group_result", "line": 6236}
{"group": "dom/indexedDB/test/mochitest-intl-api.ini", "status": "OK", "duration": 4199, "action": "group_result", "line": 6236}
{"group": "dom/l10n/tests/mochitest/mochitest.toml", "status": "OK", "duration": 1194, "action": "group_result", "line": 6236}
{"group": "dom/messagechannel/tests/mochitest.toml", "status": "OK", "duration": 8489, "action": "group_result", "line": 6236}
{"group": "dom/security/sanitizer/tests/mochitest/mochitest.toml", "status": "OK", "duration": 1175, "action": "group_result", "line": 6236}
{"group": "dom/tests/mochitest/ajax/scriptaculous/mochitest.ini", "status": "OK", "duration": 15444, "action": "group_result", "line": 6236}
{"group": "dom/tests/mochitest/dom-level1-core/mochitest.ini", "status": "OK", "duration": 192971, "action": "group_result", "line": 6236}
{"group": "dom/worklet/tests/mochitest.ini", "status": "OK", "duration": 23418, "action": "group_result", "line": 6236}
{"group": "toolkit/components/extensions/test/mochitest/mochitest.toml", "status": "OK", "duration": 3587, "action": "group_result", "line": 6236}

Failure summary is:
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_lock_orientation_with_pending_fullscreen.html logged result after SimpleTest.finish(): Fullscreen request is canceled.

dom/base/test/mochitest.toml is marked as "OK" even though dom/base/test/test_lock_orientation_with_pending_fullscreen.html failed.

The fact that the test name is /tests/dom/base/test/test_lock_orientation_with_pending_fullscreen.html instead of dom/base/test/test_lock_orientation_with_pending_fullscreen.html might have something to do with it.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is an error not specific to android, but in general to the mochitest harness- when we get an error outside of the standard test_status call, it is a raw string we create. If you look in the log file, the TEST-START has the proper test name. Even if the test name was proper, the group would need to be updated- all of this needs similar plumbing (3 cases I see so far in testrunner.js).

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit BugBot documentation.