errorsummary.log showing pass when should have failed?
Categories
(Testing :: Mochitest, defect, P1)
Tracking
(firefox79 fixed)
Tracking | Status | |
---|---|---|
firefox79 | --- | fixed |
People
(Reporter: ekyle, Assigned: gbrown)
References
Details
Attachments
(1 file)
These all look ok:
ActiveData disagrees; dom/media/tests/mochitest/mochitest.ini
did not pass
https://activedata.allizom.org/tools/query.html#query_id=HXhUPwVr
Looking at the raw log, we can see why: https://firefoxci.taskcluster-artifacts.net/FF02lqupQLGGCYVDz26ViQ/0/public/test_info/mochitest-media_raw.log
dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html
is a test in that manifest, and that test's last entry is
{"status": "FAIL", "thread": "None", "subtest": "The author of the test has indicated that flaky timeouts are expected. Reason: WebRTC inherently depends on timeouts", "pid": null, "source": "mochitest", "test": "dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html", "js_source": "TestRunner.js", "time": 1590197207249, "action": "test_status"}
the test is missing a test_end
entry, which is supposed to indicate if the test passes or not.
Reporter | ||
Updated•4 years ago
|
Comment 1•4 years ago
|
||
The failure line looks like https://searchfox.org/mozilla-central/rev/598e50d2c3cd81cd616654f16af811adceb08f9f/testing/mochitest/tests/SimpleTest/SimpleTest.js#776, but SimpleTest.todo
shouldn't cause a failure, should it?
Comment 2•4 years ago
|
||
Another data point: the task was green.
Assignee | ||
Comment 3•4 years ago
|
||
(In reply to Marco Castelluccio [:marco] from comment #1)
The failure line looks like https://searchfox.org/mozilla-central/rev/598e50d2c3cd81cd616654f16af811adceb08f9f/testing/mochitest/tests/SimpleTest/SimpleTest.js#776, but
SimpleTest.todo
shouldn't cause a failure, should it?
No, it should not.
Assignee | ||
Comment 4•4 years ago
|
||
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=303447501&repo=autoland&lineNumber=15494
[task 2020-05-23T01:26:46.133Z] 01:26:46 INFO - TEST-START | dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html
...
[task 2020-05-23T01:26:47.314Z] 01:26:47 INFO - Exception in thread ProcessReader:
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - Traceback (most recent call last):
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - self.run()
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - self.__target(*self.__args, **self.__kwargs)
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/Users/cltbld/tasks/task_1590196772/build/venv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 1116, in _read
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - callback(line.rstrip())
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/Users/cltbld/tasks/task_1590196772/build/venv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 1026, in __call__
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - e(*args, **kwargs)
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/Users/cltbld/tasks/task_1590196772/build/tests/mochitest/runtests.py", line 3067, in processOutputLine
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - message = handler(message)
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - File "/Users/cltbld/tasks/task_1590196772/build/tests/mochitest/runtests.py", line 3132, in countline
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - line = message['message']
[task 2020-05-23T01:26:47.315Z] 01:26:47 INFO - KeyError: 'message'
[task 2020-05-23T01:31:51.147Z] 01:31:51 INFO - TEST-INFO | Main app process: exit 0
[task 2020-05-23T01:31:51.147Z] 01:31:51 INFO - runtests.py | Application ran for: 0:06:46.229322
Assignee | ||
Comment 5•4 years ago
|
||
This code was updated in bug 1607763, but it looks like that change did not regress this: it would have failed just the same before that change.
Assignee | ||
Updated•4 years ago
|
Comment 6•4 years ago
|
||
The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 7•4 years ago
|
||
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8a842264000a Tolerate more log formats when searching for mochitest summary counts; r=bc
Comment 9•4 years ago
|
||
bugherder |
Description
•