Closed Bug 1643400 Opened 1 month ago Closed 1 month ago

errorsummary.log showing pass when should have failed?

Categories

(Testing :: Mochitest, defect, P1)

defect

Tracking

(firefox79 fixed)

RESOLVED FIXED
mozilla79
Tracking Status
firefox79 --- fixed

People

(Reporter: ekyle, Assigned: gbrown)

References

Details

Attachments

(1 file)

These all look ok:

https://firefoxci.taskcluster-artifacts.net/FF02lqupQLGGCYVDz26ViQ/0/public/test_info/mochitest-media_errorsummary.log

ActiveData disagrees; dom/media/tests/mochitest/mochitest.ini did not pass

https://activedata.allizom.org/tools/query.html#query_id=HXhUPwVr

Looking at the raw log, we can see why: https://firefoxci.taskcluster-artifacts.net/FF02lqupQLGGCYVDz26ViQ/0/public/test_info/mochitest-media_raw.log

dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html is a test in that manifest, and that test's last entry is

{"status": "FAIL", "thread": "None", "subtest": "The author of the test has indicated that flaky timeouts are expected. Reason: WebRTC inherently depends on timeouts", "pid": null, "source": "mochitest", "test": "dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html", "js_source": "TestRunner.js", "time": 1590197207249, "action": "test_status"}

the test is missing a test_end entry, which is supposed to indicate if the test passes or not.

Severity: -- → S3
Priority: -- → P3

Another data point: the task was green.

See Also: → 1643448

(In reply to Marco Castelluccio [:marco] from comment #1)

The failure line looks like https://searchfox.org/mozilla-central/rev/598e50d2c3cd81cd616654f16af811adceb08f9f/testing/mochitest/tests/SimpleTest/SimpleTest.js#776, but SimpleTest.todo shouldn't cause a failure, should it?

No, it should not.

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=303447501&repo=autoland&lineNumber=15494

[task 2020-05-23T01:26:46.133Z] 01:26:46     INFO - TEST-START | dom/media/tests/mochitest/test_peerConnection_addtrack_removetrack_events.html
...
[task 2020-05-23T01:26:47.314Z] 01:26:47     INFO -  Exception in thread ProcessReader:
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -  Traceback (most recent call last):
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      self.run()
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      self.__target(*self.__args, **self.__kwargs)
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/Users/cltbld/tasks/task_1590196772/build/venv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 1116, in _read
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      callback(line.rstrip())
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/Users/cltbld/tasks/task_1590196772/build/venv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 1026, in __call__
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      e(*args, **kwargs)
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/Users/cltbld/tasks/task_1590196772/build/tests/mochitest/runtests.py", line 3067, in processOutputLine
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      message = handler(message)
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -    File "/Users/cltbld/tasks/task_1590196772/build/tests/mochitest/runtests.py", line 3132, in countline
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -      line = message['message']
[task 2020-05-23T01:26:47.315Z] 01:26:47     INFO -  KeyError: 'message'
[task 2020-05-23T01:31:51.147Z] 01:31:51     INFO - TEST-INFO | Main app process: exit 0
[task 2020-05-23T01:31:51.147Z] 01:31:51     INFO - runtests.py | Application ran for: 0:06:46.229322

This code was updated in bug 1607763, but it looks like that change did not regress this: it would have failed just the same before that change.

Assignee: nobody → gbrown
Component: Mozbase → Mochitest

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Priority: P3 → --
Priority: -- → P1
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8a842264000a
Tolerate more log formats when searching for mochitest summary counts; r=bc
Status: NEW → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla79
See Also: → 1644165
You need to log in before you can comment on or make changes to this bug.