Closed Bug 1475999 Opened 6 years ago Closed 6 years ago

Treeherder should highlight Pytest failures

Categories

(Testing :: Python Test, defect)

defect
Not set
normal

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: ted, Assigned: davehunt)

Details

Attachments

(1 file)

I pushed some patches to try that made some Python unit tests fail, but the failures don't show up in the Treeherder log highlights:
https://treeherder.mozilla.org/logviewer.html#?job_id=188349147&repo=try&lineNumber=43448

Here are a few snippets that I think ought to show up:
[task 2018-07-16T13:19:38.563Z] 13:19:38     INFO - ../python/mozbuild/mozbuild/test/frontend/test_emitter.py::TestEmitterBasic::test_crate_dependency_path_resolution FAILED [  6%]

[task 2018-07-16T13:19:38.605Z] 13:19:38     INFO - =================================== FAILURES ===================================
[task 2018-07-16T13:19:38.605Z] 13:19:38     INFO - ____________ TestEmitterBasic.test_crate_dependency_path_resolution ____________
[task 2018-07-16T13:19:38.605Z] 13:19:38     INFO - self = <mozbuild.test.frontend.test_emitter.TestEmitterBasic testMethod=test_crate_dependency_path_resolution>
[task 2018-07-16T13:19:38.605Z] 13:19:38     INFO -     def test_crate_dependency_path_resolution(self):
[task 2018-07-16T13:19:38.606Z] 13:19:38     INFO -         '''Test recursive dependencies resolve with the correct paths.'''
[task 2018-07-16T13:19:38.606Z] 13:19:38     INFO -         reader = self.reader('crate-dependency-path-resolution',
[task 2018-07-16T13:19:38.606Z] 13:19:38     INFO -                              extra_substs=dict(RUST_TARGET='i686-pc-windows-msvc'))
[task 2018-07-16T13:19:38.606Z] 13:19:38     INFO -         objs = self.read_topsrcdir(reader)
[task 2018-07-16T13:19:38.607Z] 13:19:38     INFO - >       ldflags, lib = objs
[task 2018-07-16T13:19:38.607Z] 13:19:38     INFO - E       ValueError: too many values to unpack
[task 2018-07-16T13:19:38.607Z] 13:19:38     INFO - ../python/mozbuild/mozbuild/test/frontend/test_emitter.py:1590: ValueError

I don't know if this is strictly a Treeherder issue or if our Pytest implementation could be providing more informative logs.
The tests are currently using the pytest default console logging, which means Treeherder isn't parsing and highlighting the failures. We could silence the default output and use pytest-mozlog to output in TBPL format, which should make the failures stand out in Treeherder.

To enable TBPL output, add `--log-tbpl=-` to the command line.

To disable the default console output, add `-p no:terminal` to the command line. Note that this disables the core terminal writer plugin entirely, which may have some undesirable consequences. Perhaps there is a better way to disable the default console logging since I last looked at this.

We could do this for all pytests by adding these command line options to https://searchfox.org/mozilla-central/rev/b0275bc977ad7fda615ef34b822bba938f2b16fd/config/mozunit/mozunit/mozunit.py#246-253
Oh I think this got regressed by bumping the pytest version (bug 1472201), though the regression is the fault of our hacky code rather than any backwards incompatibilities.

We're *supposed* to be converting the "FAILED" string to "TEST-UNEXPECTED-FAIL":
https://searchfox.org/mozilla-central/source/python/mach_commands.py#205

But it looks like the new version of pytest adds a progress counter which breaks the fragile substitution.

As far as using pytest-mozlog goes, I'd personally be against using it unless we implemented a better formatter. While the default pytest format can be overly verbose (especially when running locally), that verbosity is really helpful in CI where having as much information as possible can save us from relying on the print/push-to-try debug cycle. So my vote would be to just fix our hack (and also make it a bit more robust).
Component: Treeherder: Log Parsing & Classification → Python Test
Product: Tree Management → Testing
Version: --- → unspecified
Ah, well we can disable the progress bar: https://docs.pytest.org/en/latest/reference.html#confval-console_output_style

> # content of pytest.ini
> [pytest]
> console_output_style = classic
Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED
Comment on attachment 8992593 [details]
Bug 1475999 - Use the classic output style with pytest;

https://reviewboard.mozilla.org/r/257466/#review264356
Attachment #8992593 - Flags: review?(ted) → review+
Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f28cfacdb5f5
Use the classic output style with pytest; r=ted
https://hg.mozilla.org/mozilla-central/rev/f28cfacdb5f5
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: