Closed
Bug 1475999
Opened 6 years ago
Closed 6 years ago
Treeherder should highlight Pytest failures
Categories
(Testing :: Python Test, defect)
Testing
Python Test
Tracking
(firefox63 fixed)
RESOLVED
FIXED
mozilla63
Tracking | Status | |
---|---|---|
firefox63 | --- | fixed |
People
(Reporter: ted, Assigned: davehunt)
Details
Attachments
(1 file)
I pushed some patches to try that made some Python unit tests fail, but the failures don't show up in the Treeherder log highlights: https://treeherder.mozilla.org/logviewer.html#?job_id=188349147&repo=try&lineNumber=43448 Here are a few snippets that I think ought to show up: [task 2018-07-16T13:19:38.563Z] 13:19:38 INFO - ../python/mozbuild/mozbuild/test/frontend/test_emitter.py::TestEmitterBasic::test_crate_dependency_path_resolution FAILED [ 6%] [task 2018-07-16T13:19:38.605Z] 13:19:38 INFO - =================================== FAILURES =================================== [task 2018-07-16T13:19:38.605Z] 13:19:38 INFO - ____________ TestEmitterBasic.test_crate_dependency_path_resolution ____________ [task 2018-07-16T13:19:38.605Z] 13:19:38 INFO - self = <mozbuild.test.frontend.test_emitter.TestEmitterBasic testMethod=test_crate_dependency_path_resolution> [task 2018-07-16T13:19:38.605Z] 13:19:38 INFO - def test_crate_dependency_path_resolution(self): [task 2018-07-16T13:19:38.606Z] 13:19:38 INFO - '''Test recursive dependencies resolve with the correct paths.''' [task 2018-07-16T13:19:38.606Z] 13:19:38 INFO - reader = self.reader('crate-dependency-path-resolution', [task 2018-07-16T13:19:38.606Z] 13:19:38 INFO - extra_substs=dict(RUST_TARGET='i686-pc-windows-msvc')) [task 2018-07-16T13:19:38.606Z] 13:19:38 INFO - objs = self.read_topsrcdir(reader) [task 2018-07-16T13:19:38.607Z] 13:19:38 INFO - > ldflags, lib = objs [task 2018-07-16T13:19:38.607Z] 13:19:38 INFO - E ValueError: too many values to unpack [task 2018-07-16T13:19:38.607Z] 13:19:38 INFO - ../python/mozbuild/mozbuild/test/frontend/test_emitter.py:1590: ValueError I don't know if this is strictly a Treeherder issue or if our Pytest implementation could be providing more informative logs.
Assignee | ||
Comment 1•6 years ago
|
||
The tests are currently using the pytest default console logging, which means Treeherder isn't parsing and highlighting the failures. We could silence the default output and use pytest-mozlog to output in TBPL format, which should make the failures stand out in Treeherder. To enable TBPL output, add `--log-tbpl=-` to the command line. To disable the default console output, add `-p no:terminal` to the command line. Note that this disables the core terminal writer plugin entirely, which may have some undesirable consequences. Perhaps there is a better way to disable the default console logging since I last looked at this. We could do this for all pytests by adding these command line options to https://searchfox.org/mozilla-central/rev/b0275bc977ad7fda615ef34b822bba938f2b16fd/config/mozunit/mozunit/mozunit.py#246-253
Comment 2•6 years ago
|
||
Oh I think this got regressed by bumping the pytest version (bug 1472201), though the regression is the fault of our hacky code rather than any backwards incompatibilities. We're *supposed* to be converting the "FAILED" string to "TEST-UNEXPECTED-FAIL": https://searchfox.org/mozilla-central/source/python/mach_commands.py#205 But it looks like the new version of pytest adds a progress counter which breaks the fragile substitution. As far as using pytest-mozlog goes, I'd personally be against using it unless we implemented a better formatter. While the default pytest format can be overly verbose (especially when running locally), that verbosity is really helpful in CI where having as much information as possible can save us from relying on the print/push-to-try debug cycle. So my vote would be to just fix our hack (and also make it a bit more robust).
Updated•6 years ago
|
Component: Treeherder: Log Parsing & Classification → Python Test
Product: Tree Management → Testing
Version: --- → unspecified
Assignee | ||
Comment 3•6 years ago
|
||
Ah, well we can disable the progress bar: https://docs.pytest.org/en/latest/reference.html#confval-console_output_style > # content of pytest.ini > [pytest] > console_output_style = classic
Comment hidden (mozreview-request) |
Assignee | ||
Updated•6 years ago
|
Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED
Reporter | ||
Comment 5•6 years ago
|
||
mozreview-review |
Comment on attachment 8992593 [details] Bug 1475999 - Use the classic output style with pytest; https://reviewboard.mozilla.org/r/257466/#review264356
Attachment #8992593 -
Flags: review?(ted) → review+
Pushed by dhunt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f28cfacdb5f5 Use the classic output style with pytest; r=ted
Comment 7•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/f28cfacdb5f5
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox63:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in
before you can comment on or make changes to this bug.
Description
•