1433390 - Enable assertion introspection for support.asserts module in wdspec tests

Status: RESOLVED FIXED

(Testing :: geckodriver, enhancement, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Log lines for failures like the following are pretty useless at the moment:

https://treeherder.mozilla.org/logviewer.html#?job_id=158638024&repo=try&lineNumber=1425

[task 2018-01-26T07:39:52.934Z] 07:39:52     INFO -             if e["type"] != "mousemove":
[task 2018-01-26T07:39:52.935Z] 07:39:52     INFO -                 assert e["pageX"] == point["x"]
[task 2018-01-26T07:39:52.935Z] 07:39:52     INFO - >               assert e["pageY"] == point["y"]
[task 2018-01-26T07:39:52.935Z] 07:39:52     INFO - E               AssertionError

There is basically no way to identify by the log what the problem is. I really want to see array introspection working here. While it is working fine if assertions are in the tests, it fails for helper modules like the above.

After searching a bit I found the following stackoverflow article:

https://stackoverflow.com/questions/41522767/pytest-assert-introspection-in-helper-function

We should enable that introspection for all of our helpers.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1433390 - Move all assert helps into support.asserts.

For a better management of assertion helpers those should be
added to the central assertion helper module.

Review commit: https://reviewboard.mozilla.org/r/215842/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215842/

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1433390 - Use register_assert_rewrite to enable introspection in assertion helper.

Pytest by default only loads test modules, and enables assert
introspection for them. For any helper method which also uses
asserts, only a plain assertion is printed.

To enable introspection also for the assertion helper, the
module has to be registered for assertion rewrite.

Review commit: https://reviewboard.mozilla.org/r/215844/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215844/

Comment 3

[James Graham [:jgraham]]

Comment on attachment 8945738 [details]
Bug 1433390 - Use register_assert_rewrite to enable introspection in assertion helper.

https://reviewboard.mozilla.org/r/215844/#review221720

Comment 4

[James Graham [:jgraham]]

Comment on attachment 8945737 [details]
Bug 1433390 - Move all assert helps into support.asserts.

https://reviewboard.mozilla.org/r/215842/#review221722

Assuming you tested the fll changeset actually works.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f90bf2ca184
Move all assert helps into support.asserts. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/955a54d5f82b
Use register_assert_rewrite to enable introspection in assertion helper. r=jgraham

Comment 6

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/3f90bf2ca184
https://hg.mozilla.org/mozilla-central/rev/955a54d5f82b

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Small change which drastically helps in test failure discovery in log files. It would be great to see uplifted to beta.

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/982cc606296b
https://hg.mozilla.org/releases/mozilla-beta/rev/24685f33b77a