1207067 - Improve unittest assertion methods for better handling of msg parameter

Status: RESOLVED WONTFIX

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The default unittest assert* methods have a big lack in terms of detailed  messages in case of an assertion is thrown. You only have the possibility to see the actual compared values OR to get the message printed out. But they do not offer both at the same time which will be way more helpful in terms of analyzing failing tests. Here an example:

self.assertEqual(1, 2)
> "AssertionError: 1 != 2

self.assertEqual(1, 2, "There is not a pair")
> AssertionError: There is not a pair


Something which would be great to have is:

self.assertEqual(1, 2, "There is not a pair")
> AssertionError: There is not a pair: 1 != 2

It could easily be done by extending the unittest assertion methods like that:

    def assertEqual(self, first, second, msg=None):
        try:
            MarionetteTestCase.assertEqual(self, first, second)
        except AssertionError:
            exception_type, ex, tb = sys.exc_info()

            msg = '{}: {}'.format(msg, ex.message) if msg else e.message
            raise AssertionError, msg, tb

Without that you would always have to write:

self.assertEqual(a, b, "There is not a pair: {} != {}".format(a, b))

I find it kinda annoying especially for our Firefox UI tests and really would like to make it easier to analyze failures by throwing the correct information directly.

David, what do you think about that proposal and especially if it could be helpful as an enhancement for MarionetteTestCase. If we want to use that we can extend the most used methods.

Comment 1

[David Burns :automatedtester]

self.assertEqual(a, b, "There is not a pair: {} != {}".format(a, b)) is considered the best practise. I don't want to get into a place where we need to support it long term.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so I got the idea to check how Python 3 actually will handle that. And interestingly it will put the message beside the actual condition:

>>> a.assertEqual(2, 3, "There is not a pair")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/unittest/case.py", line 797, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.4/unittest/case.py", line 790, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 3 : There is not a pair

So it means when we do the above proposed formatting we will have to rewrite most of our code and we cannot easily switch between Python 2.7 and 3.x - which may even not work due to compat issues.

Not sure what our story with running Python 3.x on the testing boxes is, but I think that I don't want to update all of our code with messages now, which I then have to change again. So maybe we are waiting with this bug until we have to run Python 3?

Comment 3

[David Burns :automatedtester]

Waiting for Python 3 might be a while. Python 2.7 is going to be supported until 2020. My personal opinion is that it would be great to start supporting python 3 from now but without test infrastructure we are regularly going to be breaking things. It's not worth the effort at that point. I suggest speaking to releng when they might be willing to upgrade to python 3

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, in that case I would say we close this bug as incomplete for now and reopen when we actually can do some action.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Python2 is dead soon, and given that Python3 is handling it on its own, there is nothing we should do here.