1047101 - Provide a way to do data-driven tests with Marionette test runner

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Zac C (:zac)]

It would be great to have a nice clean way to do data-driven tests in the Marionette test runner.


ddt has a nice implementation but it does not accept a variable unfortunately:
http://ddt.readthedocs.org/en/latest/example.html

Comment 1

[Julien Pagès (:parkouss)]

I think I wrote something similar for one project recently:

- http://funq.readthedocs.org/en/latest/user_api/testcase.html#funq.testcase.parameterized
- http://funq.readthedocs.org/en/latest/user_api/testcase.html#funq.testcase.with_parameters

It is pretty close to what ddt do as far as I understand - what do you mean Zac with "it does not accept a variable unfortunately" ? It seems to me that both ddt and the other implementation accepts variables to dynamically create new methods. Can you give me some examples of what you would like to see ?

Comment 2

[Zac C (:zac)]

Hey Julien, that looks really good, I wish I'd found this when I originally googled around! 
Yes I'd like to be able to pass in a Python list, for example, and have that become the test data.

I'll have to speak to the maintainers (eg mdas) to see how it could integrate into our existing framework.

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Zac C (:zac) from comment #2)
> Hey Julien, that looks really good, I wish I'd found this when I originally
> googled around! 
> Yes I'd like to be able to pass in a Python list, for example, and have that
> become the test data.
> 
> I'll have to speak to the maintainers (eg mdas) to see how it could
> integrate into our existing framework.

Looks like we just need to change the base class that CommonTestCase inherits from to be funq.testcase.FunqTestCase instead of the unittest's basic TestCase class, and then you can import the decorators and use them. I haven't tested it, but it seems like a simple change.

Comment 4

[Julien Pagès (:parkouss)]

In fact I think it may be easier to only add a __metaclass__ attribute on CommonTestCase (with a value of MetaParameterized) and copy paste the implementation of MetaParameterized, parameterized and with_parameters from funq. Then import decorators and use them will just work, but you won't require the funq package that I think is not relevant here.

The needed implementation only takes a few lines, as you can see here: http://funq.readthedocs.org/en/latest/_modules/funq/testcase.html#with_parameters. Plus you can adapt if there is something you don't like.

I can do the work, and attach a patch, if you're interested.

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Julien Pagès from comment #4)
> In fact I think it may be easier to only add a __metaclass__ attribute on
> CommonTestCase (with a value of MetaParameterized) and copy paste the
> implementation of MetaParameterized, parameterized and with_parameters from
> funq. Then import decorators and use them will just work, but you won't
> require the funq package that I think is not relevant here.
> 
> The needed implementation only takes a few lines, as you can see here:
> http://funq.readthedocs.org/en/latest/_modules/funq/testcase.
> html#with_parameters. Plus you can adapt if there is something you don't
> like.
> 
> I can do the work, and attach a patch, if you're interested.

Sure, that sounds great, thanks!

Comment 6

[Julien Pagès (:parkouss)]

Attachment: Provide a way to do data-driven tests with Marionette test runner

I modified the test_simpletest_sanity.py file in the same file to demonstrate how to use the @parameterized decorator - tell me if it is not relevant. :)

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 8479765 [details] [diff] [review]
Provide a way to do data-driven tests with Marionette test runner

Review of attachment 8479765 [details] [diff] [review]:
-----------------------------------------------------------------

works well! can you remove the extra whitespace from marionette_test.py? They show up in red if you look at that file on this page: https://bugzilla.mozilla.org/page.cgi?id=splinter.html&bug=1047101&attachment=8479765.

r- for now because I'd like a unit test for this decorator. Adding it to test_simpletest_sanity.py doesn't make much sense as a unit-test, and the other issue here is that test_simpletest_sanity.py should reset the context in a tearDown function in case anything fails during the test.

Comment 8

[Julien Pagès (:parkouss)]

Attachment: Provide a way to do data-driven tests with Marionette test runner

I removed the blank lines and the modifications done to test_simpletest_sanity.py - it was a bad idea - and wrote some unit tests instead as you recommended. :)

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 8480451 [details] [diff] [review]
Provide a way to do data-driven tests with Marionette test runner

Review of attachment 8480451 [details] [diff] [review]:
-----------------------------------------------------------------

Nice, the test looks great! But there's one more thing missing: the new test needs to be added to https://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/unit-tests.ini so it can be run on tbpl. Can you update your patch to do this? I'll r+ then, thanks!

Oh, and if you get around to this after today, can you ask for r? from either jgriffin or AutomatedTester? I'll be gone for a week.

Comment 10

[Julien Pagès (:parkouss)]

Attachment: Provide a way to do data-driven tests with Marionette test runner

:mdas, I added the new test to the tests/unit/unit-tests.ini file. :jgriffin, you are now the final reviewer for this patch since :mdas won't be here fo a week (see comment #9) - thanks !

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8481813 [details] [diff] [review]
Provide a way to do data-driven tests with Marionette test runner

Review of attachment 8481813 [details] [diff] [review]:
-----------------------------------------------------------------

Nice!  Pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=997e029ff7d5

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9906ac54c8fc

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9906ac54c8fc