1251674 - switch to pytest

Status: RESOLVED WONTFIX

(support.mozilla.org :: Code Quality, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Today we merged the QA test suite into the kitsune repository. That test suite (and the smoketests suite that preceded it) uses pytest as the test runner. Kitsune currently uses nose and django-nose.

Switching from nose/django-nose to pytest/pytest-django for the kitsune tests would have the following niceties:

1. lets us ditch code some minor goofy code that checks to see if we're running tests and if so uses the settings_test.py module for settings

2. lets us use one test runner for all the tests in the repo rather than two

3. alleviates some minor issues now and in the future with having two test runners in our requirements files

4. moves us to pytest/pytest-django which are more actively maintained projects


This bug covers switching to pytest/pytest-django.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to work on. I did this for Input and MDN and I'm pretty sure it's a small project.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bunch of complexities.

1. The QA test suite is in a folder called tests/ . That's problematic because all the folders are named "tests/" so pytest tries to run these tests because there doesn't seem to be a way to exclude just the top-level tests/ folder but include all the other ones.

Solution: We should rename it. Dave changed the name to "tests/" because the QA tests included more than just smoke tests. I think that makes sense, but "tests/" is too general and I think this should reflect that this test suite is different than the Django tests or the JS tests we have. I propose we rename it to qatests/ .


2. The QA tests require a different pytest configuration to run.

Solution: I propose we have a top-level setup.cfg file for the Django tests and a qatests/setup.cfg file for the QA tests.


3. The instructions for both the Django tests and QA tests require that they be run from the root of the repository. This is problematic because we need to have two different pytest configurations.

Solution: I propose we change the docs and all the other things to run the QA tests from the qatests/ directory same as we did for the smoketests directory that preceded it.


After doing those changes, the rest is pretty straight-forward. I did it in a day for Input and maybe a day for MDN. It's something like this for SUMO:

* adding pytest and pytest-django to the requirements
* removing all the nose stuff
* fixing the test environment setup so it's not nose testrunner specific
* implementing eq_ and ok_ and going through the test modules changing the imports
* fixing the SkipTest stuff to use pytest's decorator
* update docs
* update travis scripts and other scripts
* ...


Fixing all that will take a while. It's definitely not the "in an afternoon" project I estimated on IRC anymore. Plus it's a sufficient amount of churn that I think it's not worth doing now.

Given that, I'm unassigning myself.

Comment 3

[Michael Cooper [:mythmon]]

Given that this is not the small amount of work previously expected, I'm going to close this bug out. The amount of benefit it would provide is not large enough for the cost.