1465887 - Ability to specify python version per test

Status: RESOLVED FIXED

(Testing :: Python Test, enhancement)

Description

[Andrew Halberstadt [:ahal]]

Bug 1388013 is implementing the ability to run |mach python-test --three|, which will run the specified tests using python 3. This is great and will allow us to start adding tasks that run our tests with python 3!

But I'd like to take it a step further by allowing us to specify the python versions a test can run against in the manifest. E.g:

[DEFAULT]
py=2,3
[test_foo.py]
[test_bar.py]

This way, when you run |mach python-test --three|, we can filter out all the tests that don't work there yet. One could even imagine a world where running |mach python-test| will run tests against *both* 2 and 3 in the same invocation (similar to tox).

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I already have a version of this in my patch series for bug 1428362.

Comment 2

[Andrew Halberstadt [:ahal]]

Oh, ok. I can hold off if you like (though I also managed to get something minimal working). Heh, I should just stop working on things as you seem to be a step ahead :p.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
> Oh, ok. I can hold off if you like (though I also managed to get something
> minimal working). Heh, I should just stop working on things as you seem to
> be a step ahead :p.

If you have something working then please go ahead and upload it for review, it shouldn't be blocked on my work. I can just rebase my patch series as needed.

In my patch I'm passing `py3=True` to manifest parser when Python 3 is in use, and then `skip-if = py3 == true` in the manifests for tests that are not yet ready for Python 3. My reasoning is that I'd like tests by default to run against Python 3, and require them to be explicitly skipped. This reduces the chance of us missing tests, and also makes it relatively easy to see how much work is remaining to by compatible with Python 3.

On IRC you mentioned using a python version instead of a boolean, something like `python=3` perhaps, so we would be able to skip using something like `skip-if = python == 3`. I like this idea too, as eventually we can have py3 only packages and skip the tests on py2.

Comment 4

[Andrew Halberstadt [:ahal]]

It's ok, I'm in no rush to fix this and if you have a manifest solution implemented already then that's better than what I had. Feel free to resolve this a dupe if appropriate.

Comment 5

[Andrew Halberstadt [:ahal]]

Unassigning myself to avoid stepping on Dave's toes.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Bug 1465887 - Allow test manifests to skip based on Python version;

This patch allows us to enable Python 3 tests and skip any tests that fail so that we can work on adding support for Python 3 without risking regressing any existing support. It will also eventually allow us to skip tests from running against Python 2, when we decide to drop support for it. To skip a test against Python 3, add "skip-if = python == 3" to the [DEFAULT] or test file section of a manifest file.

Review commit: https://reviewboard.mozilla.org/r/250836/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/250836/

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8985187 [details]
Bug 1465887 - Allow test manifests to skip based on Python version;

https://reviewboard.mozilla.org/r/250836/#review257388

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a6d20e98283
Allow test manifests to skip based on Python version; r=ahal

Comment 9

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/7a6d20e98283