Closed Bug 1210759 Opened 9 years ago Closed 4 years ago

Run PYTHON_UNIT_TESTS somewhere other than `make check`

Categories

(Firefox Build System :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Assigned: rstewart)

References

(Blocks 2 open bugs)

Details

(Keywords: in-triage)

Right now we run PYTHON_UNIT_TESTS as part of a recursive `make check`. I'd like to move them out of there because:
a) `make check` is slow because it recurses and
b) We don't run `make check` on cross-compiles, and I've just recently stood up a cross-compiled Mac build of Firefox and would like to replace our existing Mac builds with these without losing test coverage.

I think maybe we could just run these at the start of the build after we populate the virtualenv. That way builds would also fail fast for test failures.

We might need to split out some tests to a separate class to run *after* the build if they need access to products of the build. I'm not sure if we have any existing Python unittests that are like this, but some of the random stuff in check:: targets runs Python scripts against build products, so those could be migrated there.
They are too slow to run as a precheck. At least some of them.

If we switched to nose and used all available cores to run them, they would run considerably faster. I have an incomplete patch series for that somewhere. It's probably only a few hours of work.
I guess as a half-measure I could rip them out of `make check`, add a mach command to invoke them, and then use that in cross-builds. If you've got a patch to improve their runtime that'd be great.
bug 1255479 makes it so if you run `mach python-test` it will run all of the PYTHON_UNIT_TESTS, so we could just invoke that as a first cut. A further improvement might be to annotate those tests that really need to be run with a build objdir, and run just those tests after the build (when we currently run `make check`), and run the rest in a separate test job.
Depends on: 1255479
And for the tests that don't need an objdir (i.e mozbase), we should make sure they run on the "lint-opt" platform, and only when the package under test has actually changed. There is a new mozharness tox job that does this for mozharness that we can use as a guideline for the relevant taskcluster configs.

(Unrelated, but "lint-opt" should be called "source-only" or something similar)
And of course, some packages (like mozprocess) need windows testing.. so the above suggestion only applies to some things I guess.
Rather than moving all the python-unittests out wholesale in one patch, I've been working towards making it easy to split them out of the default subsuite and into their own taskcluster tasks. Python unittests now use manifestparser manifests which means we can use the subsuite key to accomplish this.

We are getting close, with a few caveats:
1) Windows and mac aren't quite ready for prime-time in taskcluster yet. For now we can use the manifest notation "subsuite = <subsuite>, os == 'linux'" to only split tests out on linux.

2) Resolving python tests still depends on generating a build backend. I tried fixing this in bug 1320194, but it's turning out to be a bit trickier than anticipated. Alternatively we can run configure before running the tests.. though doing this fails in the current docker images and requires some sorting out.

Here is my latest patch series in case anyone is interested:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d7c56130400de57e60c212a1a1ff09b071b06a1

Finally, because I think it is better to tackle moving the python unittests one chunk at a time, I'd like to turn this into a tracking bug, and file dependent bugs for each new "subsuite" we create. Mozbase will be the first subsuite chunked out, and I'll use bug 1003417 to track it. Once that is done, it should be trivial to split out the rest.
Depends on: 1003417
Product: Core → Firefox Build System
See Also: → 1614626
Keywords: in-triage
Priority: -- → P3

Assigning this to myself temporarily to evaluate the work required here and what we can invest.

Assignee: nobody → rstewart
Depends on: 1651824
Depends on: 1651829
Depends on: 1652097
Depends on: 1652102
Depends on: 1652149
Depends on: 1342230
Depends on: 1654602
Depends on: 1654589
Depends on: 1614626
Depends on: 1655111

As of bug 1655111 we no longer run mach python-test in this way during make check.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED

Nice work!

You need to log in before you can comment on or make changes to this bug.