1048884 - [Tracking] Tests for the test harnesses

Status: RESOLVED DUPLICATE of bug 1229578

(Testing :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

We've been saying we need tests for our mach_commands.py files for quite some time now. I spent a bit of time thinking about how we might be able to mock out our test harnesses to accomplish this, and realized while it would be possible, it wouldn't be enough. I think we should still have tests for the mach commands, but they should be limited to verifying only the inputs and outputs of the specific command and not say anything about the correctness of the test harness itself.

Instead, I think each harness should have its own comprehensive suite of tests. Each time something breaks for a developer locally, we should add a new test along with the fix to the harness. Maybe it would make sense for a new mozbase module designed for testing (mozbase based) test harnesses. Or maybe that is overkill and just using unittest is good enough.

Obviously this is a pretty massive project, but figured we could at least get some bugs on file and discussions started.

Comment 1

[(not currently active) Ted Mielczarek]

As I've said elsewhere, the current state of the art we have here is the xpcshell harness' selftest.py:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/selftest.py

I'd be pretty happy if we had an equivalent for our other harnesses.

Comment 2

[Andrew Halberstadt [:ahal]]

Duping this, as a new bug was filed with better defined requirements in the description.