Open Bug 1197106 Opened 8 years ago Updated 5 months ago

Add some non regression tests for mozregression

Categories

(Testing :: mozregression, defect)

Product:
Testing
Component:
mozregression
Type:
defect

Tracking

(firefox43 affected)

Tracking Flags:
Tracking Status
firefox43 --- affected

People

(Reporter: jonathan.pigree, Unassigned)

Details

The safety net now are the unit tests. They are great, especially for developers but they only validate functions.

We should at least have some non regression tests which validates mozregression behaviour. We can begin with some very few basic cases. But it will be less cases to tests each time we merge a feature. And it will be easier to see major bugs.

The idea of those tests is to launch mozregression with arguments and test files (--persist to speed up the process) and verify its outputs 
(stdout/stderr/exit code).

The hard points I can see are:
- how to pass verdicts to mozregression easily
- how to identify and close any applications mozregression just launched easily

As Julien rightfully pointed out, these points can be addressed by the --command.
These tests will also ease testing for newcomers and is a way to document common use cases for mozregression so I think it is a good idea.
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.