1265423 - [tracking] Stand-up MozReview CI

Status: RESOLVED FIXED

(MozReview Graveyard :: Testing / Development Environment, defect)

Description

[Robert Wood [:rwood]]

Tracking bug for standing-up MozReview test automation.

Comment 1

[Robert Wood [:rwood]]

Update:

The hgext/reviewboard/tests/ are running after each push to the version-control-tools repo, via jenkins running on an AWS Ubuntu 16.04 instance. The entire mozreview test environment is rebuilt from scratch on each test run. The jenkins job (and creating the test env) has been running quite stable for the past week.

As for the tests themselves there are failures (some intermittent concerning the test mach commands timing out), that need to be investigated (see the 'depends on' bugs). They may be an indication of some tweaking or further setup required in the AWS instance perhaps.

I am going to make an AWS AMI of the mozreview-ci AWS instance, primarily to serve as a backup but also down the road this opens up the possibility of being able to spin up MozReview test/dev environments in the cloud on demand.

A VCT repo has been created by :emorley for us on treeherder: https://treeherder.allizom.org/#/jobs?repo=version-control-tools

I have further work to do concerning processing the test results and publishing to treeherder. Currently I am seeing an error 500 (Bug 1270167) this may be my submission code or perhaps something on the treeherder side. The ultimate plan is when a test fails, have a link on treeherder to the diff of the test-*.t and test-*.err, so that it is easy to see why the test is failing.

Comment 2

[Gregory Szorc [:gps]]

./run-tests is partially a glorified proxy to pylib/mercurial-support/run-tests.py. That test harness can emit xunit XML and a JSON report file.

I know Jenkins could consume the xunit XML directly. Shoehorning things into treeherder will almost certainly require some custom Python.

Also, I'd absolutely love to test everything in version-control-tools, not just hgext/reviewboard/tests.

Comment 3

[Robert Wood [:rwood]]

MozReviewCI is up and running. Two jenkins jobs running on AWS, reporting to treeherder staging ('nd', and 'rb') on a per-commit basis to the version-control-tools repo. Tests have intermittent failures which are tracked separately (bug 1277566).