1598958 - Modernize the version-control-tools test runner

Status: RESOLVED FIXED

(Developer Services :: General, enhancement)

Description

[Steven MacLeod [:smacleod]]

The current test running environment is fragile and slow, we should modernize it to make it easily install-able, fast, and more reliable.

Comment 1

[Steven MacLeod [:smacleod]]

Attachment: testing: add docker-compose env for running tests (Bug 1598958).

This introduces a docker-compose.yml and a test-runner
docker image for executing the tests. This image is built
with everything needed to run the non-docker tests.

when run through docker-compose the container makes use of
the host docker daemon to build and run the full docker
tests. Currently the container also requires using host
networking so that it can access the containers through
localhost. This means only a linux host is supported for
running these tests, as docker for windows and docker for
mac os do not support this feature.

Eventually as we re-implement the docker test containers
using docker files, and switch the docker-compose networking
we'll be able to remove the requirement on host networking.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/5bfcd45f5eaa
testing: add docker-compose env for running tests . r=sheehan

Comment 3

[Steven MacLeod [:smacleod]]

Attachment: testing: add a `run` command to alias common commands (Bug 1598958)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/bc1936eae462
testing: add a `run` command to alias common commands r=mhentges

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Last week, we discussed intentions and end goals with this migration: see the Friday notes in this doc.

TL;DR: We want to put each of our hg server toolchains into their own independent docker images.
This should ease incremental building (only one image should need to be updated when a change occurs), will remain relatively identical to production, and should be possible without requiring "host networking" in docker (linux-only).

---

There will be the following docker images and containers:

• (HG SSH) pash (python 2)
  • Runs Pash (tool to absorb SSH connections and only serve mercurial-related requests)
• (HG SSH) venv_bundles (python 3)
  • Generates HG bundles and throws them on S3
• (HG SSH) tools (python 2)
  • Aggregator replicator daemon (push data aggregator) (push data aggregator-pending)
  • Notifier daemons
  • Unified Process
  • Heartbeat
• (HG Web) hgweb (python 2)
  • UWSGI and hg.m.o web application
• (HG Web) tools3 (python 3)
  • IP Scaper (x3)
• (HG Web) replication (python 2)
  • Replicate (x8) head (x1)
• (HG Web) tools2 (python 2)
  • FF Scraper
• (HG Web) Apache (not python)
• (both) Kafka (not python)

---

Work we need to do:

1. Confirm with "enterprise security" that this is a good path forward
2. Migrate custom deployment/testing scripts to use docker-compose
3. Migrate ansible setup to individual docker images and containers as shown ^

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: testing: make `create_virtualenv` support installing the global venv (Bug 1598958) r?mhentges,zeid

Make the name argument optional and allow passing a python
parameter which is intended to be the name of the Python executable
that will be used to create the virtualenv (ie if python3 is
passed, then the interpreter found on PATH with that name will
be used). We also add the HGPYTHON3 environment variable for
testing Mercurial 5.0 on Python 3.

Comment 7

[Connor Sheehan [:sheehan]]

Attachment: testing: add a python function to install cinnabar (Bug 1598958) r?mhentges,zeid

This replaces the cinnabar install related lines from
create-test-environment.

Depends on D69089

Comment 8

[Connor Sheehan [:sheehan]]

Attachment: testing: add a python function to replace create-test-environment (Bug 1598958) r?mhentges,zeid

This function performs the same function as create-test-environment.
It could be refined to directly call the Python code that underlies
./d0cker build-all, but that can be performed as a followup.

Depends on D69090

Comment 9

[Connor Sheehan [:sheehan]]

Attachment: testing: teach ./create-environment to install global venv (Bug 1598958) r?mhentges,zeid

With this change, ./create-environment test will do the same
thing as ./create-test-environment.

Depends on D69091

Comment 10

[Connor Sheehan [:sheehan]]

Attachment: testing: remove create-test-environment (Bug 1598958) r?mhentges,zeid

This is no longer needed now that ./create-environment test
exists.

Depends on D69092

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c42af2c60a84
testing: make `create_virtualenv` support installing the global venv r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/594a903bdd76
testing: add a python function to install cinnabar r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ccc2a01d1910
testing: add a python function to replace create-test-environment r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3e42a09182c4
testing: teach ./create-environment to install global venv r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6d9e410523df
testing: remove create-test-environment r=mhentges

Comment 12

[Connor Sheehan [:sheehan]]

Attachment: testing: add an `env` command to `./run` (Bug 1598958) r?zeid

To use the Docker test runner, a .env file is needed
with several values (gid/uid of the current user, gid
of the Docker group) to allow the container to access
the Docker host and Docker socket from within the container.
This commit adds a ./run env command that will output
the required contents of this file to stdout, so the file
can be easily created via ./run env > .env.

To achieve this the test runner is extended to support running
a Python function as a command instead of turning the ./run
process into a docker-compose process via os.execvp.

Comment 13

[Connor Sheehan [:sheehan]]

Attachment: testing: support passing a more convenient string to `--with-hg` (Bug 1598958) r?zeid

--with-hg expects a path to a Mercurial installation which will be
used as the hg run in t-tests. Previously when running the tests
from the on-host venv, I would use fzf's fuzzy file finding to
quickly select the version of Mercurial I wanted to run the tests
against. Now that the Docker test environment is the preferred option,
using fzf is not feasible.

This commit teaches run-tests to parse a "convenience string" to
determine which version of Mercurial to run the tests against. The
string is simply hg_version:py_version, so to run the tests against
Mercurial 5.3.2 running on Python 3, one would pass --with-hg=5.3.2:3
to the test runner.

Depends on D73079

Comment 14

[Connor Sheehan [:sheehan]]

Attachment: testing: add a `run build` convenience command (Bug 1598958) r?zeid

Add a very simple run build command to rebuild the Docker
test runner.

Depends on D73080

Comment 15

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator: run `hgmo clean` at the end of `test-mozrepohash.t` (Bug 1598958) r?zeid

So the test cluster is spun down correctly.

Comment 16

[Connor Sheehan [:sheehan]]

Attachment: mozext: update pushlog version in test-mozext.t (Bug 1598958) r?zeid

The version of Mercurial used to load pushlog in test-mozext.t
should always stay up to date with the current production version
the extension is compatible with.

Depends on D74754

Comment 17

[Connor Sheehan [:sheehan]]

Attachment: testing: return `hgmo start` exit code on failure instead of `80` (Bug 1598958) r?zeid

This causes any tests to fail with a stack trace from the cluster startup
process, which gives an indication as to what went wrong. Currently a
failure just results in the test being skipped.

Depends on D74755

Comment 18

[Connor Sheehan [:sheehan]]

Attachment: testing: restore and update `install-mercurials` hack (Bug 1598958) r?zeid

Resore the install-mercurials hack and add a new argument
specifying which version of Python to use for installs.
This is a temporary hack that can be removed once Py2 support
is dropped.

Depends on D73081

Comment 19

[Connor Sheehan [:sheehan]]

Attachment: testing: set shebang python to just `env python` (Bug 1598958) r?zeid

So we can run these scripts with Py3.

Depends on D74758

Comment 20

[Connor Sheehan [:sheehan]]

Attachment: testing: make `run-tests` forget about the various venvs (Bug 1598958) r?zeid

run-tests tries to be smart about running tests compatible with
the currently activated virtualenv. This commit removes that checking
code and assumes the global virtualenv is being used. This will cause
tests to simply fail or be skipped when run with insufficient requirements,
which should never happen in the Docker test environment.

Depends on D74759

Comment 21

[Connor Sheehan [:sheehan]]

Attachment: testing: update `run-tests.py` with upstream (Bug 1598958) r?zeid

Vendor the changes from upstream repo at release 5.4.

Disable _usecorrectpython since we know the Python used
will always be from the activate virtualenv.

Also update the TRACKING MOZ changes to be Py3 compatible.

Depends on D74760

Comment 22

[Connor Sheehan [:sheehan]]

Attachment: testing: make some in-test Python code Py3 compatible (Bug 1598958) r?zeid

Since we're executing the test runner with Python 3 these snippets
need to be updated.

Also add a warning about missing Docker libraries causing Docker to
be silently deactivated.

Depends on D74761

Comment 23

[Connor Sheehan [:sheehan]]

Attachment: testing: run `auto_clean_orphans` against all Docker tests (Bug 1598958) r?zeid

I noticed a few tests would fail and their clusters would not shut down.
Adding this line forces shutdown on all tests.

Depends on D74762

Comment 24

[Connor Sheehan [:sheehan]]

Attachment: testing: add `mock` to test requirements and update `mach` (Bug 1598958) r?zeid

mock seems to be missing from the test environment and is causing
some tests to fail with the Docker environment. mach is upgraded
to v1.0.0 for Py3 compatibility.

Depends on D74763

Comment 25

[Connor Sheehan [:sheehan]]

Attachment: testing: update test runner code to be Py3 compatible (Bug 1598958) r?zeid

This changeset includes many small changes to make the test runner
code compatible with Python 3. First we revert the @check arguments
to regular unicode strings to be compatible with the earlier run-test.py
upstream changes. We add many decode and encode calls to fix encoding
inconsistencies between Python 2/3. We change the file mode in many
operations from bytes to non-bytes for consistencies with the interfacing
libraries. Calls to a few dict.keys() are wrapped in list to avoid
issues when indexing the resulting object. All keys passed to LDAP are
converted from byte-strings to regular unicode to conform to that library's
API. Finally, we avoid failing when the Docker state file cannot be read
This should not be a problem since the new test runner looks for an image
tag instead of using the file (all out removal is a pending TODO).

Depends on D74764

Comment 26

[Connor Sheehan [:sheehan]]

Attachment: pylib/*: add Python 3 compatibility to `pylib/` (Bug 1598958) r?zeid

Make lots of changes to code under pylib/ to enable running
tests under Python 3. Most changes involve converting to byte
strings or reverting a byte-string to regular str. A patch to
paper over sorted function cmp/key function differences is
applied. We also invert some logic to avoid an isinstance call
that used basestring as the type to compare with.

Depends on D74765

Comment 27

[Connor Sheehan [:sheehan]]

Attachment: testing: create native Docker images for test environment (Bug 1598958) r?zeid

This commit converts the test runner Docker image into a Py2 only
image and a Py3 only image. We introduce an /app directory and put
the source code and virtualenv in separate directories (/app/vct and
/app/venv respectively). We set PYTHONUNBUFFERED=1 instead of
manually modifying sys.stdout and remove the NO_DOCKER=1 environment
variable. We use separate cache volumes for each new image to avoid
package problems in each environment.

After moving the virtualenv out of the /vct/venv directory, a few
hard-coded assumptions about the location of the test files need to be
updated. We do so in this commit since it is assumed that running the
tests outside of the Docker environment is no longer supported after this
change.

We also add several new commands to run that alias common operations.

Depends on D74766

Comment 28

[Connor Sheehan [:sheehan]]

Attachment: testing: make `TRACKING MOZ` changes in `run-tests.py` Py3 compatible (Bug 1598958) r?zeid

Disable _usecorrectpython since we know the Python used
will always be from the activate virtualenv.

Also update the TRACKING MOZ changes to be Py3 compatible.

Depends on D74761

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d973a083cfec
vcsreplicator: run `hgmo clean` at the end of `test-mozrepohash.t` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ae41b9b8f1ed
mozext: update pushlog version in test-mozext.t r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/91a1662e6031
testing: return `hgmo start` exit code on failure instead of `80` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e86de3e83035
testing: add an `env` command to `./run` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8c43d34a8f90
testing: support passing a more convenient string to `--with-hg` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/54d72a8a8096
testing: add a `run build` convenience command r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/64f248c81ee3
testing: restore and update `install-mercurials` hack r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/525d21339c70
testing: set shebang python to just `env python` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8e10a17ce895
testing: make `run-tests` forget about the various venvs r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a8821b06117e
testing: update `run-tests.py` with upstream r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7a3a3ff053ce
testing: make `TRACKING MOZ` changes in `run-tests.py` Py3 compatible r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e6c4fa0384ba
testing: make some in-test Python code Py3 compatible r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/adf1af047d81
testing: run `auto_clean_orphans` against all Docker tests r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4e63283159bd
testing: add `mock` to test requirements and update `mach` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4f1f55c7d638
testing: update test runner code to be Py3 compatible r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/fc52de3babe0
pylib/*: add Python 3 compatibility to `pylib/` r=zeid
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d8c862917720
testing: create native Docker images for test environment r=zeid

Comment 30

[Connor Sheehan [:sheehan]]

Attachment: testing: remove `venv` cache in testing Docker image (Bug 1598958) r?zeid

After landing the native Docker test image patches I attempted to
rebuild the images to install Mercurial 5.4 instead of 5.4rc0.
I noticed the old release candidate build persisted despite the
updated version in the install_mercurials function. Since the
image virtualenv is saved as a Docker volume in docker-compose,
changes made after rebuilding are essentially ignored when re-running
the tests.

This commit removes the volume, since the virtualenv should be
considered part of the Docker image itself, and not something that
should persist between image rebuilds.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9182861f3c88
testing: remove `venv` cache in testing Docker image r=zeid

Comment 32

[Connor Sheehan [:sheehan]]

Attachment: testing: implement Docker-label based container cleanup (Bug 1598958) r?zeid

After running tests using the new Docker based test suite, I noticed
the docker-compose based shutdown sometimes failed to cleanup containers
when tests did not complete successfully. This commit implements a better
shutdown system by adding a Docker label with a UUID generated by run-tests
and shutting down all running containers and networks with a matching label.

The previous implementation of the orphan container cleanup (from b2f42c234e8a)
did something similar but was less intelligent about which containers to
clean up, and ran the shutdown command using futures. This implementation
does everything serially, until we notice enough of a performance hit to
implement parallel execution.

Comment 33

[Connor Sheehan [:sheehan]]

Attachment: run-tests: add `int_floor` function and make `-j` guessing use it (Bug 1598958) r?zeid

Since I was only running 1 test, I ran ./run tests without using
-j and hit an error since the returned value of 14.0 is not an
integer.

This commit fixes -j by introducing an int_floor function that
returns the integer floor of a number (ie 14 when 14.5 is passed)
and uses the function on the calculated number of concurrent jobs
to be run when -j isn't passed to the test runner.

Depends on D76007

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8509f33e4843
testing: implement Docker-label based container cleanup r=zeid

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b96a4a8a774a
run-tests: add `int_floor` function and make `-j` guessing use it r=zeid

Comment 36

[Connor Sheehan [:sheehan]]

After the work done in this bug, the test environment:

• is fully Dockerized, as the tests run in a Docker container that has a mount of the developers working directory
• uses docker-compose to execute various commands such as running tests, opening a shell, starting up a test hg cluster, etc
• uses docker-compose to startup and teardown the test cluster (instead of the custom code that made calls to the Docker API directly)
• runs on Python 3, instead of Python 2 with a Py3 Mercurial loaded into it

There are a million more changes we could make (looking at you, Ansible-to-Docker image building code...), but most of our goals have been achieved and we're in a much better state. Closing this out and we can open smaller bugs for any more changes.