1363509 - Add support for running BMO Docker container to hgdev environment

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Gregory Szorc [:gps]]

Bug 1360062 tracked implementing a new, minimal development environment for Mercurial extensions and hooks. The environment purposefully didn't support Docker because we were aiming for simplicity by default.

A handful of Mercurial extensions do rely on Docker for testing. With the exception of MozReview and hg.mo-specific extensions, most Docker dependencies are on the bmoweb container. So let's add optional support for creating an hgdev environment with support for running the bmoweb Docker container.

Comment 1

[Gregory Szorc [:gps]]

Attachment: mozautomation: move bugzilla xmlrpc transports code from mozreview (bug 1363509);

vcttesting.bugzilla needs this code. Since it isn't unique to MozReview
and we don't want to create the dependency mess to import the mozreview
Python package in minimal virtualenvs, let's just move the code to
somewhere that's always available.

The mozreview package already imports from mozautomation, so this
shouldn't need any ops changes.

Review commit: https://reviewboard.mozilla.org/r/137720/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137720/

Comment 2

[Gregory Szorc [:gps]]

Attachment: testing: consistently name bmoweb environment variable (bug 1363509);

Environment variables are used by the test harness to pass the Docker
image IDs to tests. The environment variable name is typically
DOCKER_*_IMAGE where the wildcard is the name of the directory under
testing/docker. This naming was inconsistent for the bmoweb image,
so this commit fixes it.

Review commit: https://reviewboard.mozilla.org/r/137722/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137722/

Comment 3

[Gregory Szorc [:gps]]

Attachment: testing: look for Docker image environment variables (bug 1363509);

hghave is used by the Mercurial test runner to determine what
capabilities are available and whether to skip tests or parts of
tests.

Currently, our capabilities for Docker support (bmodocker, hgmodocker,
and mozreviewdocker) simply return whether Docker is available.
Our test runner - run-tests - differentiates between the various
capability strings and ensures that required Docker images are
built *before* invoking the Mercurial test harness. Our test harness
also exports environment variables declaring the Docker image IDs
to use. This prevents e.g. `d0cker` invocations in .t tests from
having to rebuild the images (which takes a long time).

This commit updates the hghave checks to look for the Docker
image environment variables actually used by tests. In theory,
behavior should be identical to before. However, checking for
these environment variables in hghave has advantages.

First, if we forget to populate a Docker image in run-tests, the
test would generate the image and this would add a ton of overhead
to each test. (This is why we generate images in run-tests). So
checking for the environment variable in hghave will ensure the test
is skipped and help flush out bugs in image population.

Second, upcoming changes will allow our new, leaner development
environments to support a subset of Docker. By looking for the
exact Docker image requirements in hghave, we can skip tests
requiring Docker images not "available" to the current environment.

Review commit: https://reviewboard.mozilla.org/r/137724/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137724/

Comment 4

[Gregory Szorc [:gps]]

Attachment: testing: delay import modules to facilitate wider use (bug 1363509);

We now have smaller, leaner development environments. Some
environments/virtualenvs don't have all the bells and whistles
like support for Docker, Kafka, SSH, etc.

In an upcoming commit, I'll be teaching the new hgdev environment
to have *optional* support for Docker. For this, I'll need to
import vcttesting.docker. However, this module imports util.py
which imports tons of random modules used for various service
tests. These modules are related to functionality unused by the
hgdev environment. Since we are striving for minimalism in our
new environments, we don't want to take on the Python package
dependencies.

This commit moves importing of uncommon modules to the functions
that use them so as to not require extra dependencies to import
util.py.

Review commit: https://reviewboard.mozilla.org/r/137726/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137726/

Comment 5

[Gregory Szorc [:gps]]

Attachment: testing: utility function to find path to Mercurial (bug 1363509);

Without this, this code attempts to look for `hg` in the global
virtualenv. This path may not exist if the global virtualenv isn't
being used. We use the `hg` from the activated virtualenv first then
fall back to the global virtualenv.

This may still fail in some cases. But it is better than it was before.

Review commit: https://reviewboard.mozilla.org/r/137728/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137728/

Comment 6

[Gregory Szorc [:gps]]

Attachment: testing: convert argument parser to sub-parsers (bug 1363509);

This will allow us to add per-environment arguments to control
behavior.

Review commit: https://reviewboard.mozilla.org/r/137730/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137730/

Comment 7

[Gregory Szorc [:gps]]

Attachment: testing: refactor code for determining docker requirements (bug 1363509);

The code for parsing test files to determine Docker requirements isn't
very reusable or extensible. This commit changes that.

Functionality should be unchaged. Only some APIs for functions
with single call sites changed. The code will be further refactored
in a future commit, so don't get too attached to it.

Review commit: https://reviewboard.mozilla.org/r/137732/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137732/

Comment 8

[Gregory Szorc [:gps]]

Attachment: testing: filter docker requirements by what's allowed by virtualenv (bug 1363509);

Previously, get_docker_state() would attempt to build all Docker images
required by the set of passed tests. This behavior was fine with a
monolithic test environment. But it is no longer appropriate for
different virtualenvs.

The hgdev virtualenv for example pulls in tests that require BMO,
MozReview, and hg.mo Docker images. However, we don't (yet) wish
to support all these Docker environments in that dev environment.

We add a dict mapping environment name to a set of allowed Docker
requirements. Then we change the function that populates Docker state
to filter the Docker requirements as specified by tests against
what is allowed by the virtualenv. For example, the hgdev virtualenv
will now only build the bmoweb Docker image.

Review commit: https://reviewboard.mozilla.org/r/137734/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137734/

Comment 9

[Gregory Szorc [:gps]]

Attachment: testing: support BMO Docker container in hgdev environment (bug 1363509);

This commit adds the --docker-bmo flag to `create-environment hgdev`.
The flag will enable Docker support in the environment by installing
Python packages required by Docker. It will also trigger generation
of the bmoweb Docker image.

The pip requirements file also has a number of extra packages. These
are required by various tests that get unskipped as part of adding
Docker support to the virtualenv.

$ ./run-tests -j8
no docker: 35.9s wall; Ran 112 tests, 66 skipped, 0 warned, 0 failed
w/ docker: 48.0s wall; Ran 118 tests, 60 skipped, 0 warned, 0 failed

The newly-runable tests are everything with "#require bmodocker", which
is a handful of tests for the bzexport, bzpost, and qimportbz extensions.

Review commit: https://reviewboard.mozilla.org/r/137736/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137736/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8866119 [details]
mozautomation: move bugzilla xmlrpc transports code from mozreview (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137720/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8866120 [details]
testing: consistently name bmoweb environment variable (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137722/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8866121 [details]
testing: look for Docker image environment variables (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137724/diff/1-2/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8866122 [details]
testing: delay import modules to facilitate wider use (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137726/diff/1-2/

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8866123 [details]
testing: utility function to find path to Mercurial (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137728/diff/1-2/

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8866124 [details]
testing: convert argument parser to sub-parsers (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137730/diff/1-2/

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8866125 [details]
testing: refactor code for determining docker requirements (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137732/diff/1-2/

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8866126 [details]
testing: filter docker requirements by what's allowed by virtualenv (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137734/diff/1-2/

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8866127 [details]
testing: support BMO Docker container in hgdev environment (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137736/diff/1-2/

Comment 19

[Gregory Szorc [:gps]]

Comment on attachment 8866119 [details]
mozautomation: move bugzilla xmlrpc transports code from mozreview (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137720/diff/2-3/

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 8866120 [details]
testing: consistently name bmoweb environment variable (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137722/diff/2-3/

Comment 21

[Gregory Szorc [:gps]]

Comment on attachment 8866121 [details]
testing: look for Docker image environment variables (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137724/diff/2-3/

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8866122 [details]
testing: delay import modules to facilitate wider use (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137726/diff/2-3/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8866123 [details]
testing: utility function to find path to Mercurial (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137728/diff/2-3/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8866124 [details]
testing: convert argument parser to sub-parsers (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137730/diff/2-3/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8866125 [details]
testing: refactor code for determining docker requirements (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137732/diff/2-3/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8866126 [details]
testing: filter docker requirements by what's allowed by virtualenv (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137734/diff/2-3/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8866127 [details]
testing: support BMO Docker container in hgdev environment (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137736/diff/2-3/

Comment 28

[:glob ✱]

Comment on attachment 8866119 [details]
mozautomation: move bugzilla xmlrpc transports code from mozreview (bug 1363509);

https://reviewboard.mozilla.org/r/137720/#review152706

Comment 29

[:glob ✱]

Comment on attachment 8866120 [details]
testing: consistently name bmoweb environment variable (bug 1363509);

https://reviewboard.mozilla.org/r/137722/#review152710

Comment 30

[:glob ✱]

Comment on attachment 8866121 [details]
testing: look for Docker image environment variables (bug 1363509);

https://reviewboard.mozilla.org/r/137724/#review152712

Comment 31

[:glob ✱]

Comment on attachment 8866122 [details]
testing: delay import modules to facilitate wider use (bug 1363509);

https://reviewboard.mozilla.org/r/137726/#review152718

Comment 32

[:glob ✱]

Comment on attachment 8866123 [details]
testing: utility function to find path to Mercurial (bug 1363509);

https://reviewboard.mozilla.org/r/137728/#review152720

::: testing/vcttesting/vctutil.py:16
(Diff revision 3)
>  
>  HERE = os.path.abspath(os.path.dirname(__file__))
>  ROOT = os.path.normpath(os.path.join(HERE, '..', '..'))
>  
>  
> +def hg_path():

hg_path implies it returns a path; however this returns the hg executable; perhaps `hg_executable` or `hg_bin` would be clearer.

Comment 33

[:glob ✱]

Comment on attachment 8866124 [details]
testing: convert argument parser to sub-parsers (bug 1363509);

https://reviewboard.mozilla.org/r/137730/#review152722

Comment 34

[:glob ✱]

Comment on attachment 8866125 [details]
testing: refactor code for determining docker requirements (bug 1363509);

https://reviewboard.mozilla.org/r/137732/#review152742

Comment 35

[:glob ✱]

Comment on attachment 8866125 [details]
testing: refactor code for determining docker requirements (bug 1363509);

https://reviewboard.mozilla.org/r/137732/#review152744

Comment 36

[:glob ✱]

Comment on attachment 8866126 [details]
testing: filter docker requirements by what's allowed by virtualenv (bug 1363509);

https://reviewboard.mozilla.org/r/137734/#review152750

Comment 37

[:glob ✱]

Comment on attachment 8866127 [details]
testing: support BMO Docker container in hgdev environment (bug 1363509);

https://reviewboard.mozilla.org/r/137736/#review152752

::: testing/vcttesting/environment.py:176
(Diff revision 3)
> +    from .docker import (
> +        Docker,
> +        params_from_env,
> +    )
> +
> +    state_path = os.path.join(ROOT, '.dockerstate')

.dockerstate is a file, so this should be `state_file`

Comment 38

[:glob ✱]

Comment on attachment 8866123 [details]
testing: utility function to find path to Mercurial (bug 1363509);

https://reviewboard.mozilla.org/r/137728/#review152756

Comment 39

[Gregory Szorc [:gps]]

Comment on attachment 8866119 [details]
mozautomation: move bugzilla xmlrpc transports code from mozreview (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137720/diff/3-4/

Comment 40

[Gregory Szorc [:gps]]

Comment on attachment 8866120 [details]
testing: consistently name bmoweb environment variable (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137722/diff/3-4/

Comment 41

[Gregory Szorc [:gps]]

Comment on attachment 8866121 [details]
testing: look for Docker image environment variables (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137724/diff/3-4/

Comment 42

[Gregory Szorc [:gps]]

Comment on attachment 8866122 [details]
testing: delay import modules to facilitate wider use (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137726/diff/3-4/

Comment 43

[Gregory Szorc [:gps]]

Comment on attachment 8866123 [details]
testing: utility function to find path to Mercurial (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137728/diff/3-4/

Comment 44

[Gregory Szorc [:gps]]

Comment on attachment 8866124 [details]
testing: convert argument parser to sub-parsers (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137730/diff/3-4/

Comment 45

[Gregory Szorc [:gps]]

Comment on attachment 8866125 [details]
testing: refactor code for determining docker requirements (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137732/diff/3-4/

Comment 46

[Gregory Szorc [:gps]]

Comment on attachment 8866126 [details]
testing: filter docker requirements by what's allowed by virtualenv (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137734/diff/3-4/

Comment 47

[Gregory Szorc [:gps]]

Comment on attachment 8866127 [details]
testing: support BMO Docker container in hgdev environment (bug 1363509);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137736/diff/3-4/

Comment 48

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/75da877a54b8
mozautomation: move bugzilla xmlrpc transports code from mozreview ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/acd6b2e56b9f
testing: consistently name bmoweb environment variable ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4868036bfc07
testing: look for Docker image environment variables ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/99b79d6cd116
testing: delay import modules to facilitate wider use ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/08110cd74a21
testing: utility function to find path to Mercurial ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f641f9776c17
testing: convert argument parser to sub-parsers ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e18b398b929f
testing: refactor code for determining docker requirements ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/360d6eeed6e9
testing: filter docker requirements by what's allowed by virtualenv ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a89ed437ebe3
testing: support BMO Docker container in hgdev environment ; r=glob