1290783 - dev env broken since docker v1.12 upgrade: "starting container with HostConfig was deprecated since v1.10 and removed in v1.12"

Status: RESOLVED FIXED

(MozReview Graveyard :: Testing / Development Environment, defect)

Description

[:glob ✱]

APIError: 400 Client Error: Bad Request ("{"message":"starting container with HostConfig was deprecated since v1.10 and removed in v1.12"}")

  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/mozreview_mach_commands.py", line 90, in start
    verbose=True)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/mozreview.py", line 219, in start
    verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 989, in start_mozreview
    max_workers=max_workers, verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 807, in build_mozreview
    bmo_images = f_bmo_images.result()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 735, in build_bmo
    ], existing=images, verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 579, in ensure_images_built
    with self.vct_container(verbose=verbose) as vct_state, \
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 1583, in vct_container
    self.client.start(cid, port_bindings={873: None})
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/api/container.py", line 355, in start
    self._raise_for_status(res)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/client.py", line 142, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)

Comment 1

[:glob ✱]

the fix is to move the host_config from the start() call to create()

ie. instead of:
> cid = self.client.create_container(image, volumes=['/vct-mount'], ports=[873], labels=['vct'])['Id']
> self.client.start(cid, port_bindings={873: None})
do:
> host_cfg = self.client.create_host_config(port_bindings={873: None})
> cid = self.client.create_container(image, volumes=['/vct-mount'], ports=[873], labels=['vct'], host_config=host_cfg)['Id']
> self.client.start(cid)

we call client.start() with a host_config number of places, some trivial to fix, others require investigation.
working on it.

Comment 2

[:glob ✱]

to avoid this you need to both not upgrade docker, and specify a specific image to docker-machine when building the container.
not upgrading docker can be easy (eg. pin your current version if you're using homebrew).
docker-machine needs a driver specific switch.  for me, using parallels, i execute the following:

  docker-machine create mozreview-docker-main \
    --driver parallels \
    --parallels-cpu-count 8 \
    --parallels-memory 4096 \
    --parallels-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.11.2/boot2docker.iso

Comment 3

[:glob ✱]

i've made some progress on fixing this.  going to focus on my quarterly goal for a while then pick this up again once my goal is in better shape.

Comment 4

[Gregory Szorc [:gps]]

This was blocking me on some other work. I think I've almost got this done.

Comment 5

[Gregory Szorc [:gps]]

Attachment: docker: use user-defined network for mozreview cluster (bug 1290783);

Docker 1.12 requires things like links to be specified at container
create time. This undoes a lot of optimizations around our finely tuned
startup sequence because it requires us to have prerequisite containers
started before dependent containers are created (instead of merely
creating the dependent containers).

That's the bad news.

The good news is that user-defined networks allow you to not use links.
Instead, you create a container alias inside the user-defined network
and these aliases magically resolve via DNS.

This commit switches the MozReview cluster to use user-defined networks.
The network name is randomly generated. Containers in the network have
aliases like you would expect them to.

Review commit: https://reviewboard.mozilla.org/r/73086/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/73086/

Comment 6

[:glob ✱]

thanks for looking at this.

it looks like you've already committed a bunch of changes (without review or bug ids) that broke the dev environment:

> The details of the failure are as follows:
> 
> TypeError: 'NoneType' object has no attribute '__getitem__'
> 
>   File "version-control-tools/testing/vcttesting/mozreview_mach_commands.py", line 90, in start
>     verbose=True)
>   File "version-control-tools/testing/vcttesting/mozreview.py", line 219, in start
>     verbose=verbose)
>   File "version-control-tools/testing/vcttesting/docker.py", line 1219, in start_mozreview
>     self._get_host_hostname_port(hgrb_state, '22/tcp')
>   File "version-control-tools/testing/vcttesting/docker.py", line 1770, in _get_host_hostname_port
>     host_port = state['NetworkSettings']['Ports'][port][0]

Comment 7

[:glob ✱]

gps - can you either backout or fix this?

Comment 8

[:glob ✱]

Attachment: full log

Comment 9

[:glob ✱]

Attachment: 'state' variable at point of failure

Comment 10

[Gregory Szorc [:gps]]

I backed out the regressing commits. I'll hopefully have another go at making MozReview work with 1.12 when I have enough aroundtoits. I'll get review before landing anything that could impact MozReview development.

Comment 11

[Gregory Szorc [:gps]]

I can reproduce the "TypeError: 'NoneType' object has no attribute '__getitem__'" failure in Docker 1.11.2. This means I can test my fixes for compatibility against 1.11 and 1.12. Yay.

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/73086/diff/1-2/

Comment 13

[:glob ✱]

Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

https://reviewboard.mozilla.org/r/73086/#review72698

this looks great, and actually starts up, but all the autoland tests are failing for me with the same error:
(500, u'{\n  "error": "could not translate host name \\"db\\" to address: Name or service not known\\n"\n}')
reviewboard tests all pass.

::: testing/vcttesting/docker.py:1031
(Diff revision 2)
> +        def network_config(alias):
> +            return self.client.create_networking_config(
> +                endpoints_config={
> +                    network_name: self.client.create_endpoint_config(
> +                        aliases=[alias],
> +                    )
> +                }
> +            )

this is the exact same function as the one defined in start_bmo; it probably should be a class-level function.

Comment 14

[Gregory Szorc [:gps]]

Attachment: testing: change link name of autoland db from "db" to "autolanddb";

Making it more descriptive in the context of a larger cluster.

Review commit: https://reviewboard.mozilla.org/r/75014/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/75014/

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/73086/diff/2-3/

Comment 16

[:glob ✱]

Comment on attachment 8786039 [details]
testing: change link name of autoland db from "db" to "autolanddb";

https://reviewboard.mozilla.org/r/75014/#review73056

Comment 17

[:glob ✱]

Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

https://reviewboard.mozilla.org/r/73086/#review73062

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3af0c0fbd9ad
docker: use user-defined network for mozreview cluster ; r=glob