dev env broken since docker v1.12 upgrade: "starting container with HostConfig was deprecated since v1.10 and removed in v1.12"

RESOLVED FIXED

Status

--
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: glob, Assigned: gps)

Tracking

Production

Details

Attachments

(4 attachments)

(Reporter)

Description

2 years ago
APIError: 400 Client Error: Bad Request ("{"message":"starting container with HostConfig was deprecated since v1.10 and removed in v1.12"}")

  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/mozreview_mach_commands.py", line 90, in start
    verbose=True)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/mozreview.py", line 219, in start
    verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 989, in start_mozreview
    max_workers=max_workers, verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 807, in build_mozreview
    bmo_images = f_bmo_images.result()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 735, in build_bmo
    ], existing=images, verbose=verbose)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 579, in ensure_images_built
    with self.vct_container(verbose=verbose) as vct_state, \
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/testing/vcttesting/docker.py", line 1583, in vct_container
    self.client.start(cid, port_bindings={873: None})
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/api/container.py", line 355, in start
    self._raise_for_status(res)
  File "/Users/byron/dev/mozreview/_reviews/version-control-tools/venv/lib/python2.7/site-packages/docker/client.py", line 142, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
(Reporter)

Comment 1

2 years ago
the fix is to move the host_config from the start() call to create()

ie. instead of:
> cid = self.client.create_container(image, volumes=['/vct-mount'], ports=[873], labels=['vct'])['Id']
> self.client.start(cid, port_bindings={873: None})
do:
> host_cfg = self.client.create_host_config(port_bindings={873: None})
> cid = self.client.create_container(image, volumes=['/vct-mount'], ports=[873], labels=['vct'], host_config=host_cfg)['Id']
> self.client.start(cid)

we call client.start() with a host_config number of places, some trivial to fix, others require investigation.
working on it.
(Reporter)

Comment 2

2 years ago
to avoid this you need to both not upgrade docker, and specify a specific image to docker-machine when building the container.
not upgrading docker can be easy (eg. pin your current version if you're using homebrew).
docker-machine needs a driver specific switch.  for me, using parallels, i execute the following:

  docker-machine create mozreview-docker-main \
    --driver parallels \
    --parallels-cpu-count 8 \
    --parallels-memory 4096 \
    --parallels-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.11.2/boot2docker.iso
(Reporter)

Comment 3

2 years ago
i've made some progress on fixing this.  going to focus on my quarterly goal for a while then pick this up again once my goal is in better shape.
(Assignee)

Comment 4

2 years ago
This was blocking me on some other work. I think I've almost got this done.
Assignee: glob → gps
Status: NEW → ASSIGNED
Comment hidden (mozreview-request)
(Reporter)

Comment 6

2 years ago
thanks for looking at this.

it looks like you've already committed a bunch of changes (without review or bug ids) that broke the dev environment:

> The details of the failure are as follows:
> 
> TypeError: 'NoneType' object has no attribute '__getitem__'
> 
>   File "version-control-tools/testing/vcttesting/mozreview_mach_commands.py", line 90, in start
>     verbose=True)
>   File "version-control-tools/testing/vcttesting/mozreview.py", line 219, in start
>     verbose=verbose)
>   File "version-control-tools/testing/vcttesting/docker.py", line 1219, in start_mozreview
>     self._get_host_hostname_port(hgrb_state, '22/tcp')
>   File "version-control-tools/testing/vcttesting/docker.py", line 1770, in _get_host_hostname_port
>     host_port = state['NetworkSettings']['Ports'][port][0]
(Reporter)

Comment 7

2 years ago
gps - can you either backout or fix this?
Flags: needinfo?(gps)
(Reporter)

Comment 8

2 years ago
Created attachment 8783806 [details]
full log
(Reporter)

Comment 9

2 years ago
Created attachment 8783827 [details]
'state' variable at point of failure
(Assignee)

Comment 10

2 years ago
I backed out the regressing commits. I'll hopefully have another go at making MozReview work with 1.12 when I have enough aroundtoits. I'll get review before landing anything that could impact MozReview development.
Flags: needinfo?(gps)
(Assignee)

Comment 11

2 years ago
I can reproduce the "TypeError: 'NoneType' object has no attribute '__getitem__'" failure in Docker 1.11.2. This means I can test my fixes for compatibility against 1.11 and 1.12. Yay.
Comment hidden (mozreview-request)
(Reporter)

Comment 13

2 years ago
mozreview-review
Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

https://reviewboard.mozilla.org/r/73086/#review72698

this looks great, and actually starts up, but all the autoland tests are failing for me with the same error:
(500, u'{\n  "error": "could not translate host name \\"db\\" to address: Name or service not known\\n"\n}')
reviewboard tests all pass.

::: testing/vcttesting/docker.py:1031
(Diff revision 2)
> +        def network_config(alias):
> +            return self.client.create_networking_config(
> +                endpoints_config={
> +                    network_name: self.client.create_endpoint_config(
> +                        aliases=[alias],
> +                    )
> +                }
> +            )

this is the exact same function as the one defined in start_bmo; it probably should be a class-level function.
Attachment #8783136 - Flags: review?(glob)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Reporter)

Comment 16

2 years ago
mozreview-review
Comment on attachment 8786039 [details]
testing: change link name of autoland db from "db" to "autolanddb";

https://reviewboard.mozilla.org/r/75014/#review73056
Attachment #8786039 - Flags: review?(glob) → review+
(Reporter)

Comment 17

2 years ago
mozreview-review
Comment on attachment 8783136 [details]
docker: use user-defined network for mozreview cluster (bug 1290783);

https://reviewboard.mozilla.org/r/73086/#review73062
Attachment #8783136 - Flags: review?(glob) → review+

Comment 18

2 years ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3af0c0fbd9ad
docker: use user-defined network for mozreview cluster ; r=glob
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.