1353770 - Update docker-py to docker 2.2.1 or higher

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Axel Hecht [:Pike]]

I figured I'd give it a shot to upgrade the docker python library in vct, and see if that makes me a more happy camper.

I'm getting further, but I'm stuck building hgweb.

I'm also not sure if I break other things on the way, most notably, I'm hardcoding that docker listens on localhost for the rsync stuff to work. Seems that at least docker-for-mac doesn't expose the bridge gateway IP on the host, so rsync times out.

I did look a bit on the network details, see the paste below, but nothing stood out that I could use to find out that I should be talking to localhost. Maybe that's good these days, now that the explicit docker-machine isn't a thing anymore on mac and win?

[
    {
        "Name": "bridge",
        "Id": "79ee934aad9f77a1cd0b6490b8f1a15cfd1c928cb3c862a235f1ab4cf35f7774",
        "Created": "2017-04-04T17:09:50.692019387Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Comment 1

[Axel Hecht [:Pike]]

Attachment: failure when building hgweb, no service or tool found for a couple of things

This is stdout from ./d0cker build-hgmo, failing to find a bunch of servers on hgweb.

Comment 2

[Axel Hecht [:Pike]]

Attachment: bug 1353770, update docker-py to docker 2.2.1,

The changes are mostly mechanical:

docker.Client -> docker.DockerClient
self.client -> self.client.api

I used DockerClient because there's an opportunity to streamline
a few hand-woven things. At least execute is one.

Real changes:

Hardcoding the gateway to be localhost. The gateway IP of the
bridge network isn't reachable from the host on docker-for-mac.

Use docker.utils.json_stream.json_stream, which gets around
the json.loads errors reading more than one line at a time.

Review commit: https://reviewboard.mozilla.org/r/126850/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126850/

Comment 3

[Gregory Szorc [:gps]]

glob may be interested in this patch, since I think you and him and running into similar rsync problems.

Comment 4

[Axel Hecht [:Pike]]

I flagged this patch for review, but I kinda doubt is really done, given that I still fail on stuff locally.

Also, the update to the docker python library is a bit funky, as it changed name. You probably need to explicitly uninstall docker-py and then pip install docker. No idea how to do that automagically.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8854926 [details]
bug 1353770, update docker-py to docker 2.2.1,

https://reviewboard.mozilla.org/r/126850/#review135956

Everything except the gateway change looks OK. Although I didn't give it too much scrutiny.

There are also some consumers of the raw Docker client in testing/vcttesting/hgmo.py which will presumably also need updated.

::: testing/vcttesting/docker.py:231
(Diff revision 1)
> -            networks = self.client.networks()
> +            networks = self.client.api.networks()
>              for network in networks:
>                  if network['Name'] == 'bridge':
>                      ipam = network['IPAM']
>                      try:
> -                        addr = ipam['Config'][0]['Gateway']
> +                        addr = '127.0.0.1'

I'm pretty sure we can't make this change because it breaks environments where Docker isn't running its bridge network on 127.0.0.1.

Comment 6

[Axel Hecht [:Pike]]

It'd be nice to have this.

Comment 7

[Gregory Szorc [:gps]]

I'll land patches to do the upgrade shortly.

If you are still having problems with networking after the upgrade, please open a new bug. Let's keep this bug to track the docker Python package upgrade.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/283a174220ae
testing: upgrade docker-py==1.10.6 to docker==2.0.2 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0123eb3dfe16
testing: update docker==2.0.2 to docker==2.5.1