1199037 - version-control-tools test setup make me cry

Status: NEW

(Developer Services :: General, task)

Description

[Steve Fink [:sfink] [:s:]]

So, I've been trying for a while now to get my test setup working so I can write some bzexport tests. I have been less than successful.

First, I ran into the usual "can't find any modules, nya nya" that I am slowly learning means I have to run things from a virtualenv that I find lying around, created by who knows what.

Next, I ran into "Docker is not available." (cf bug 1080955, I didn't have the same issues.)

The first reason was because apparently I need to run as root. [REQUEST] Can we get an error message that indicates that as the problem?

By the time I'd figured that out, I had upgraded my machine from Fedora 20 -> 22. Sadly, that meant that I was faced with "Docker is not available." again, but this time for a different reason: the upgrade had turned off my docker daemon. I had to systemctl enable docker; systemctl start docker. I'm less concerned with a better error message here, since I've had to track that one down a few times now. (Different machines etc.)

I had some other problem next, but I forget what.

Next it did a bunch of stuff, but eventually complained about some Ansible thing. [REQUEST] Can something check for this dependency before launching into a long-running process?

After that, it died because something needed some libpcap-ng things. [REQUEST] And that dependency too?

Then it actually managed to start building docker images. Lots of them seemed to succeed, even. Then it hung on me. Ctrl-C did not do anything to stop it. I think this is the problematic part of the output:

ERROR rbweb> TASK: [mozreview-virtualenv | populate virtualenv] **************************** 
ERROR rbweb> failed: [localhost] => {"cmd": "/venv/bin/peep install -r /venv/requirements.txt", "failed": true}
ERROR rbweb> msg: 
ERROR rbweb> :stderr: /venv/bin/python: /venv/bin/python: cannot execute binary file

I don't remember the exact command I ran to get that, but I get the same thing running |sudo ./run-tests hgext/bzexport|.

But when I first saw it, I thought "ok, maybe I can give up on getting reviewboard tests to run, and just do what's needed for bzexport tests." So I ran |./run-tests hgext/bzexport|, and it skipped everything interesting with no indication why. [REQUEST] Could it say that it's skipping things because once again, docker is unavailable?

It's also hard for me to type the commands to run tests as root. It just seems wrong. Should I be doing this inside a VM image or something? It doesn't make me feel warm and fuzzy to run a command as root that fetches tons of stuff from the network, writes all over the filesystem, and spews lots of cryptic output (including warning messages) that I don't understand.

Anyway, where I ended up was the above sudo ./run-tests line, which again tries to build some image and fails to run /venv/bin/python (understandable; I am not running this from /).

I can keep forging ahead, but I'd just like to say that create-test-environment is a big meanie and makes me cry.

Comment 1

[Steve Fink [:sfink] [:s:]]

Latest round: I presumably have whatever fixes I made for this earlier, and I updated version-control-tools. Rerunning, I'm seeing

...
Successfully installed HTML-Scrubber-0.15
bmoweb> <== Installed dependencies for .. Finishing.
154 distributions installed
hgrb> failed: [localhost] => (item=httpd,httpd-devel,mod_authnz_external,openldap-devel,python27,python-ldap,rsyslog,sudo) => {"failed": true, "item": "httpd,httpd-devel,mod_authnz_external,openldap-devel,python27,python-ldap,rsyslog,sudo"}
hgrb> msg: Error from repoquery: ['/usr/bin/repoquery', '--show-duplicates', '--plugins', '--quiet', '-q', '--disablerepo=*', '--pkgnarrow=installed', '--qf', '%{name}-%{version}-%{release}.%{arch}', 'python-ldap-2.3.10-1.el6.x86_64']: rpmdb: /var/lib/rpm/__db.002: Read-only file system
hgrb> error: db3 error(30) from dbenv->open: Read-only file system
hgrb> error: cannot open Packages index using db3 - Read-only file system (30)
hgrb> error: cannot open Packages database in /var/lib/rpm
hgrb> Error: rpmdb open failed
Obhgrb> 
hgrb> 
hgrb> FATAL: all hosts have already failed -- aborting
hgrb> 
hgrb> PLAY RECAP ******************************************************************** 
hgrb> 
hgrb> ERROR: could not create retry file. Check the value of 
hgrb> the configuration variable 'retry_files_save_path' or set 
hgrb> 'retry_files_enabled' to False to avoid this message.
hgrb> 
hgrb> localhost                  : ok=21   changed=17   unreachable=0    failed=1   
hgrb>

Comment 2

[Gregory Szorc [:gps]]

That's a very wonky error. I have no clue what to tell you other than to try deleting and recreating all your images.

If all you care about is testing BMO, you can build just the BMO docker image by running `./d0cker build bmoweb`. You can ^C the Docker building that runs as part of `create-test-environment`.

Comment 3

[Steve Fink [:sfink] [:s:]]

Ok, rerunning again made it get further, then run out of disk space (since it uses a ton for the docker stuff, I think.) Deleting some random stuff to free up space, then rerunning gives the message below. Note that I am successfully using Docker right now for some taskcluster stuff, so I know it's installed and running properly.

---

Building Docker images.
This could take a while and may consume a lot of internet bandwidth.
If you don't want Docker images, it is safe to hit CTRL+c to abort this.
Error running mach:

    ['build-all']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

  File "/home/sfink/lib/version-control-tools/testing/vcttesting/docker_mach_commands.py", line 90, in build_all
    self.d.build_all_images(verbose=True)
  File "/home/sfink/lib/version-control-tools/testing/vcttesting/docker.py", line 1211, in build_all_images
    ansibles=ansible_images, verbose=verbose, use_last=use_last)
  File "/home/sfink/lib/version-control-tools/testing/vcttesting/docker.py", line 513, in ensure_images_built
    with self.vct_container(verbose=verbose) as vct_state, futures.ThreadPoolExecutor(len(missing)) as e:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/sfink/lib/version-control-tools/testing/vcttesting/docker.py", line 1407, in vct_container
    state = self.client.inspect_container(existing_cid)
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/docker/api/container.py", line 173, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/docker/client.py", line 110, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/sfink/lib/version-control-tools/venv/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
You will not be able to run tests that require Docker.
Please see https://docs.docker.com/installation/ for how to install Docker.
When Docker is installed, re-run this script
finished creating test environment

Comment 4

[Gregory Szorc [:gps]]

It certainly appears that it can't connect to the Docker TCP server. We require a Docker TCP server (connecting to Docker via a UNIX domain socket won't work).

Comment 5

[Steve Fink [:sfink] [:s:]]

(In reply to Gregory Szorc [:gps] from comment #4)
> It certainly appears that it can't connect to the Docker TCP server. We
> require a Docker TCP server (connecting to Docker via a UNIX domain socket
> won't work).

Oh. Maybe that's the problem? I think the default docker install is unix only.

I'm attempting to run ./venv/bin/python d0cker build bm0web. It seems to be doing something; at least, there are running docker and yum processes churning away.

Comment 6

[Gregory Szorc [:gps]]

I thought we had code checking for the UNIX domain socket and failing fast. Perhaps it broke. If you can pinpoint the problem in testing/vcttesting/docker.py, I'd love to have a proper fix for that. Look at params_from_env() and the end of Docker.__init__ for the environment detection code.