1322313 - create-test-environment failed with ValueError in json.loads(stream)

Status: UNCONFIRMED

(Developer Services :: General, task)

Description

[Saurabh Singhal [:saurabhs]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161129173726

Steps to reproduce:

MacBook-Pro:version-control-tools saurabhs$ ./create-test-environment


Actual results:

Error running mach:

    ['build-all']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ValueError: Extra data: line 2 column 1 - line 3 column 1 (char 40 - 122)

  File "/Users/saurabhs/.mozbuild/version-control-tools/testing/vcttesting/docker_mach_commands.py", line 96, in build_all
    self.d.build_all_images(verbose=True, max_workers=forks)
  File "/Users/saurabhs/.mozbuild/version-control-tools/testing/vcttesting/docker.py", line 1441, in build_all_images
    use_last=use_last)
  File "/Users/saurabhs/.mozbuild/version-control-tools/testing/vcttesting/docker.py", line 626, in ensure_images_built
    with self.vct_container(verbose=verbose) as vct_state, \
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/saurabhs/.mozbuild/version-control-tools/testing/vcttesting/docker.py", line 1625, in vct_container
    image = self.ensure_built('vct', verbose=verbose)
  File "/Users/saurabhs/.mozbuild/version-control-tools/testing/vcttesting/docker.py", line 521, in ensure_built
    s = json.loads(stream)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
You will not be able to run tests that require Docker.
Please see https://docs.docker.com/installation/ for how to install Docker.
When Docker is installed, re-run this script


Expected results:

json.loads(data) breaks when data contains more than one dictionary.

I printed the value of the stream variable in the loop:

Building Docker images.
This could take a while and may consume a lot of internet bandwidth.
If you don't want Docker images, it is safe to hit CTRL+c to abort this.
stream= {"stream":"Step 1 : FROM sha256:78d6bff1a66903b976fecc9f5a806081fb84ceab7ff8a7275930abd62d3b1d53\n"}

vct> Step 1 : FROM sha256:78d6bff1a66903b976fecc9f5a806081fb84ceab7ff8a7275930abd62d3b1d53
stream= {"stream":" ---\u003e 78d6bff1a669\n"}
{"stream":"Step 2 : RUN apt-get update \u0026\u0026 apt-get install -y rsync\n"}

We clearly have two dictionaries here, and that's why ValueError: Extra data...

Comment 1

[Josh Matthews [:jdm]]

gps said that they're a good person to consult here.

Comment 2

[Saurabh Singhal [:saurabhs]]

Hi,

Do we know what needs to be done here? I am willing to take it up if someone tells me what to do.

Comment 3

[Gregory Szorc [:gps]]

This is a somewhat weird error.

The line in question from docker.py (`s = json.loads(stream)`) assumes that every chunk emitted from self.client.build() is JSON. But according to your debug output, a chunk has 2 JSON payloads separated by a newline.

It looks like this is a bug in the docker-py Python package: https://github.com/docker/docker-py/issues/1059

So the solution to this is to update docker-py to 1.10+. Should be a simple patch to test-requirements.txt. Note that updating docker-py has historically been a bit painful. Hopefully it "just works."

Comment 4

[Saurabh Singhal [:saurabhs]]

Attachment: upgradeDockerPy.patch

This patch fixes the JSON parsing issue. However, I am facing an rsync issue now. Looking into that.

Comment 5

[Saurabh Singhal [:saurabhs]]

I am still seeing the JSON parsing issue with docker-py 1.10.6. According to their github repo, the issue was fixed in 1.10.0.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8820984 [details] [diff] [review]
upgradeDockerPy.patch

Review of attachment 8820984 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM. I'll check this in for you.

Comment 7

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/47d00a7662a575ae5632ba9e45e3eef1ce614de2
testing: upgrade docker-py to 1.10.6 (bug 1322313); r=gps

Comment 8

[Saurabh Singhal [:saurabhs]]

Thanks Gregory.

Comment 9

[Saurabh Singhal [:saurabhs]]

I am hitting this bug even after upgrading docker-py. What should I do?

Comment 10

[Gregory Szorc [:gps]]

Hmmm.

Can you reproduce your investigation from the initial comment? Specifically, I'm curious if the raw values being iterated over still have multiple dicts. Also, you may want to catch that ValueError during json.loads(stream) in docker.py and print out the value it is choking on.