1616302 - TypeError: the JSON object must be str, not 'bytes' during bootstrap

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

Pulled latest gecko via cinnabar after ~7 months and doing a ./mach bootstrap produces this error (on Linux):

Error running mach:

    ['artifact', 'toolchain', '--from-build', 'linux64-node']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: the JSON object must be str, not 'bytes'

  File "/home/kats/zspace/gecko-wr/python/mozbuild/mozbuild/artifact_commands.py", line 325, in artifact_toolchain
    record = ArtifactRecord(task_id, artifact_name)
  File "/home/kats/zspace/gecko-wr/python/mozbuild/mozbuild/artifact_commands.py", line 252, in __init__
    data = json.loads(cot.content)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/bin/python3', '/home/kats/zspace/gecko-wr/mach', 'artifact', 'toolchain', '--from-build', 'linux64-node']' returned non-zero exit status 1

  File "/home/kats/zspace/gecko-wr/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
    bootstrapper.bootstrap()
  File "/home/kats/zspace/gecko-wr/python/mozboot/mozboot/bootstrap.py", line 522, in bootstrap
    checkout_root)
  File "/home/kats/zspace/gecko-wr/python/mozboot/mozboot/bootstrap.py", line 380, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "/home/kats/zspace/gecko-wr/python/mozboot/mozboot/linux_common.py", line 91, in ensure_node_packages
    self.install_toolchain_artifact(state_dir, checkout_root, node.LINUX)
  File "/home/kats/zspace/gecko-wr/python/mozboot/mozboot/base.py", line 346, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I was able to run bootstrap successfully on my mac, so there's probably something specific in my Linux env that's causing this.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Worked around this locally by modifying the offending line to use str(cot.content, "utf-8") instead of cot.content.

Comment 3

[Ricky Stewart]

Attachment: Bug 1616302 - Pass in text_type to json.loads() rather than binary_type in bootstrap

Comment 4

[hgallagher]

I just got this exact same problem on Linux Mint 18.3. I think it's because the default Python version is 3.5 on Ubuntu et al. Python 3.6+ automatically deals with this [1] but if you change the default Python3 version from 3.5 you can break a lot of things up to and including the terminal itself! [2]

[1] https://docs.python.org/3/whatsnew/3.6.html#json
[2] https://jcutrer.com/linux/upgrade-python37-ubuntu1810

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/087d8a6abf33
Pass in text_type to json.loads() rather than binary_type in bootstrap r=firefox-build-system-reviewers,mshal

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/087d8a6abf33