1605749 - ImportError: No module named livereload

Status: RESOLVED DUPLICATE of bug 1563797

(Firefox Build System :: General, defect)

Description

[Ethan Glasser-Camp (:glasserc)]

I'm trying to use ./mach doc. I'm also running on NixOS, which might be making this more complicated than normal.

> env SOURCE_DATE_EPOCH=315532800 python2.7 ./mach doc
New python executable in /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
Also creating executable in /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
Installing setuptools, pip, wheel...
done.
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil

Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing ./third_party/python/certifi
Building wheels for collected packages: certifi
  Building wheel for certifi (setup.py) ... done
  Created wheel for certifi: filename=certifi-2018.4.16-py2.py3-none-any.whl size=148661 sha256=f39065233a1efbcd663e830eea9b32ab23f608676d6e47400a156289f7d0e6d5
  Stored in directory: /run/user/1000/pip-ephem-wheel-cache-GL5B6_/wheels/f9/fd/39/c1f2cd90ea204ca49a6d69abd957f9657d28e6287704e841a3
Successfully built certifi
Installing collected packages: certifi
  Found existing installation: certifi 2019.6.16
    Not uninstalling certifi at /nix/store/6b4cya265hbf8vvha6qjac3n713ar1lz-python2.7-certifi-2019.6.16/lib/python2.7/site-packages, outside environment /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/init
    Can't uninstall 'certifi'. No files were found to uninstall.
Successfully installed certifi-2018.4.16
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing ./third_party/python/pipenv
Building wheels for collected packages: pipenv
  Building wheel for pipenv (setup.py) ... done
  Created wheel for pipenv: filename=pipenv-2018.5.18-cp27-none-any.whl size=6353724 sha256=7170b6898e08598ef8b0d8de0aaf15def8f4a33617a18ecb43002e80518284c9
  Stored in directory: /run/user/1000/pip-ephem-wheel-cache-4pQLd5/wheels/65/c0/dd/727d0cfab3f78be8f5c095da3156710932f7207aa79c1b5f65
Successfully built pipenv
Installing collected packages: pipenv
Successfully installed pipenv-2018.5.18
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing ./third_party/python/six
Building wheels for collected packages: six
  Building wheel for six (setup.py) ... done
  Created wheel for six: filename=six-1.13.0-py2.py3-none-any.whl size=10747 sha256=fe2fec7dcfde1e651deab83f61d8ff86fafb5ce33f34ecd74a5231657e6aacf4
  Stored in directory: /run/user/1000/pip-ephem-wheel-cache-6RIzHk/wheels/ea/53/ea/3fa3b74531d5be329fc192128c9c1763cc2e2ddb867417cd86
Successfully built six
Installing collected packages: six
  Found existing installation: six 1.12.0
    Not uninstalling six at /nix/store/mra5bhz3dj332nni636q0p4f5j3ara09-python2.7-six-1.12.0/lib/python2.7/site-packages, outside environment /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/init
    Can't uninstall 'six'. No files were found to uninstall.
Successfully installed six-1.13.0
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing ./third_party/python/virtualenv
    Preparing wheel metadata ... done
Building wheels for collected packages: virtualenv
  Building wheel for virtualenv (PEP 517) ... done
  Created wheel for virtualenv: filename=virtualenv-16.7.8-cp27-none-any.whl size=3353919 sha256=25541eea4fa1fc1760e33d308881200bf672d1fd370f0d8e4d155eb10e4fd552
  Stored in directory: /run/user/1000/pip-ephem-wheel-cache-xAZ74S/wheels/42/fc/ab/8e71e4e554c0c1e4eead259e3e8459eb474781fc0686b176ec
Successfully built virtualenv
Installing collected packages: virtualenv
Successfully installed virtualenv-16.7.8
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing ./third_party/python/virtualenv-clone
Building wheels for collected packages: virtualenv-clone
  Building wheel for virtualenv-clone (setup.py) ... done
  Created wheel for virtualenv-clone: filename=virtualenv_clone-0.3.0-cp27-none-any.whl size=5810 sha256=f11fa9bc1fc626d71ff2d14aebeb2aa94e140a0faf2900c5f14613fca322afc1
  Stored in directory: /home/ethan/.cache/pip/wheels/ac/45/ce/3de5a31da5fec3233991e60032a6e4890bac599396a8e2f97e
Successfully built virtualenv-clone
Installing collected packages: virtualenv-clone
Successfully installed virtualenv-clone-0.3.0
Creating a virtualenv for this project…
Using /nix/store/7r7v9sf2zbqp3zdqx7p5g18qc7h79vb1-python3-3.5.9/bin/python3 (3.5.9) to create virtualenv…
⠋Running virtualenv with interpreter /nix/store/7r7v9sf2zbqp3zdqx7p5g18qc7h79vb1-python3-3.5.9/bin/python3
Already using interpreter /nix/store/7r7v9sf2zbqp3zdqx7p5g18qc7h79vb1-python3-3.5.9/bin/python3
Using base prefix '/nix/store/7r7v9sf2zbqp3zdqx7p5g18qc7h79vb1-python3-3.5.9'
New python executable in /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/docs-olH7paHv/bin/python3
Also creating executable in /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/docs-olH7paHv/bin/python
Installing setuptools, pip, wheel...
done.

Virtualenv location: /home/ethan/Jobs/Mozilla/gecko-git/obj-x86_64-pc-linux-gnu/_virtualenvs/docs-olH7paHv
Installing dependencies from Pipfile.lock (9f94f1)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 44/44 — 00:00:15
To activate this project's virtualenv, run the following:
 $ pipenv shell
Error running mach:

    ['doc']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ImportError: No module named livereload

  File "/home/ethan/Jobs/Mozilla/gecko-git/tools/docs/mach_commands.py", line 79, in build_docs
    from livereload import Server
  File "/home/ethan/Jobs/Mozilla/gecko-git/build/mach_bootstrap.py", line 431, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)

It looks like livereload is installed into some virtualenv, but not any virtualenv to which the code has access to, so when the code directly imports livereload, it fails. One obvious issue is that the virtualenv that livereload was installed into is a Python 3.5 environment, but I'm running mach with Python 2.7. I'm not sure if I'm still supposed to be using Python 2 for mach -- it seems like https://bugzilla.mozilla.org/show_bug.cgi?id=1577599 is still open -- but running it with Python 3 gives me a syntax error in the which module.

Comment 1

[Sylvestre Ledru [:Sylvestre]]

mach doc is now built with python 3.7: see bug 1601578 and https://bugzilla.mozilla.org/show_bug.cgi?id=1601612#c5
I don't think we want to support Python 2 anymore for mach doc

What is the error with which?

Comment 2

[Ethan Glasser-Camp (:glasserc)]

Attachment: error with which

Hmm, actually I'm running Python 3.5.9, that may be part of the problem. Edit: No, same problem with a newer Python.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

I think you should use
"./mach doc". It will create the virtualenv with the correct python version.
and don't prefix with python3

Comment 4

[Ethan Glasser-Camp (:glasserc)]

Running ./mach doc produces the same error.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Can you try to remove obj-*/_virtualenvs/*doc and try again?
the build system is easily confused when moving from Python 2 to 3

Comment 6

[Ethan Glasser-Camp (:glasserc)]

It looks like trying to instantiate the docs command is failing because part of instantiating a MozCommandBase involves loading mozconfig, which fails because it tries to execute shell, which fails because it wants find_program, which imports which. The which module in third_party is clearly written for Python 2. Maybe there's another one I'm missing?

I bet this is related somehow to bug 1584075, but running ./mach clobber does not fix it for me.

Comment 7

[Ethan Glasser-Camp (:glasserc)]

I guess this is a duplicate of the remaining part of bug 1563797. For now I will try to work around it.