1655107 - Stop using pipenv to manage |mach doc| dependencies

Status: RESOLVED FIXED

(Developer Infrastructure :: Source Documentation, task, P3)

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1655107 - [moztreedocs] Use pip-tools rather than pipenv to manage doc dependencies, r?rstewart

One side effect of this is that the dependencies will now be installed into the
root build venv rather than an independent docs one. Ideally we would keep it
independent, but I'd like to solve that properly in the build system /
virtualenv.py rather than do something hacky here.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1655107 - [moztreedocs] Use pip-tools rather than pipenv to manage doc dependencies

One side effect of this is that the dependencies will now be installed into the
root build venv rather than an independent docs one. Ideally we would keep it
independent, but I'd like to solve that properly in the build system /
virtualenv.py rather than do something hacky here.

Depends on D85047

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6124d9a5422b
[moztreedocs] Use pip-tools rather than pipenv to manage doc dependencies r=championshuttler,sylvestre

Comment 4

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 6124d9a5422b (bug 1655107) for pip related failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=GK0LH2N5SsG-00MCVZlEWQ.0&searchStr=documentation%2Copt%2Cdocumentation%2Csource-test-doc-generate%2Cgenerate&fromchange=6124d9a5422b8abfc35f6a1bbc7c0066483e1bf9&tochange=e8b2d748699a05bc28afde7536eb057d38adc8c2

Backout link: https://hg.mozilla.org/integration/autoland/rev/e8b2d748699a05bc28afde7536eb057d38adc8c2

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313212396&repo=autoland&lineNumber=220

...
[task 2020-08-17T14:16:45.374Z] Requirement already satisfied: setuptools in ./obj-x86_64-pc-linux-gnu/_virtualenvs/docs/lib/python3.6/site-packages (from sphinx==3.1.2->-r /builds/worker/checkouts/gecko/tools/moztreedocs/requirements.txt (line 203)) (41.6.0)
[task 2020-08-17T14:16:45.420Z] Collecting zipp>=0.5
[task 2020-08-17T14:16:45.602Z] ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
[task 2020-08-17T14:16:45.602Z]     zipp>=0.5 from https://files.pythonhosted.org/packages/b2/34/bfcb43cc0ba81f527bc4f40ef41ba2ff4080e047acb0586b56b3d017ace4/zipp-3.1.0-py3-none-any.whl#sha256=aa36550ff0c0b7ef7fa639055d797116ee891440eac1a56f378e2d3179e0320b (from importlib-metadata; python_version < "3.8"->markdown==3.2.2->-r /builds/worker/checkouts/gecko/tools/moztreedocs/requirements.txt (line 71))
[task 2020-08-17T14:16:45.773Z] WARNING: You are using pip version 19.3.1; however, version 20.2.2 is available.
[task 2020-08-17T14:16:45.773Z] You should consider upgrading via the 'pip install --upgrade pip' command.
[task 2020-08-17T14:16:45.820Z] Error running mach:
[task 2020-08-17T14:16:45.820Z] 
[task 2020-08-17T14:16:45.820Z]     ['doc', '--outdir', 'docs-out', '--no-open', '--no-serve', '--archive']
[task 2020-08-17T14:16:45.821Z] 
[task 2020-08-17T14:16:45.821Z] The error occurred in code that was called by the mach command. This is either
[task 2020-08-17T14:16:45.821Z] a bug in the called code itself or in the way that mach is calling it.
[task 2020-08-17T14:16:45.821Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2020-08-17T14:16:45.821Z] isn't, please use |./mach busted file doc| to report it. If |./mach busted| is
[task 2020-08-17T14:16:45.821Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2020-08-17T14:16:45.821Z] 
[task 2020-08-17T14:16:45.821Z] If filing a bug, please include the full output of mach, including this error
[task 2020-08-17T14:16:45.821Z] message.
[task 2020-08-17T14:16:45.821Z] 
[task 2020-08-17T14:16:45.821Z] The details of the failure are as follows:
[task 2020-08-17T14:16:45.821Z] 
[task 2020-08-17T14:16:45.821Z] subprocess.CalledProcessError: Command '['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/docs/bin/pip', 'install', '--requirement', '/builds/worker/checkouts/gecko/tools/moztreedocs/requirements.txt', '--require-hashes']' returned non-zero exit status 1.
[task 2020-08-17T14:16:45.821Z] 
[task 2020-08-17T14:16:45.821Z]   File "/builds/worker/checkouts/gecko/tools/moztreedocs/mach_commands.py", line 84, in build_docs
[task 2020-08-17T14:16:45.821Z]     self.virtualenv_manager.install_pip_requirements(os.path.join(here, 'requirements.txt'))
[task 2020-08-17T14:16:45.821Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 565, in install_pip_requirements
[task 2020-08-17T14:16:45.821Z]     return self._run_pip(args)
[task 2020-08-17T14:16:45.821Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 577, in _run_pip
[task 2020-08-17T14:16:45.821Z]     universal_newlines=PY3)
[task 2020-08-17T14:16:45.821Z]   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
[task 2020-08-17T14:16:45.821Z]     raise CalledProcessError(retcode, cmd)
[taskcluster 2020-08-17 14:16:46.136Z] === Task Finished ===
[taskcluster 2020-08-17 14:16:46.249Z] Artifact "public/docs.tar.gz" not found at "/builds/worker/checkouts/gecko/docs-out/main.tar.gz"
[taskcluster 2020-08-17 14:16:46.438Z] Unsuccessful task run with exit code: 1 completed in 102.877 seconds

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/feb7f8598151
[moztreedocs] Use pip-tools rather than pipenv to manage doc dependencies r=championshuttler,sylvestre

Comment 6

[Andrew Halberstadt [:ahal]]

This issue only showed up when running on Python < 3.8 (we use 3.6 in CI). We needed to add importlib-metadata to the list of packages. Already fixed and relanded.

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/feb7f8598151