1464038 - Update our copy of the in-tree virtualenv to v16

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

This will upgrade us from pip 9.0.3 -> 10.0.1 which will start building wheels when installing from local paths instead of running `setup.py install`.

There aren't any concrete reasons to do this upgrade now, but I was investigating whether the upgrade would fix a bug, so have the patch ready and it looks green on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5be1f76244fa324b6defec842c48bd557189e591

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1464038 - Upgrade vendored copy of virtualenv to version 16.0.0,

This will in turn upgrade our version of pip from 9.0.3 -> 10.0.1

Review commit: https://reviewboard.mozilla.org/r/246414/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246414/

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8980249 [details]
Bug 1464038 - Upgrade vendored copy of virtualenv to version 16.0.0,

https://reviewboard.mozilla.org/r/246414/#review252512

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fdf92ddfbf86
Upgrade vendored copy of virtualenv to version 16.0.0, r=davehunt

Comment 4

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/fdf92ddfbf86

Comment 5

[Cristian Brindusan [:cbrindusan]]

Backed out changeset fdf92ddfbf86 (bug 1464038) for causing nightly bustages after virtualenv upgrade to 16.0.0

Backout: https://hg.mozilla.org/mozilla-central/rev/1bdf8e7d1cfe6d8b9bfa891f43473345210ee281

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=0df854d34c01bafb3c88d595f8464ed381b95d03&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&selectedJob=180152207

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=180156551&repo=mozilla-central&lineNumber=194

[task 2018-05-25T02:47:21.305Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2018-05-25T02:47:21.305Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2018-05-25T02:47:22.966Z] Installing setuptools, pip, wheel...done.
[task 2018-05-25T02:47:24.008Z] running build_ext
[task 2018-05-25T02:47:24.008Z] building 'psutil._psutil_linux' extension
[task 2018-05-25T02:47:24.008Z] creating build
[task 2018-05-25T02:47:24.008Z] creating build/temp.linux-x86_64-2.7
[task 2018-05-25T02:47:24.008Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2018-05-25T02:47:24.008Z] creating build/lib.linux-x86_64-2.7
[task 2018-05-25T02:47:24.008Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2018-05-25T02:47:24.008Z] building 'psutil._psutil_posix' extension
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-05-25T02:47:24.008Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-05-25T02:47:24.008Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-05-25T02:47:24.008Z] 
[task 2018-05-25T02:47:24.008Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-05-25T02:47:24.212Z] Traceback (most recent call last):
[task 2018-05-25T02:47:24.212Z]   File "build/upload_generated_sources.py", line 161, in <module>
[task 2018-05-25T02:47:24.212Z]     sys.exit(main(sys.argv[1:]))
[task 2018-05-25T02:47:24.212Z]   File "build/upload_generated_sources.py", line 152, in main
[task 2018-05-25T02:47:24.212Z]     config.virtualenv_manager.install_pip_package('boto3==1.4.4')
[task 2018-05-25T02:47:24.212Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 476, in install_pip_package
[task 2018-05-25T02:47:24.212Z]     from pip.req import InstallRequirement
[task 2018-05-25T02:47:24.212Z]   File "/builds/worker/checkouts/gecko/build/mach_bootstrap.py", line 365, in __call__
[task 2018-05-25T02:47:24.212Z]     module = self._original_import(name, globals, locals, fromlist, level)
[task 2018-05-25T02:47:24.212Z] ImportError: No module named req

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Aryx pinged me earlier to ask about nightlies sanity as, potentially, a change in this bug broke the "upload-generated-sources" tasks in the graphs. 

Did a bit of digging and the tasks failing are cross-platform (e.g. the linux one is here[1] with the logs here[2]). 
The error seems to be:

[task 2018-05-25T02:08:09.631Z] Traceback (most recent call last):
[task 2018-05-25T02:08:09.631Z]   File "build/upload_generated_sources.py", line 161, in <module>
[task 2018-05-25T02:08:09.631Z]     sys.exit(main(sys.argv[1:]))
[task 2018-05-25T02:08:09.631Z]   File "build/upload_generated_sources.py", line 152, in main
[task 2018-05-25T02:08:09.631Z]     config.virtualenv_manager.install_pip_package('boto3==1.4.4')
[task 2018-05-25T02:08:09.631Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 476, in install_pip_package
[task 2018-05-25T02:08:09.631Z]     from pip.req import InstallRequirement
[task 2018-05-25T02:08:09.631Z]   File "/builds/worker/checkouts/gecko/build/mach_bootstrap.py", line 365, in __call__
[task 2018-05-25T02:08:09.631Z]     module = self._original_import(name, globals, locals, fromlist, level)
[task 2018-05-25T02:08:09.632Z] ImportError: No module named req

which indeed looks somehow related if this patch is the only one related to python changes in our infra in the past 24-36h. 
In https://hg.mozilla.org/mozilla-central/rev/1bdf8e7d1cfe6d8b9bfa891f43473345210ee281 (patch from this bug) it seems like we're updating the virtualenv from  15.2.0 to 16.0.0. 

Judging by the release notes[3] it sounds like pip was upgradede from 9.0.3 to 10.0.1 along other vendor updates and drop support for python 2.6. AFAIK, the pip upgrade from 9.0.3 to 10.0.1 is quite large (release notes confirm[4]) so I'm leaning towards a culprit somewhere around that upgrade. 

Digging a bit further the error itself, I found other related answers/questions on Stackoverflow, out of which this [5] one is the most representative. 

I'm not 100% yet, but I strongly suspect our error is related to the upgrade, hence a backout of this patch + retrigger nightlies should not harm.

[1]: https://tools.taskcluster.net/groups/V7A_FK3ARVWz_rqLH-zoog
[2]: https://taskcluster-artifacts.net/OsDfT0ZpSEaYrm6NKbx2Lw/0/public/logs/live_backing.log
[3]: https://virtualenv.pypa.io/en/stable/changes/#id1
[4]: https://pip.pypa.io/en/stable/news/
[5]: https://stackoverflow.com/questions/25192794/no-module-named-pip-req

Comment 7

[Andrew Halberstadt [:ahal]]

Bummer, thanks for the investigation Mihai.

Like I mentioned in comment 0, there isn't any particular need to do this upgrade at this time. While I'm sure we'll want to prioritize this eventually, I don't have the bandwidth to look into this now. So unassigning myself for the time being. If anyone wants to take over please feel free.

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1464038 - Update virtualenv-clone to 0.5.3

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1464038 - Update pipenv to 2018.11.26

Depends on D49975

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1464038 - Update virtualenv to version 16.7.8

Depends on D49979

Comment 11

[Andrew Halberstadt [:ahal]]

The last patch in the stack upgrade to 16.7.5 and also fixes the issue that caused the backout last year. Basically pip was refactored and the pip.req module no longer existed. I had to re-write some of the logic in virtualenv.py to get it working again.

I had been sitting on these patches for awhile because they haven't been properly tested. I'm uploading them now so A) I don't forget about them, and B) so other people don't try to re-solve the same issue. Probably won't have time to push them through right now though, so feel free to steal these patches and get them landed.

Comment 13

[Andrew Halberstadt [:ahal]]

I discovered our vendored copy of pip-tools is broken with pip >= 19 (which this change would update us to). So we need to update to at least pip-tools==3.3.1 first. Though updating to the latest pip-tools causes problem with our current pip. I found pip-tools==3.9.0 seemed to work.

Though I don't have a patch because ./mach vendor python seems to update everything, not just the specified package. If someone wants to pick this up I'd be very happy to help!

Comment 14

[Māris Fogels [:mars] (please needinfo)]

I've been seeing an error when running mach with the upgraded virtualenv library.

 ~/w/firefox  ./mach python-test xxx                                                          835ms  Wed 04 Dec 2019 10:38:15 AM EST
New python executable in /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
Also creating executable in /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
Installing setuptools, pip, wheel...
done.
....
Processing ./third_party/python/virtualenv
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7 /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lMM6io/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index -- 'setuptools >= 40.6.3' 'wheel >= 0.29.0'
       cwd: None
  ERROR: Could not find a version that satisfies the requirement setuptools>=40.6.3 (from versions: none)
  ERROR: No matching distribution found for setuptools>=40.6.3

This is because updating the virtualenv package also updates the version of pip that is installed in our init/ virtualenv. Our current version of virtulaenv bundles pip 9.0.3 and installs it into into the virtualenv during venv creation. The upgraded virtualenv package bundles pip 19.3.1. Our code in mozbuild.virtualenv.VirtualenvManager.install_pip_package() that installs packages into the init/ virtualenv after initial setup works differently on pip v19, causing the error I'm seeing above.

Our code installs packages with the --no-index and --no-deps flags. This is how we achieve build isolation. You can simulate this on a m-c checkout with the following command:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip install --no-index --no-deps third_party/python/virtualenv              1179ms  Tue 03 Dec 2019 10:38:24 PM EST
Processing ./third_party/python/virtualenv
Installing collected packages: virtualenv
  Running setup.py install for virtualenv ... done
Successfully installed virtualenv-15.2.0

When we run mach python-test the python-test command setup tries to install the virtualenv package into the init/ venv (the virtualenv package is a pipenv dependency, and we use pipenv to run the python test suite). The virtualenv package has a setup_requires dependency on setuptools. Because our code calls pip install with the --no-index flag, pip fails to find the setuptools package that is already present in the virtualenv and dies.

We can see this if we switch to the branch with the updated virtualenv. We can verify that the virtualenv is pre-populated with the correct dependencies:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip list | grep setuptools        564ms  Wed 04 Dec 2019 11:02:39 AM EST
setuptools       41.6.0   

And simulate the package installation:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip install --no-index --no-deps third_party/python/virtualenv
...
  ERROR: No matching distribution found for setuptools>=40.6.3

We might be able to work around this using --no-index --find-links pointing at the third_party/python directory (example). Or we could add virtualenv to the build/virtualenv_packages.txt file so mozbuild.base.MozbuildObject.ensure_pipenv() does not need to install it.

:ahal what do you think?

Comment 15

[Ricky Stewart]

Attachment: Bug 1464038 - Update virtualenv to version 16.7.8

Comment 16

[Andrew Halberstadt [:ahal]]

I circled back with :mars on IRC but just to update the bug. It seems like the root of some of the issues was the use of pipenv to manage virtualenvs in our mach python-test infra. I put up a patch in bug 1601445 that changes this assumption (removes pipenv).

I'm not 100% sure if it unblocks this bug, but would be my preferred approach if it does. Though, that change itself causes some mozbuild test failures. I'm also fine with playing around with --no-index / --find-links if we can get something working with those however. Basically anything that can unblock us here gets a big +1 from me.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/85b9d48318c4
Update virtualenv to version 16.7.8 r=mars

Comment 18

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/85b9d48318c4