Closed Bug 1464038 Opened 2 years ago Closed 8 months ago

Update our copy of the in-tree virtualenv to v16

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: ahal, Assigned: rstewart)

References

(Depends on 1 open bug)

Details

Attachments

(1 file, 4 obsolete files)

This will upgrade us from pip 9.0.3 -> 10.0.1 which will start building wheels when installing from local paths instead of running `setup.py install`.

There aren't any concrete reasons to do this upgrade now, but I was investigating whether the upgrade would fix a bug, so have the patch ready and it looks green on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5be1f76244fa324b6defec842c48bd557189e591
Comment on attachment 8980249 [details]
Bug 1464038 - Upgrade vendored copy of virtualenv to version 16.0.0,

https://reviewboard.mozilla.org/r/246414/#review252512
Attachment #8980249 - Flags: review?(dave.hunt) → review+
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fdf92ddfbf86
Upgrade vendored copy of virtualenv to version 16.0.0, r=davehunt
https://hg.mozilla.org/mozilla-central/rev/fdf92ddfbf86
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Backed out changeset fdf92ddfbf86 (bug 1464038) for causing nightly bustages after virtualenv upgrade to 16.0.0

Backout: https://hg.mozilla.org/mozilla-central/rev/1bdf8e7d1cfe6d8b9bfa891f43473345210ee281

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=0df854d34c01bafb3c88d595f8464ed381b95d03&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&selectedJob=180152207

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=180156551&repo=mozilla-central&lineNumber=194

[task 2018-05-25T02:47:21.305Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2018-05-25T02:47:21.305Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2018-05-25T02:47:22.966Z] Installing setuptools, pip, wheel...done.
[task 2018-05-25T02:47:24.008Z] running build_ext
[task 2018-05-25T02:47:24.008Z] building 'psutil._psutil_linux' extension
[task 2018-05-25T02:47:24.008Z] creating build
[task 2018-05-25T02:47:24.008Z] creating build/temp.linux-x86_64-2.7
[task 2018-05-25T02:47:24.008Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2018-05-25T02:47:24.008Z] creating build/lib.linux-x86_64-2.7
[task 2018-05-25T02:47:24.008Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2018-05-25T02:47:24.008Z] building 'psutil._psutil_posix' extension
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-05-25T02:47:24.008Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-05-25T02:47:24.008Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-05-25T02:47:24.008Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-05-25T02:47:24.008Z] 
[task 2018-05-25T02:47:24.008Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-05-25T02:47:24.212Z] Traceback (most recent call last):
[task 2018-05-25T02:47:24.212Z]   File "build/upload_generated_sources.py", line 161, in <module>
[task 2018-05-25T02:47:24.212Z]     sys.exit(main(sys.argv[1:]))
[task 2018-05-25T02:47:24.212Z]   File "build/upload_generated_sources.py", line 152, in main
[task 2018-05-25T02:47:24.212Z]     config.virtualenv_manager.install_pip_package('boto3==1.4.4')
[task 2018-05-25T02:47:24.212Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 476, in install_pip_package
[task 2018-05-25T02:47:24.212Z]     from pip.req import InstallRequirement
[task 2018-05-25T02:47:24.212Z]   File "/builds/worker/checkouts/gecko/build/mach_bootstrap.py", line 365, in __call__
[task 2018-05-25T02:47:24.212Z]     module = self._original_import(name, globals, locals, fromlist, level)
[task 2018-05-25T02:47:24.212Z] ImportError: No module named req
Status: RESOLVED → REOPENED
Flags: needinfo?(ahal)
Resolution: FIXED → ---
Target Milestone: mozilla62 → ---
Aryx pinged me earlier to ask about nightlies sanity as, potentially, a change in this bug broke the "upload-generated-sources" tasks in the graphs. 

Did a bit of digging and the tasks failing are cross-platform (e.g. the linux one is here[1] with the logs here[2]). 
The error seems to be:

[task 2018-05-25T02:08:09.631Z] Traceback (most recent call last):
[task 2018-05-25T02:08:09.631Z]   File "build/upload_generated_sources.py", line 161, in <module>
[task 2018-05-25T02:08:09.631Z]     sys.exit(main(sys.argv[1:]))
[task 2018-05-25T02:08:09.631Z]   File "build/upload_generated_sources.py", line 152, in main
[task 2018-05-25T02:08:09.631Z]     config.virtualenv_manager.install_pip_package('boto3==1.4.4')
[task 2018-05-25T02:08:09.631Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 476, in install_pip_package
[task 2018-05-25T02:08:09.631Z]     from pip.req import InstallRequirement
[task 2018-05-25T02:08:09.631Z]   File "/builds/worker/checkouts/gecko/build/mach_bootstrap.py", line 365, in __call__
[task 2018-05-25T02:08:09.631Z]     module = self._original_import(name, globals, locals, fromlist, level)
[task 2018-05-25T02:08:09.632Z] ImportError: No module named req

which indeed looks somehow related if this patch is the only one related to python changes in our infra in the past 24-36h. 
In https://hg.mozilla.org/mozilla-central/rev/1bdf8e7d1cfe6d8b9bfa891f43473345210ee281 (patch from this bug) it seems like we're updating the virtualenv from  15.2.0 to 16.0.0. 

Judging by the release notes[3] it sounds like pip was upgradede from 9.0.3 to 10.0.1 along other vendor updates and drop support for python 2.6. AFAIK, the pip upgrade from 9.0.3 to 10.0.1 is quite large (release notes confirm[4]) so I'm leaning towards a culprit somewhere around that upgrade. 

Digging a bit further the error itself, I found other related answers/questions on Stackoverflow, out of which this [5] one is the most representative. 

I'm not 100% yet, but I strongly suspect our error is related to the upgrade, hence a backout of this patch + retrigger nightlies should not harm.

[1]: https://tools.taskcluster.net/groups/V7A_FK3ARVWz_rqLH-zoog
[2]: https://taskcluster-artifacts.net/OsDfT0ZpSEaYrm6NKbx2Lw/0/public/logs/live_backing.log
[3]: https://virtualenv.pypa.io/en/stable/changes/#id1
[4]: https://pip.pypa.io/en/stable/news/
[5]: https://stackoverflow.com/questions/25192794/no-module-named-pip-req
Bummer, thanks for the investigation Mihai.

Like I mentioned in comment 0, there isn't any particular need to do this upgrade at this time. While I'm sure we'll want to prioritize this eventually, I don't have the bandwidth to look into this now. So unassigning myself for the time being. If anyone wants to take over please feel free.
Assignee: ahal → nobody
Flags: needinfo?(ahal)
Version: Version 3 → 3 Branch
See Also: → 1416924
Summary: Update our copy of the in-tree virtualenv to 16.0.0 → Update our copy of the in-tree virtualenv to v16
Attachment #8980249 - Attachment is obsolete: true

Depends on D49975

Depends on D49979

The last patch in the stack upgrade to 16.7.5 and also fixes the issue that caused the backout last year. Basically pip was refactored and the pip.req module no longer existed. I had to re-write some of the logic in virtualenv.py to get it working again.

I had been sitting on these patches for awhile because they haven't been properly tested. I'm uploading them now so A) I don't forget about them, and B) so other people don't try to re-solve the same issue. Probably won't have time to push them through right now though, so feel free to steal these patches and get them landed.

Duplicate of this bug: 1596493

I discovered our vendored copy of pip-tools is broken with pip >= 19 (which this change would update us to). So we need to update to at least pip-tools==3.3.1 first. Though updating to the latest pip-tools causes problem with our current pip. I found pip-tools==3.9.0 seemed to work.

Though I don't have a patch because ./mach vendor python seems to update everything, not just the specified package. If someone wants to pick this up I'd be very happy to help!

Depends on: 1600782
Depends on: 1600805
Depends on: 1600812
Attachment #9103080 - Attachment description: Bug 1464038 - Update virtualenv to 16.7.5 → Bug 1464038 - Update virtualenv to 16.7.7
Attachment #9103080 - Attachment description: Bug 1464038 - Update virtualenv to 16.7.7 → Bug 1464038 - Update virtualenv to version 16.7.8

I've been seeing an error when running mach with the upgraded virtualenv library.

 ~/w/firefox  ./mach python-test xxx                                                          835ms  Wed 04 Dec 2019 10:38:15 AM EST
New python executable in /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
Also creating executable in /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
Installing setuptools, pip, wheel...
done.
....
Processing ./third_party/python/virtualenv
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7 /home/mars/work/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lMM6io/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index -- 'setuptools >= 40.6.3' 'wheel >= 0.29.0'
       cwd: None
  ERROR: Could not find a version that satisfies the requirement setuptools>=40.6.3 (from versions: none)
  ERROR: No matching distribution found for setuptools>=40.6.3

This is because updating the virtualenv package also updates the version of pip that is installed in our init/ virtualenv. Our current version of virtulaenv bundles pip 9.0.3 and installs it into into the virtualenv during venv creation. The upgraded virtualenv package bundles pip 19.3.1. Our code in mozbuild.virtualenv.VirtualenvManager.install_pip_package() that installs packages into the init/ virtualenv after initial setup works differently on pip v19, causing the error I'm seeing above.

Our code installs packages with the --no-index and --no-deps flags. This is how we achieve build isolation. You can simulate this on a m-c checkout with the following command:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip install --no-index --no-deps third_party/python/virtualenv              1179ms  Tue 03 Dec 2019 10:38:24 PM EST
Processing ./third_party/python/virtualenv
Installing collected packages: virtualenv
  Running setup.py install for virtualenv ... done
Successfully installed virtualenv-15.2.0

When we run mach python-test the python-test command setup tries to install the virtualenv package into the init/ venv (the virtualenv package is a pipenv dependency, and we use pipenv to run the python test suite). The virtualenv package has a setup_requires dependency on setuptools. Because our code calls pip install with the --no-index flag, pip fails to find the setuptools package that is already present in the virtualenv and dies.

We can see this if we switch to the branch with the updated virtualenv. We can verify that the virtualenv is pre-populated with the correct dependencies:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip list | grep setuptools        564ms  Wed 04 Dec 2019 11:02:39 AM EST
setuptools       41.6.0   

And simulate the package installation:

 ~/w/firefox  obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/pip install --no-index --no-deps third_party/python/virtualenv
...
  ERROR: No matching distribution found for setuptools>=40.6.3

We might be able to work around this using --no-index --find-links pointing at the third_party/python directory (example). Or we could add virtualenv to the build/virtualenv_packages.txt file so mozbuild.base.MozbuildObject.ensure_pipenv() does not need to install it.

:ahal what do you think?

Flags: needinfo?(ahal)
Assignee: nobody → rstewart

I circled back with :mars on IRC but just to update the bug. It seems like the root of some of the issues was the use of pipenv to manage virtualenvs in our mach python-test infra. I put up a patch in bug 1601445 that changes this assumption (removes pipenv).

I'm not 100% sure if it unblocks this bug, but would be my preferred approach if it does. Though, that change itself causes some mozbuild test failures. I'm also fine with playing around with --no-index / --find-links if we can get something working with those however. Basically anything that can unblock us here gets a big +1 from me.

Flags: needinfo?(ahal)
Pushed by rstewart@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/85b9d48318c4
Update virtualenv to version 16.7.8 r=mars
Blocks: 1601578
Version: 3 Branch → unspecified
Status: REOPENED → RESOLVED
Closed: 2 years ago8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
Attachment #9103074 - Attachment is obsolete: true
Attachment #9103078 - Attachment is obsolete: true
Attachment #9103080 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.