805127 - Upgrade virtualenv to 1.8.2

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

Our in-tree virtualenv is 1.7.1.2. Virtualenv 1.8.2 is available. We may be seeing a virtualenv bug in bug 805091. I propose we upgrade virtualenv to 1.8.2. It may make that bug go away. If it doesn't, at least we have a new version of virtualenv in the tree!

Try at https://tbpl.mozilla.org/?tree=Try&rev=d9e0e050e471

Comment 1

[Gregory Szorc [:gps]]

Attachment: Upgrade virtualenv to 1.8.2, v1

Grabbed upstream 1.8.2 tarball. Deleted all files in python/virtualenv/. Uncompressed tarball to python/virtualenv/. Removed egg-info directory. Deleted old files and added new ones. Essentially python/virtualenv is the upstream tarball minus egg-info.

Comment 2

[Gregory Szorc [:gps]]

Try build was green.

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 674773 [details] [diff] [review]
Upgrade virtualenv to 1.8.2, v1

Review of attachment 674773 [details] [diff] [review]:
-----------------------------------------------------------------

As long as this doesn't break try, go for it.

Comment 4

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/50ce28b49e32

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

I imported that changeset into the tree on the latest failing machine.  It still fails, but differently

checking for python2.7... /usr/bin/python2.7
Creating Python environment
New python executable in /builds/slave/m-cen-osx64-gecko-ntly/build2/obj-x86_64-apple-darwin11.2.0/_virtualenv/bin/python
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/python/virtualenv/virtualenv.py", line 2471, in <module>
    main()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/python/virtualenv/virtualenv.py", line 942, in main
    never_download=options.never_download)
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/python/virtualenv/virtualenv.py", line 1043, in create_environment
    site_packages=site_packages, clear=clear))
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/python/virtualenv/virtualenv.py", line 1293, in install_python
    shutil.copyfile(executable, py_executable)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 62] Too many levels of symbolic links: '/builds/slave/m-cen-osx64-gecko-ntly/build2/obj-x86_64-apple-darwin11.2.0/_virtualenv/bin/python'
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/build/virtualenv/populate_virtualenv.py", line 321, in <module>
    manager.ensure()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/build/virtualenv/populate_virtualenv.py", line 75, in ensure
    return self.build()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/build/virtualenv/populate_virtualenv.py", line 258, in build
    self.create()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build2/build/virtualenv/populate_virtualenv.py", line 105, in create
    raise Exception('Error creating virtualenv.')
Exception: Error creating virtualenv.
*** Fix above errors and then restart with               "make -f client.mk build"
make: *** [configure] Error 1

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

needed to clobber because of infinite recursion in the symlinks inside the venv

checking for python2.7... /usr/bin/python2.7
Creating Python environment
New python executable in /builds/slave/m-cen-osx64-gecko-ntly/build2/obj-x86_64-apple-darwin11.2.0/_virtualenv/bin/python
Installing setuptools............done.
Installing pip...............done.
running build_ext

This change requires clobbering, but we do that for all nightly builds and don't have dependency builds for desktop b2g yet.

I would call this fixed, i've verified that we no longer fail configure in a directory that previously failed.  the only change was to hg import Greg's patch from inbound.  I did this twice, and verified that it fixed it both times.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/50ce28b49e32

Comment 8

[Marco Castelluccio [:marco]]

I can't build Firefox anymore because of this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695707
Can we integrate that patch in our in-tree virtualenv?

Comment 9

[Marco Castelluccio [:marco]]

I've tried to manually apply that patch, and it doesn't work.
This is the build log:

Creating Python environment
New python executable in /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python2.7
Not overwriting existing python script /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python (you must use /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python2.7)
Installing setuptools...................................
  Complete output from command /home/marco/Scrivani...ualenv/bin/python2.7 -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/marco/Scrivani...ols-0.6c11-py2.7.egg:
  Traceback (most recent call last):
  File "<string>", line 279, in <module>
  File "<string>", line 240, in main
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 125, in finalize_options
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1121, in _expand
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 32, in finalize_options
  File "/usr/lib/python2.7/distutils/command/install.py", line 321, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/lib/python2.7/distutils/__init__.py", line 92, in sysconfig_get_config_vars
    real_vars = old_get_config_vars(*args)
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 503, in get_config_vars
    func()
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 447, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 2476, in <module>
    main()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 942, in main
    never_download=options.never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 1052, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 599, in install_setuptools
    search_dirs=search_dirs, never_download=never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 571, in _install_req
    cwd=cwd)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 1020, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/marco/Scrivani...ualenv/bin/python2.7 -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/marco/Scrivani...ols-0.6c11-py2.7.egg failed with error code 1
Traceback (most recent call last):
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 356, in <module>
    manager.ensure()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 100, in ensure
    return self.build()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 290, in build
    self.create()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 119, in create
    raise Exception('Error creating virtualenv.')
Exception: Error creating virtualenv.