837631 - Virtualenv failing on multi-arch Debian-based distros

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Marco Castelluccio [:marco]]

I can't build Firefox on Linux anymore.
This is the build log:

Creating Python environment
New python executable in /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python2.7
Not overwriting existing python script /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python (you must use /home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python2.7)
Installing setuptools...................................
  Complete output from command /home/marco/Scrivani...ualenv/bin/python2.7 -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/marco/Scrivani...ols-0.6c11-py2.7.egg:
  Traceback (most recent call last):
  File "<string>", line 279, in <module>
  File "<string>", line 240, in main
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 125, in finalize_options
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1121, in _expand
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 32, in finalize_options
  File "/usr/lib/python2.7/distutils/command/install.py", line 321, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/home/marco/Scrivania/FD/src/obj-x86_64-unknown-linux-gnu/_virtualenv/lib/python2.7/distutils/__init__.py", line 92, in sysconfig_get_config_vars
    real_vars = old_get_config_vars(*args)
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 503, in get_config_vars
    func()
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 447, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 2476, in <module>
    main()
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 942, in main
    never_download=options.never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 1052, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 599, in install_setuptools
    search_dirs=search_dirs, never_download=never_download)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 571, in _install_req
    cwd=cwd)
  File "/home/marco/Scrivania/FD/src/python/virtualenv/virtualenv.py", line 1020, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/marco/Scrivani...ualenv/bin/python2.7 -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/marco/Scrivani...ols-0.6c11-py2.7.egg failed with error code 1
Traceback (most recent call last):
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 356, in <module>
    manager.ensure()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 100, in ensure
    return self.build()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 290, in build
    self.create()
  File "/home/marco/Scrivania/FD/src/build/virtualenv/populate_virtualenv.py", line 119, in create
    raise Exception('Error creating virtualenv.')
Exception: Error creating virtualenv.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Part 1: Upgrade Virtualenv to 1.8.4

Looks like this is fixed in Virtualenv 1.8.5 which is unreleased. I plan to upgrade to 1.8.4 then cherry-pick the multi-arch change from 1.8.5.

In this patch, I |rm -rf| python/virtualenv then uncompressed the 1.8.4 archive in its place then did |hg add| |hg rm| as appropriate. r+ should pretty much be a rubber stamp.

Comment 2

[Mike Hommey [:glandium]]

Comment on attachment 709752 [details] [diff] [review]
Part 1: Upgrade Virtualenv to 1.8.4

rs=me, assuming it is what is advertized. While you're here, could you just add a script handling these updates, like we have for various other third-party imports? (like jemalloc, unwind, etc.)

Comment 3

[Gregory Szorc [:gps]]

Attachment: Part 2: Unbust virtualenv on Debians, v1

This is the same patch as posted on http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695707. This is an upstream patch, so r+ should be a rubber stamp.

Comment 4

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f6d2ec030613
https://hg.mozilla.org/integration/mozilla-inbound/rev/661f7866da20

Comment 5

[Marco Castelluccio [:marco]]

With those two patches applied, I still can't build.

Comment 6

[Gregory Szorc [:gps]]

Even with a clobber?

Comment 7

[Gregory Szorc [:gps]]

Marco reported on IRC that a clobber didn't work either :/

Marco is also running 2.7.3-15. I'm not sure what is going on in Debian land.

Comment 8

[Daniel Holbert [:dholbert]]

FWIW, I was able to work around this by adding a symlink to /usr/lib/python2.7 , with the commands that I noted over on the ubuntu bug report:
 https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1115466/comments/7
...but it's a bit scary to go manually mucking around in /usr/lib, so it's not a great long-term workaround.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/f6d2ec030613
https://hg.mozilla.org/mozilla-central/rev/661f7866da20

Comment 10

[Chris Coulson]

Attachment: Refresh virtualenv.py to pick up the changes from 661f7866da20

This still doesn't work because virtualenv.py needs to be refreshed to pick up the changes to site.py

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 710215 [details] [diff] [review]
Refresh virtualenv.py to pick up the changes from 661f7866da20

Review of attachment 710215 [details] [diff] [review]:
-----------------------------------------------------------------

Good catch. I didn't realize you needed to do this :/

I /think/ if we call into the Virtualenv Python API directly we wouldn't need to do this. Something to consider for a follow-up bug.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4a6b8dd4dfe3

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/4a6b8dd4dfe3

Comment 14

[Vicamo Yang [:vicamo][:vyang]]

But http://dxr.mozilla.org/mozilla-central/testing/marionette/client/marionette/venv_test.sh#l6 still uses "1.8.2". B2G Marionette still suffers from the same problem and need at least commit 38d56d9[1] to include the same change.

[1]: https://github.com/pypa/virtualenv/commit/38d56d9b83b9902c942412482b401fd72c2320bd

Comment 15

[(not currently active) Ted Mielczarek]

Please file a new bug for this problem.

Comment 16

[Justin Lebar (not reading bugmail)]

Can we please uplift this to b2g?  Now that people are updating to the new version of Ubuntu, they're going to run into this problem.

This is blocking development (I think -- I think I have another, separate build problem), so tef?

Comment 17

[Justin Lebar (not reading bugmail)]

Actually...I see these errors, but now that my other build problem (needed to recompile clang) is cleared up, the virtualenv errors don't seem to be preventing me from getting a successful build.

Do we need to care about these errors?

Comment 18

[Jens Müller (:tessarakt)]

I updated comm-central on Sunday, May 5th. From all I can see, it is still affected. When will this be fixed for comm-central?

Comment 19

[Jens Müller (:tessarakt)]

Apparently there is another virtualenv.py in comm at mail/test/resources/virtualenv/virtualenv.py, and that one is still 1.7 ...

File a new bug on this?

Comment 20

[Jens Müller (:tessarakt)]

See bug 837631

Comment 21

[Jens Müller (:tessarakt)]

Bug 869609 I meant, sorry.

Comment 22

[Alex Keybl [:akeybl]]

We'd rather not take this change on v1.0.1 (devs can apply locally as necessary). leo+ though to get this into v1.1.

Comment 23

[Gregory Szorc [:gps]]

Attachment: Upgrade Virtualenv to 1.8.4 and backport upstream fixes

python/virtualenv from mozilla-central at feccfce43b59 replaces the
existing virtualenv.

Comment 24

[Mike Hommey [:glandium]]

Comment on attachment 750606 [details] [diff] [review]
Upgrade Virtualenv to 1.8.4 and backport upstream fixes

Review of attachment 750606 [details] [diff] [review]:
-----------------------------------------------------------------

I'd like this on esr17 too (it now fails to build on debian unstable because of this bug)

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/8a8c4863530b

Comment 26

[Mike Hommey [:glandium]]

Comment on attachment 750606 [details] [diff] [review]
Upgrade Virtualenv to 1.8.4 and backport upstream fixes

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: This prevents building ESR on Debian and probably Ubuntu.
Fix Landed on Version: 21 and b2g18.
Risk to taking this patch (and alternatives if risky): Low
String or UUID changes made by this patch: None

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr17/rev/ecab7f032c13