869609 - mail: Virtualenv failing on multi-arch Debian-based distros

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect)

Description

[Jens Müller (:tessarakt)]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:22.0) Gecko/20130505 Firefox/22.0
Build ID: 20130505004014

Steps to reproduce:

Tried to build recent comm-central.


Actual results:

Build failed with the following log entries:

rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools...................................
  Complete output from command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg:
  Traceback (most recent call last):
  File "<string>", line 279, in <module>
  File "<string>", line 240, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 125, in finalize_options
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1121, in _expand
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 32, in finalize_options
  File "/usr/lib/python2.7/distutils/command/install.py", line 321, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py", line 88, in sysconfig_get_config_vars
    real_vars = old_get_config_vars(*args)
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 489, in get_config_vars
    func()
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 433, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2098, in <module>
    main()
  File "virtualenv/virtualenv.py", line 928, in main
    never_download=options.never_download)
  File "virtualenv/virtualenv.py", line 1040, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 593, in install_setuptools
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 567, in _install_req
    cwd=cwd)
  File "virtualenv/virtualenv.py", line 1006, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg failed with error code 1
Failure to install virtualenv



Expected results:

Build should have been successful.

Comment 1

[Jens Müller (:tessarakt)]

I replaced the contents of mail/test/resources/virtualenv with those of mozilla/python/virtualenv.

Apparently this works now. Here the relevant part of the logfile:

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/fakeserver/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mail/base/test/unit/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Overwriting /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py with new content
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...
make[5]: Leaving directory '/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mail/test/mozmill'


Is this ok? Or is there anything else I have to care about?

Comment 2

[Jens Müller (:tessarakt)]

Attachment: Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv.

Comment 3

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Wouldn't it be better to just use the code from m-c?

Comment 4

[Jens Müller (:tessarakt)]

Probably. That needs to be done by someone who understands the build system.

Comment 5

[Jens Müller (:tessarakt)]

Attachment: Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv.

Comment 6

[Jens Müller (:tessarakt)]

(In reply to Jens Müller from comment #4)
> Probably. That needs to be done by someone who understands the build system.

I guess in mail/resources/installmozmill.py the following bits need to be changed:

  # check for existence of necessary files
  if not os.path.exists('virtualenv'):
    print "File not found: virtualenv"
    sys.exit(1)

and

  returncode = call([sys.executable, os.path.join('virtualenv', 'virtualenv.py'),
                     destination], env=env)

Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the correct syntax?

Comment 7

[Jens Müller (:tessarakt)]

(In reply to Jens Müller from comment #6)
> Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the
> correct syntax?

Apparently, it is more complex than that: The script is executed in the obj directory. And the mozilla/python/virtualenv is not copied to the obj dir.

Comment 8

[Jens Müller (:tessarakt)]

OK, should not be too difficult. Apparently, the following block in mail/test/mozmill/Makefile.in needs to be changed.

# Copy MozMill and its dependencies over, and set up a virtualenv. The
# virtualenv directory is outside because we don't want to bundle it up during
# stage-package.
VIRTUALENV_DIR = $(_DEST_DIR)/../mozmill-virtualenv
mozmill-virtualenv: NSDISTMODE=copy
mozmill-virtualenv:
	$(INSTALL) $(topsrcdir)/mail/test/resources $(_DEST_DIR)
	rm -rf $(VIRTUALENV_DIR) && \
	mkdir $(VIRTUALENV_DIR) && \
	unset MACOSX_DEPLOYMENT_TARGET && \
	$(PYTHON) $(_DEST_DIR)/resources/installmozmill.py $(VIRTUALENV_DIR)

I will try that later on.

Comment 9

[Jens Müller (:tessarakt)]

OK, that one worked as well. I'll prepare the patch now.

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...

Comment 10

[Jens Müller (:tessarakt)]

Attachment: Use virtualenv from Mozilla

Comment 11

[Jens Müller (:tessarakt)]

Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

Do you prefer the added line with or without a / at the end?

Comment 12

[Mark Banner (:standard8)]

Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

This looks fine to me, I did have an idea about building mozmill into the actual virtualenv that m-c hooks up, but that's definitely a different bug, and re-using the m-c virtualenv is a big win.

Thanks.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/36be1bd38b3c