Last Comment Bug 869609 - mail: Virtualenv failing on multi-arch Debian-based distros
: mail: Virtualenv failing on multi-arch Debian-based distros
Status: RESOLVED FIXED
:
Product: Thunderbird
Classification: Client Software
Component: Testing Infrastructure (show other bugs)
: Trunk
: All Linux
: -- normal (vote)
: Thunderbird 24.0
Assigned To: Jens Müller (:tessarakt)
:
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-07 12:24 PDT by Jens Müller (:tessarakt)
Modified: 2013-06-09 16:19 PDT (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv. (218.72 KB, patch)
2013-05-20 04:15 PDT, Jens Müller (:tessarakt)
no flags Details | Diff | Splinter Review
Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv. (1.22 MB, patch)
2013-05-20 05:19 PDT, Jens Müller (:tessarakt)
no flags Details | Diff | Splinter Review
Use virtualenv from Mozilla (205.85 KB, patch)
2013-05-20 06:26 PDT, Jens Müller (:tessarakt)
standard8: review+
Details | Diff | Splinter Review

Description Jens Müller (:tessarakt) 2013-05-07 12:24:53 PDT
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:22.0) Gecko/20130505 Firefox/22.0
Build ID: 20130505004014

Steps to reproduce:

Tried to build recent comm-central.


Actual results:

Build failed with the following log entries:

rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools...................................
  Complete output from command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg:
  Traceback (most recent call last):
  File "<string>", line 279, in <module>
  File "<string>", line 240, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 125, in finalize_options
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1121, in _expand
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 32, in finalize_options
  File "/usr/lib/python2.7/distutils/command/install.py", line 321, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py", line 88, in sysconfig_get_config_vars
    real_vars = old_get_config_vars(*args)
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 489, in get_config_vars
    func()
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 433, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2098, in <module>
    main()
  File "virtualenv/virtualenv.py", line 928, in main
    never_download=options.never_download)
  File "virtualenv/virtualenv.py", line 1040, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 593, in install_setuptools
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 567, in _install_req
    cwd=cwd)
  File "virtualenv/virtualenv.py", line 1006, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg failed with error code 1
Failure to install virtualenv



Expected results:

Build should have been successful.
Comment 1 Jens Müller (:tessarakt) 2013-05-20 04:06:38 PDT
I replaced the contents of mail/test/resources/virtualenv with those of mozilla/python/virtualenv.

Apparently this works now. Here the relevant part of the logfile:

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/fakeserver/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mail/base/test/unit/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Overwriting /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py with new content
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...
make[5]: Leaving directory '/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mail/test/mozmill'


Is this ok? Or is there anything else I have to care about?
Comment 2 Jens Müller (:tessarakt) 2013-05-20 04:15:06 PDT
Created attachment 751641 [details] [diff] [review]
Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv.
Comment 3 :Ms2ger (⌚ UTC+1/+2) 2013-05-20 05:10:37 PDT
Wouldn't it be better to just use the code from m-c?
Comment 4 Jens Müller (:tessarakt) 2013-05-20 05:15:29 PDT
Probably. That needs to be done by someone who understands the build system.
Comment 5 Jens Müller (:tessarakt) 2013-05-20 05:19:20 PDT
Created attachment 751650 [details] [diff] [review]
Replaced the contents of mail/test/resources/virtualenv with the contents of mozilla/python/virtualenv.
Comment 6 Jens Müller (:tessarakt) 2013-05-20 05:50:11 PDT
(In reply to Jens Müller from comment #4)
> Probably. That needs to be done by someone who understands the build system.

I guess in mail/resources/installmozmill.py the following bits need to be changed:

  # check for existence of necessary files
  if not os.path.exists('virtualenv'):
    print "File not found: virtualenv"
    sys.exit(1)

and

  returncode = call([sys.executable, os.path.join('virtualenv', 'virtualenv.py'),
                     destination], env=env)

Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the correct syntax?
Comment 7 Jens Müller (:tessarakt) 2013-05-20 06:08:01 PDT
(In reply to Jens Müller from comment #6)
> Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the
> correct syntax?

Apparently, it is more complex than that: The script is executed in the obj directory. And the mozilla/python/virtualenv is not copied to the obj dir.
Comment 8 Jens Müller (:tessarakt) 2013-05-20 06:11:47 PDT
OK, should not be too difficult. Apparently, the following block in mail/test/mozmill/Makefile.in needs to be changed.

# Copy MozMill and its dependencies over, and set up a virtualenv. The
# virtualenv directory is outside because we don't want to bundle it up during
# stage-package.
VIRTUALENV_DIR = $(_DEST_DIR)/../mozmill-virtualenv
mozmill-virtualenv: NSDISTMODE=copy
mozmill-virtualenv:
	$(INSTALL) $(topsrcdir)/mail/test/resources $(_DEST_DIR)
	rm -rf $(VIRTUALENV_DIR) && \
	mkdir $(VIRTUALENV_DIR) && \
	unset MACOSX_DEPLOYMENT_TARGET && \
	$(PYTHON) $(_DEST_DIR)/resources/installmozmill.py $(VIRTUALENV_DIR)

I will try that later on.
Comment 9 Jens Müller (:tessarakt) 2013-05-20 06:18:27 PDT
OK, that one worked as well. I'll prepare the patch now.

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...
Comment 10 Jens Müller (:tessarakt) 2013-05-20 06:26:18 PDT
Created attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla
Comment 11 Jens Müller (:tessarakt) 2013-05-20 06:55:50 PDT
Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

Do you prefer the added line with or without a / at the end?
Comment 12 Mark Banner (:standard8, afk until Dec) 2013-05-21 03:42:50 PDT
Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

This looks fine to me, I did have an idea about building mozmill into the actual virtualenv that m-c hooks up, but that's definitely a different bug, and re-using the m-c virtualenv is a big win.

Thanks.
Comment 13 Ryan VanderMeulen [:RyanVM] 2013-05-22 06:55:59 PDT
https://hg.mozilla.org/comm-central/rev/36be1bd38b3c

Note You need to log in before you can comment on or make changes to this bug.