Closed Bug 869609 Opened 11 years ago Closed 11 years ago

mail: Virtualenv failing on multi-arch Debian-based distros

Categories

(Thunderbird :: Testing Infrastructure, defect)

Product:
Thunderbird
Component:
Testing Infrastructure
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 24.0

People

(Reporter: tessarakt, Assigned: tessarakt)

References

Details

Attachments

(1 file, 2 obsolete files)

Use virtualenv from Mozilla
205.85 KB, patch
standard8
: review+
Details | Diff | Splinter Review 
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:22.0) Gecko/20130505 Firefox/22.0
Build ID: 20130505004014

Steps to reproduce:

Tried to build recent comm-central.


Actual results:

Build failed with the following log entries:

rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools...................................
  Complete output from command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg:
  Traceback (most recent call last):
  File "<string>", line 279, in <module>
  File "<string>", line 240, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 125, in finalize_options
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1121, in _expand
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill/resources/virtualenv/virtualenv_support/setuptools-0.6c11-py2.7.egg/setuptools/command/install.py", line 32, in finalize_options
  File "/usr/lib/python2.7/distutils/command/install.py", line 321, in finalize_options
    (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
  File "/home/jens/devel/mozilla-comm-central/obj-mail/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py", line 88, in sysconfig_get_config_vars
    real_vars = old_get_config_vars(*args)
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 489, in get_config_vars
    func()
  File "/usr/lib/python2.7/distutils/sysconfig.py", line 433, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2098, in <module>
    main()
  File "virtualenv/virtualenv.py", line 928, in main
    never_download=options.never_download)
  File "virtualenv/virtualenv.py", line 1040, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 593, in install_setuptools
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv/virtualenv.py", line 567, in _install_req
    cwd=cwd)
  File "virtualenv/virtualenv.py", line 1006, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jens/devel/moz...irtualenv/bin/python -c "#!python
\"\"\"Bootstra...sys.argv[1:])






" /home/jens/devel/moz...ols-0.6c11-py2.7.egg failed with error code 1
Failure to install virtualenv



Expected results:

Build should have been successful.
See Also: → 837631
I replaced the contents of mail/test/resources/virtualenv with those of mozilla/python/virtualenv.

Apparently this works now. Here the relevant part of the logfile:

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mailnews/test/fakeserver/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -R /home/jens/devel/mozilla-comm-central/mail/base/test/unit/resources/* ../../../mozilla/_tests/mozmill/resources
/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Overwriting /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/lib/python2.7/distutils/__init__.py with new content
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...
make[5]: Leaving directory '/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mail/test/mozmill'


Is this ok? Or is there anything else I have to care about?
Attachment #751641 - Flags: review?(bugspam.Callek)
Wouldn't it be better to just use the code from m-c?
Probably. That needs to be done by someone who understands the build system.
Attachment #751641 - Flags: review?(bugspam.Callek)
Attachment #751650 - Flags: review?(bugspam.Callek)
(In reply to Jens Müller from comment #4)
> Probably. That needs to be done by someone who understands the build system.

I guess in mail/resources/installmozmill.py the following bits need to be changed:

  # check for existence of necessary files
  if not os.path.exists('virtualenv'):
    print "File not found: virtualenv"
    sys.exit(1)

and

  returncode = call([sys.executable, os.path.join('virtualenv', 'virtualenv.py'),
                     destination], env=env)

Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the correct syntax?
(In reply to Jens Müller from comment #6)
> Is "os.path.join('..', '..', '..', 'mozilla', 'python', 'virtualenv')" the
> correct syntax?

Apparently, it is more complex than that: The script is executed in the obj directory. And the mozilla/python/virtualenv is not copied to the obj dir.
OK, should not be too difficult. Apparently, the following block in mail/test/mozmill/Makefile.in needs to be changed.

# Copy MozMill and its dependencies over, and set up a virtualenv. The
# virtualenv directory is outside because we don't want to bundle it up during
# stage-package.
VIRTUALENV_DIR = $(_DEST_DIR)/../mozmill-virtualenv
mozmill-virtualenv: NSDISTMODE=copy
mozmill-virtualenv:
	$(INSTALL) $(topsrcdir)/mail/test/resources $(_DEST_DIR)
	rm -rf $(VIRTUALENV_DIR) && \
	mkdir $(VIRTUALENV_DIR) && \
	unset MACOSX_DEPLOYMENT_TARGET && \
	$(PYTHON) $(_DEST_DIR)/resources/installmozmill.py $(VIRTUALENV_DIR)

I will try that later on.
Attachment #751650 - Flags: review?(bugspam.Callek)
OK, that one worked as well. I'll prepare the patch now.

/home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/config/nsinstall -t /home/jens/devel/mozilla-comm-central/mail/test/resources ../../../mozilla/_tests/mozmill
rm -rf ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        mkdir ../../../mozilla/_tests/mozmill/../mozmill-virtualenv && \
        unset MACOSX_DEPLOYMENT_TARGET && \
        /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python ../../../mozilla/_tests/mozmill/resources/installmozmill.py ../../../mozilla/_tests/mozmill/../mozmill-virtualenv
Python: 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3]
Using real prefix '/usr'
New python executable in /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin/python
Installing setuptools............done.
Installing pip...............done.
Unpacking ./ManifestDestiny
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/ManifestDestiny
    
Unpacking ./simplejson-2.1.6
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/simplejson-2.1.6
    
Unpacking ./mozrunner
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozrunner
    
Unpacking ./jsbridge
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
Unpacking ./mozmill
  Running setup.py egg_info for package from file:///home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill/resources/mozmill
    
Installing collected packages: ManifestDestiny, simplejson, mozrunner, jsbridge, mozmill
  Running setup.py install for ManifestDestiny
    
    Installing manifestparser script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for simplejson
    building 'simplejson._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c simplejson/_speedups.c -o build/temp.linux-x86_64-2.7/simplejson/_speedups.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/simplejson/_speedups.o -o build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    
  Running setup.py install for mozrunner
    
    Installing mozrunner script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing addon_id script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for jsbridge
    
    warning: no files found matching '*' under directory 'jsbridge/xpi'
    Installing jsbridge script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
  Running setup.py install for mozmill
    
    Installing mozmill-thunderbird script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill-restart script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
    Installing mozmill script to /home/jens/devel/mozilla-comm-central/obj-x86_64-unknown-linux-gnu/mozilla/_tests/mozmill-virtualenv/bin
Successfully installed ManifestDestiny simplejson mozrunner jsbridge mozmill
Cleaning up...

        Attachment #751650 -
        Attachment is obsolete: true

        Attachment #751670 -
        Flags: review?(bugspam.Callek)

    
  

        Attachment #751670 -
        Flags: review?(bugspam.Callek)

    
    

    
  
Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

Do you prefer the added line with or without a / at the end?

    
  

        Attachment #751670 -
        Flags: review?(mbanner)

    
  
OS: All → Linux
Assignee: nobody → blog

    
    

    
  
Comment on attachment 751670 [details] [diff] [review]
Use virtualenv from Mozilla

This looks fine to me, I did have an idea about building mozmill into the actual virtualenv that m-c hooks up, but that's definitely a different bug, and re-using the m-c virtualenv is a big win.

Thanks.

        Attachment #751670 -
        Flags: review?(mbanner) → review+
Status: UNCONFIRMED → NEW
Ever confirmed: true

    
  
Keywords: checkin-needed

    
  
Component: Build Config → Testing Infrastructure
Product: MailNews Core → Thunderbird

    
    

    
  
https://hg.mozilla.org/comm-central/rev/36be1bd38b3c
Status: NEW → RESOLVED
Closed: 11 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 24.0

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: