1443506 - Failure to create virtualenv leading to failed builds in some circumstances

Status: NEW

(Firefox Build System :: General, defect, P5)

Description

[Alex Gaynor [:Alex_Gaynor]]

I'm not quite sure what the steps to reproduce this are -- it's happening consistently for me when attempting do an ARM Android build from an x86 Linux host.

     0:00.66 Automatically clobbering /home/osboxes/mozilla-central/obj-arm-linux-androideabi
     0:00.66 Successfully completed auto clobber.
     0:00.66 Adding make options from /home/osboxes/mozilla-central/mozconfig
        AUTOCLOBBER=1
        CONFIG_GUESS=arm-linux-androideabi
        MOZ_MAKE_FLAGS=-j4
        MOZ_OBJDIR=/home/osboxes/mozilla-central/obj-arm-linux-androideabi
        OBJDIR=/home/osboxes/mozilla-central/obj-arm-linux-androideabi
        FOUND_MOZCONFIG=/home/osboxes/mozilla-central/mozconfig
        export FOUND_MOZCONFIG
     0:00.67 /usr/bin/make -f client.mk MOZ_PARALLEL_BUILD=4 -s configure
     0:00.69 make: Warning: File 'client.mk' has modification time 307244 s in the future
     0:00.74 cd /home/osboxes/mozilla-central/obj-arm-linux-androideabi
     0:00.74 /home/osboxes/mozilla-central/configure
     0:00.98 Creating Python environment
     0:05.46 New python executable in /home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/bin/python2.7
     0:05.46 Also creating executable in /home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/bin/python
     0:05.46 Installing setuptools, pip, wheel...done.
     0:06.00 running build_ext
     0:06.00 copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
     0:06.01 copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
     0:06.01
     0:06.01 Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
     0:06.01 Reexecuting in the virtualenv
     0:06.44 Creating Python environment
     0:06.83 Please use the *system* python to run this script
     0:06.83 Traceback (most recent call last):
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 2325, in <module>
     0:06.83     main()
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 711, in main
     0:06.83     symlink=options.symlink and hasattr(os, 'symlink')) # MOZ: Make sure we don't use symlink when we don't have it
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 924, in create_environment
     0:06.83     site_packages=site_packages, clear=clear, symlink=symlink))
     0:06.83   File "/home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/lib/python2.7/posixpath.py", line 360, in abspath
     0:06.83     if not isabs(path):
     0:06.83   File "/home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/lib/python2.7/posixpath.py", line 54, in isabs
     0:06.83     return s.startswith('/')
     0:06.84 AttributeError: 'NoneType' object has no attribute 'startswith'
     0:06.84 Traceback (most recent call last):
     0:06.84   File "/home/osboxes/mozilla-central/configure.py", line 127, in <module>
     0:06.84     sys.exit(main(sys.argv))
     0:06.84   File "/home/osboxes/mozilla-central/configure.py", line 29, in main
     0:06.84     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
     0:06.84   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 399, in run
     0:06.84     self.include_file(path)
     0:06.84   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 390, in include_file
     0:06.84     exec_(code, self)
     0:06.84   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/util.py", line 59, in exec_
     0:06.84     exec(object, globals, locals)
     0:06.85   File "/home/osboxes/mozilla-central/moz.configure", line 7, in <module>
     0:06.85     include('build/moz.configure/init.configure')
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 694, in include_impl
     0:06.85     self.include_file(what)
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 390, in include_file
     0:06.85     exec_(code, self)
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/util.py", line 59, in exec_
     0:06.85     exec(object, globals, locals)
     0:06.85   File "/home/osboxes/mozilla-central/build/moz.configure/init.configure", line 207, in <module>
     0:06.85     @imports('distutils.sysconfig')
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 677, in decorator
     0:06.85     depends = DependsFunction(self, func, dependencies, when=when)
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 96, in __init__
     0:06.85     sandbox._value_for(self)
     0:06.85   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 474, in _value_for
     0:06.86     return self._value_for_depends(obj, need_help_dependency)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/util.py", line 944, in method_call
     0:06.86     cache[args] = self.func(instance, *args)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 483, in _value_for_depends
     0:06.86     return obj.result(need_help_dependency)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/util.py", line 944, in method_call
     0:06.86     cache[args] = self.func(instance, *args)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 123, in result
     0:06.86     return self._func(*resolved_args)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/configure/__init__.py", line 1003, in wrapped
     0:06.86     return new_func(*args, **kwargs)
     0:06.86   File "/home/osboxes/mozilla-central/build/moz.configure/init.configure", line 255, in virtualenv_python
     0:06.86     manager.build(python)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/virtualenv.py", line 439, in build
     0:06.86     self.create(python)
     0:06.86   File "/home/osboxes/mozilla-central/python/mozbuild/mozbuild/virtualenv.py", line 203, in create
     0:06.86     'Failed to create virtualenv: %s' % self.virtualenv_root)
     0:06.86 Exception: Failed to create virtualenv: /home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv
     0:06.87 *** Fix above errors and then restart with\
     0:06.87                "/usr/bin/make -f client.mk build"
     0:06.87 client.mk:145: recipe for target 'configure' failed
     0:06.87 make: *** [configure] Error 1

Comment 1

[Alex Gaynor [:Alex_Gaynor]]

The root cause here appears to be a VM filesystem with messed up mtimes.

It'd be nice if mach build handled this situation more cleanly.

Comment 2

[Gregory Szorc [:gps]]

If mtimes can't be trusted, virtualenv management failing will be the least of your concerns. I'd expect the build system to give inconsistent results.

Anyway, the underlying exception seems to be in virtualenv itself:

0:06.44 Creating Python environment
     0:06.83 Please use the *system* python to run this script
     0:06.83 Traceback (most recent call last):
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 2325, in <module>
     0:06.83     main()
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 711, in main
     0:06.83     symlink=options.symlink and hasattr(os, 'symlink')) # MOZ: Make sure we don't use symlink when we don't have it
     0:06.83   File "/home/osboxes/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 924, in create_environment
     0:06.83     site_packages=site_packages, clear=clear, symlink=symlink))
     0:06.83   File "/home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/lib/python2.7/posixpath.py", line 360, in abspath
     0:06.83     if not isabs(path):
     0:06.83   File "/home/osboxes/mozilla-central/obj-arm-linux-androideabi/_virtualenv/lib/python2.7/posixpath.py", line 54, in isabs
     0:06.83     return s.startswith('/')
     0:06.84 AttributeError: 'NoneType' object has no attribute 'startswith'

Although this might be in code that we've hacked up in virtualenv. Ugh.

Or are we passing "None" into virtualenv somehow and causing it to error?

Anyway, this feels like a low-priority bug. I doubt any build peers will look at it unless others complain.