805091 - OS X Desktop B2G nightly builds are having trouble creating virtualenv

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

checking for python2.7... /tools/buildbot/bin/python2.7
Creating Python environment
Using real prefix '/tools/python-2.7.2/Python.framework/Versions/2.7'
New python executable in /builds/slave/m-cen-osx64-gecko-ntly/build/obj-firefox/_virtualenv/bin/python2.7
Also creating executable in /builds/slave/m-cen-osx64-gecko-ntly/build/obj-firefox/_virtualenv/bin/python
Overwriting /builds/slave/m-cen-osx64-gecko-ntly/build/obj-firefox/_virtualenv/lib/python2.7/distutils/__init__.py with new content
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/python/virtualenv/virtualenv.py", line 2427, in <module>
    main()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/python/virtualenv/virtualenv.py", line 939, in main
    never_download=options.never_download)
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/python/virtualenv/virtualenv.py", line 1042, in create_environment
    install_distutils(home_dir)
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/python/virtualenv/virtualenv.py", line 1500, in install_distutils
    writefile(os.path.join(distutils_path, '__init__.py'), DISTUTILS_INIT)
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/python/virtualenv/virtualenv.py", line 454, in writefile
    f = open(dest, 'wb')
IOError: [Errno 13] Permission denied: '/builds/slave/m-cen-osx64-gecko-ntly/build/obj-firefox/_virtualenv/lib/python2.7/distutils/__init__.py'
Traceback (most recent call last):
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/build/virtualenv/populate_virtualenv.py", line 321, in <module>
    manager.ensure()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/build/virtualenv/populate_virtualenv.py", line 75, in ensure
    return self.build()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/build/virtualenv/populate_virtualenv.py", line 258, in build
    self.create()
  File "/builds/slave/m-cen-osx64-gecko-ntly/build/build/virtualenv/populate_virtualenv.py", line 105, in create
    raise Exception('Error creating virtualenv.')
Exception: Error creating virtualenv.
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[1]: *** [obj-firefox/Makefile] Error 2
make: *** [build] Error 2
program finished with exit code 2
elapsedTime=2.763659

This is happening on different machines

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

This is caused by http://hg.mozilla.org/mozilla-central/rev/1423fc67d73e

Comment 2

[Gregory Szorc [:gps]]

wat?

IOError: [Errno 13] Permission denied: '/builds/slave/m-cen-osx64-gecko-ntly/build/obj-firefox/_virtualenv/lib/python2.7/distutils/__init__.py'

Do a clobber build. Make sure it is actually a clobber build and that obj-firefox/_virtualenv disappears completely.

If that doesn't fix it, WTF you have something really funky going on.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Greg, these are nightly builds, which should all be clobbered right at the start.

Comment 4

[Gregory Szorc [:gps]]

If they were clobbered, I don't think distutils/__init__.py would be present and need to be overwritten.

Comment 5

[Gregory Szorc [:gps]]

Looking at the delta between our in-tree virtualenv and the latest release (1.8.2), the code that resolves the path to distutils has changed. I suspect upgrading virtualenv might magically fix things.

I still contend that there is a minor difference between the configurations for B2G and Firefox that is triggering this. Who knows.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Gregory Szorc [:gps] from comment #5)
> Looking at the delta between our in-tree virtualenv and the latest release
> (1.8.2), the code that resolves the path to distutils has changed. I suspect
> upgrading virtualenv might magically fix things.
> 
> I still contend that there is a minor difference between the configurations
> for B2G and Firefox that is triggering this. Who knows.

They use the same bit of configure.in to invoke the virtualenv setup, nothing in either b2g or browser mozconfig changes anything about python, they are using the same python interpreter on the machine, both b2g and browser have both source and obj dirs on the same volume, they both clobber before starting (for nightlies), their environment from buildbot is the same, automation calls into build system the same and these builds happen on the same machines

I can't think of anything off the top of my head to explain this.  Maybe signing out a 10.7 build slave is the correct thing.  I'm going to ask for one momentarily

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

Looks like bug 805127 resolves this, though we don't really know why this was broken for b2g but not firefox.

Comment 8

[Frank Wein [:mcsmurf]]

So this bug here is fixed now? Just wondering as we (SeaMonkey project) currently have the same problem with an OS X 10.7 buildslave. But we still need to check if there aren't multiple Python versions on that machine that are conflicting with each other (for whatever reason).

Comment 9

[Frank Wein [:mcsmurf]]

Just for reference: Also see Bug 758723/Bug 763419, Thunderbird had the same problem a few month ago, they fixed this by switching the PATH env var from
PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
to
PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
(this will probably also be the solution for our SeaMonkey OS X buildslave)
The python in /tools/buildbot/bin is already a python virtualenv which broke the build. So maybe the B2G OS X nightly builds have/had the same problem.