556562 - Old mozrunner .egg in site-packages causes AttributeError: 'module' object has no attribute 'FirefoxProfile'

Status: RESOLVED WONTFIX

(Add-on SDK Graveyard :: General, defect, P3)

Description

[Austin King [:ozten]]

Steps:
1) Install hg trunk SDK or jetpack-sdk 2.0
2) source bin/activate
3) cfx testall

Expected:
All tests run and pass

Actual:
AttributeError: 'module' object has no attribute 'FirefoxRunner'
Full details:
http://pastebin.mozilla.org/712133

Root cause:
Thanks to Atul -
python -c "import mozrunner; print mozrunner.__path__"
['/Library/Python/2.5/site-packages/mozrunner-1.0-py2.5.egg/mozrunner']

I have installed mozmill (and many other Mozilla products).
I moved this egg out of /Library/Python/2.5/site-packages and cfx testall works properly.

Comment 1

[Austin King [:ozten]]

Discussion (for reference, all details are in this bug)
http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/9e726faeaaf918af

Comment 2

[Atul Varma [:atul]]

Alas! The bin/activate script sets PYTHONPATH, which apparently is put at the end of sys.path rather than the beginning, so any previously-installed mozrunner takes precedence over the SDK's bundled version.

The only way I can think of changing this is by mucking with the sys.path at the beginning of each script, ugh.

Is there any other way to use environment variables to prepend a path to the beginning of sys.path?

Comment 3

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Clint: any thoughts on how to fix this?

Comment 5

[Jeff Hammel]

I'm not sure there is a way around this -- its a python/virtualenv thing not a mozrunner thing.  Recommendation for avoiding this situation: never install e.g. mozmill (etc) in site-packages

Comment 6

[cmtalbert]

(In reply to comment #4)
> Clint: any thoughts on how to fix this?
The problem here is that the addon-sdk ships with a customized, already created virtualenv.  That's a great thing and it should (in theory) save you from this exact problem.

However, if you have any of the python tools that jetpack uses in your system wide site-packages of python, then the virtualenv that jetpack uses will pick up those tools rather than using the ones in its python-libs directory.

WORKAROUND:
Note, this is slightly destructive because it overwrites the standard bin/activate script, which is customized for jetpack:
(assuming you have virtualenv installed:)
1. cd addon-sdk
2. virtualenv --no-site-packages .
3. source bin/activate
4. Run your stuff.

This will force virtualenv to use what you've got inside the directory rather than looking in the system site-package area.  I'll diff the activate scripts that are generated this way with the one that addon-sdk ships with, maybe there will be something there. In the meantime, if you need a quick way to get on your feet, the above workaround should serve (tested on linux).

Comment 7

[Jeff Hammel]

So if I understand correctly, the already created virtualenv is checked in.  I'm not sure if this is a contributing factor to this problem, but its certainly an unusual thing to do.  

A few things:
- I'm not sure what customizations are in this virtualenv. I believe I've heard that its not particularly the same as the stock version.  I would recommend against this strategy.  There is virtualenvwrapper: http://www.doughellmann.com/projects/virtualenvwrapper/ ; not sure if that is helpful here? (I don't know the problem space)
- there is poor man's virtualenv: PYTHONHOME; I actually hate this variable, but it + some duct-tape and chicken wire can be used to accomplish most of what virtualenv does
- and again, installing user-land software in system site-packages is probably bad
- checking in a virtualenv will not work across platforms

I'd be happy working towards a better solution, but I'm not really sure what the problem being solved is.

Comment 8

[Myk Melez [:myk] [@mykmelez]]

Hmm, it sounds like the long term solution is to implement some alternative to having a checked-in virtualenv.

Comment 9

[Atul Varma [:atul]]

One option is to put virtualenv.py in the resource, along with a bootstrap script that automatically bootstraps the virtualenv for us. I do this at the beginning of htmlpad's manage.py here:

  https://github.com/hackasaurus/htmlpad/blob/master/manage.py

The advantage here is that it relieves end-users of having to know what a python virtualenv is, which is particularly important for the add-on SDK, as it's made for add-on developers, not python developers.

It should also be noted that the thing being shipped with the add-on SDK is not actually a virtualenv in the sense of virtualenv.py--it's actually a "generic" virtual environment created for the add-on SDK, which adds a custom directory to PYTHONPATH. It was created to be relocatable and cross-platform, and pretty much succeeds in almost every way that doesn't involve this bug.

Comment 10

[Brian Warner [:warner :bwarner]]

FWIW, I've seen a similar problem in other projects (Tahoe-LAFS), in which .egg files installed in a system directory take precedence over python packages found on $PYTHONPATH. I consider it a bug (others consider it a feature) in the way setuptools' .pth file modifies sys.path: it puts the .egg files that it finds at the *start* of sys.path, overriding the user's preferences expressed in $PYTHONPATH. The only workaround I've found has been to remove the .egg file, but the setuptools approach would probably suggest putting the SDK's mozrunner code in an egg-shaped directory (with .egg-info and a real version number), then import it with pkg_resources.require("mozrunner=VERSION") so that it gets the exact version.

Comment 11

[Ian Bicking (:ianbicking)]

I'd highly recommend making custom scripts, so long as you don't actually need to expose importable modules (which in this case I believe is the case).

In this case say you put all your libraries in lib/vendor/ (e.g., using pip install --install-option="--install-libs=$(pwd)/lib/vendor").  Then the script (say bin/cfx) does:

import os, sys
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
lib_dir = os.path.join(base, 'lib', 'vendor')
if not os.path.exists(lib_dir):
    sys.stderr.write('Cannot find %s (did you move the cfx script?)\n' % lib_dir)
    sys.exit(1)
sys.path.insert(0, lib_dir)
... normal script invocation stuff ...


This keeps system packages from overriding anything.  Note if you have .pth files in lib/vendor/ they won't get loaded, and if you use site.addsitedir(lib_dir) they will be added to sys.path, but at the end.  So just avoid .pth files (note though that easy_install does create .pth files, but pip does not unless you use pip -e).

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(Pushing all open bugs to the --- milestone for the new triage system)

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

can we get instructions for removing the other version of mozrunner as we suggest?  Not very usable to get pointed to a bug without a clear solution.  Maybe a wiki would be better!

Comment 16

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #15)
> can we get instructions for removing the other version of mozrunner as we
> suggest?  Not very usable to get pointed to a bug without a clear solution. 
> Maybe a wiki would be better!

This is an instance of the general case "How do you uninstall a python package?" which, sadly, python (or setuptools, or blame whoever else you want) did not really think ahead too much.  There is `pip uninstall`, which may work.  Googling "removing python packages" or variants gives a wealth of poorly organized knowledge.  The short answer is:

1. Find where it is installed.  If its globally, this might be something like /usr/lib/python2.6/site-packages  or a few other variants
2. Remove the .egg file or directory or .egg-link that corresponds to your package.
3. Edit e.g. easy-install.pth in that directory to remove it from there as well

Hard? No. Annoying? Yes. Scriptable?  Well, people have tried

I'm happy to go more into detail about this but I'm guessing a bug comment really isn't the place, so if you want to know more mail me or ping me on #ateam

Comment 17

[arky [:arky]]

Darn! The only way I could get rid of this problem is to delete the mozbase directory from my computer.

Comment 18

[Hallvord R. M. Steen [:hallvors]]

So, it's not possible to have a working installation of Mozilla Marionette *and* a working installation of the add-on SDK?

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, if you are going to install several tools which depend on mozrunner, you should create a venv for each of those. Otherwise if you need Marionette to be installed beside the mozrunner version you depend on in the Add-on SDK, you should check if both are dependent on the same versions of mozrunner.

Marionette:
http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/requirements.txt => mozrunner >= 5.15

Addon SDK:
http://mxr.mozilla.org/mozilla-central/source/addon-sdk/source/python-lib/mozrunner/ => mozrunner ??

As it looks like the SDK is using a very outdated version of mozrunner. So get this upgraded or as expressed above use different virtual environments. You can't have both versions of mozrunner installed at the same time.

Comment 20

[yt]

I'm commenting because I just spent 2 days on this problem and the workaround seems to be outdated (or didn't work for me at least). Just in case any other noobs like myself are out there struggling with this, I figured I'd go ahead and post what just worked for me.

I'm using a Mac which already had mozrunner and addon-sdk installed from previous stuff. The first problem I ran into was that testing addon-sdk from my repo (where I was working on a bug patch) wasn't working because the tests kept being auto-directed to the addon-sdk pack inside my Cellar directory. When I uninstalled addon-sdk from Cellar and source bin/activated in my repo, I was getting a warning message about mozrunner being installed in Library/Python/2.7/site-packages. I tried to pip uninstall mozrunner but got permission denied. If I tried to run the tests I got "AttributeError: 'module' object has no attribute 'profile'"

So here's what I did: 
1. build a VirtualMachine (Ubuntu) http://timtaubert.de/blog/2014/04/a-ready-to-use-virtual-build-environment-for-firefox/ 
2. on your VM, install firefox nightly
3. clone addon-sdk
4. source bin/activate
5. cfx test

Now it all works.

Comment 21

[Hallvord R. M. Steen [:hallvors]]

My workaround is to use the add-ons SDK on a different computer from the one I'm usually running Marionette stuff on.. :-/

Comment 22

[Erik Vold [:erikvold] (please needinfo? me)]

Sorry we won't be releasing any new versions of cfx, jpm is the replacement https://www.npmjs.com/package/jpm