759836 - setup_development.py doesnt always actually setup for development

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

https://github.com/mozilla/mozbase/blob/master/setup_development.py
*should* change into each directory, in dependency order, and run
`python setup.py develop`.  For whatever reason, this does not always
suffice:

(mozmill)│./setup_development.py 
running develop
running egg_info
writing mozhttpd.egg-info/PKG-INFO
writing top-level names to mozhttpd.egg-info/top_level.txt
writing dependency_links to mozhttpd.egg-info/dependency_links.txt
writing entry points to mozhttpd.egg-info/entry_points.txt
reading manifest file 'mozhttpd.egg-info/SOURCES.txt'
writing manifest file 'mozhttpd.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozhttpd.egg-link
(link to .)
mozhttpd 0.3 is already the active version in easy-install.pth
Installing mozhttpd script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozhttpd
Processing dependencies for mozhttpd==0.3
Finished processing dependencies for mozhttpd==0.3
running develop
running egg_info
writing mozinfo.egg-info/PKG-INFO
writing top-level names to mozinfo.egg-info/top_level.txt
writing dependency_links to mozinfo.egg-info/dependency_links.txt
writing entry points to mozinfo.egg-info/entry_points.txt
reading manifest file 'mozinfo.egg-info/SOURCES.txt'
writing manifest file 'mozinfo.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozinfo.egg-link
(link to .)
Removing mozinfo 0.3.3 from easy-install.pth file
Adding mozinfo 0.3.3 to easy-install.pth file
Installing mozinfo script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozinfo
Processing dependencies for mozinfo==0.3.3
Finished processing dependencies for mozinfo==0.3.3
running develop
running egg_info
writing mozdevice.egg-info/PKG-INFO
writing top-level names to mozdevice.egg-info/top_level.txt
writing dependency_links to mozdevice.egg-info/dependency_links.txt
writing entry points to mozdevice.egg-info/entry_points.txt
reading manifest file 'mozdevice.egg-info/SOURCES.txt'
writing manifest file 'mozdevice.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozdevice.egg-link
(link to .)
mozdevice 0.2 is already the active version in easy-install.pth
Installing sut script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozdevice
Processing dependencies for mozdevice==0.2
Finished processing dependencies for mozdevice==0.2
running develop
running egg_info
writing mozlog.egg-info/PKG-INFO
writing top-level names to mozlog.egg-info/top_level.txt
writing dependency_links to mozlog.egg-info/dependency_links.txt
reading manifest file 'mozlog.egg-info/SOURCES.txt'
writing manifest file 'mozlog.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozlog.egg-link
(link to .)
mozlog 1.0 is already the active version in easy-install.pth

Installed /home/jhammel/mozmill/src/mozbase/mozlog
Processing dependencies for mozlog==1.0
Finished processing dependencies for mozlog==1.0
running develop
running egg_info
writing requirements to mozprocess.egg-info/requires.txt
writing mozprocess.egg-info/PKG-INFO
writing top-level names to mozprocess.egg-info/top_level.txt
writing dependency_links to mozprocess.egg-info/dependency_links.txt
writing entry points to mozprocess.egg-info/entry_points.txt
reading manifest file 'mozprocess.egg-info/SOURCES.txt'
writing manifest file 'mozprocess.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozprocess.egg-link
(link to .)
Removing mozprocess 0.1b2 from easy-install.pth file
Adding mozprocess 0.1b2 to easy-install.pth file

Installed /home/jhammel/mozmill/src/mozbase/mozprocess
Processing dependencies for mozprocess==0.1b2
Searching for mozinfo==0.3.3
Best match: mozinfo 0.3.3
Processing mozinfo-0.3.3-py2.6.egg
Removing mozinfo 0.3.3 from easy-install.pth file
Adding mozinfo 0.3.3 to easy-install.pth file
Installing mozinfo script to /home/jhammel/mozmill/bin

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/mozinfo-0.3.3-py2.6.egg
Finished processing dependencies for mozprocess==0.1b2
running develop
running egg_info
writing ManifestDestiny.egg-info/PKG-INFO
writing top-level names to ManifestDestiny.egg-info/top_level.txt
writing dependency_links to
ManifestDestiny.egg-info/dependency_links.txt
writing entry points to ManifestDestiny.egg-info/entry_points.txt
reading manifest file 'ManifestDestiny.egg-info/SOURCES.txt'
writing manifest file 'ManifestDestiny.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/ManifestDestiny.egg-link
(link to .)
Removing ManifestDestiny 0.5.4 from easy-install.pth file
Adding ManifestDestiny 0.5.4 to easy-install.pth file
Installing manifestparser script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/manifestdestiny
Processing dependencies for ManifestDestiny==0.5.4
Finished processing dependencies for ManifestDestiny==0.5.4
running develop
running egg_info
writing requirements to mozprofile.egg-info/requires.txt
writing mozprofile.egg-info/PKG-INFO
writing top-level names to mozprofile.egg-info/top_level.txt
writing dependency_links to mozprofile.egg-info/dependency_links.txt
writing entry points to mozprofile.egg-info/entry_points.txt
reading manifest file 'mozprofile.egg-info/SOURCES.txt'
writing manifest file 'mozprofile.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozprofile.egg-link
(link to .)
mozprofile 0.3.2 is already the active version in easy-install.pth
Installing mozprofile script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozprofile
Processing dependencies for mozprofile==0.3.2
Searching for ManifestDestiny==0.5.4
Best match: ManifestDestiny 0.5.4
Processing ManifestDestiny-0.5.4-py2.6.egg
Removing ManifestDestiny 0.5.4 from easy-install.pth file
Adding ManifestDestiny 0.5.4 to easy-install.pth file
Installing manifestparser script to /home/jhammel/mozmill/bin

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/ManifestDestiny-0.5.4-py2.6.egg
Finished processing dependencies for mozprofile==0.3.2
running develop
running egg_info
writing requirements to mozInstall.egg-info/requires.txt
writing mozInstall.egg-info/PKG-INFO
writing top-level names to mozInstall.egg-info/top_level.txt
writing dependency_links to mozInstall.egg-info/dependency_links.txt
writing entry points to mozInstall.egg-info/entry_points.txt
reading manifest file 'mozInstall.egg-info/SOURCES.txt'
writing manifest file 'mozInstall.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozInstall.egg-link
(link to .)
mozInstall 0.3 is already the active version in easy-install.pth
Installing mozinstall script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozinstall
Processing dependencies for mozInstall==0.3
Searching for mozinfo==0.3.3
Best match: mozinfo 0.3.3
Processing mozinfo-0.3.3-py2.6.egg
mozinfo 0.3.3 is already the active version in easy-install.pth
Installing mozinfo script to /home/jhammel/mozmill/bin

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/mozinfo-0.3.3-py2.6.egg
Finished processing dependencies for mozInstall==0.3
running develop
running egg_info
writing requirements to mozrunner.egg-info/requires.txt
writing mozrunner.egg-info/PKG-INFO
writing top-level names to mozrunner.egg-info/top_level.txt
writing dependency_links to mozrunner.egg-info/dependency_links.txt
writing entry points to mozrunner.egg-info/entry_points.txt
reading manifest file 'mozrunner.egg-info/SOURCES.txt'
writing manifest file 'mozrunner.egg-info/SOURCES.txt'
running build_ext
Creating
/home/jhammel/mozmill/lib/python2.6/site-packages/mozrunner.egg-link
(link to .)
mozrunner 5.5 is already the active version in easy-install.pth
Installing mozrunner script to /home/jhammel/mozmill/bin

Installed /home/jhammel/mozmill/src/mozbase/mozrunner
Processing dependencies for mozrunner==5.5
Searching for mozprofile==0.3.2
Best match: mozprofile 0.3.2
mozprofile 0.3.2 is already the active version in easy-install.pth
Installing mozprofile script to /home/jhammel/mozmill/bin

Using /home/jhammel/mozmill/src/mozbase/mozprofile
Searching for mozprocess==0.1b2
Best match: mozprocess 0.1b2
Processing mozprocess-0.1b2-py2.6.egg
Removing mozprocess 0.1b2 from easy-install.pth file
Adding mozprocess 0.1b2 to easy-install.pth file

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/mozprocess-0.1b2-py2.6.egg
Searching for mozinfo==0.3.3
Best match: mozinfo 0.3.3
Processing mozinfo-0.3.3-py2.6.egg
mozinfo 0.3.3 is already the active version in easy-install.pth
Installing mozinfo script to /home/jhammel/mozmill/bin

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/mozinfo-0.3.3-py2.6.egg
Searching for ManifestDestiny==0.5.4
Best match: ManifestDestiny 0.5.4
Processing ManifestDestiny-0.5.4-py2.6.egg
ManifestDestiny 0.5.4 is already the active version in
easy-install.pth
Installing manifestparser script to /home/jhammel/mozmill/bin

Using
/home/jhammel/mozmill/lib/python2.6/site-packages/ManifestDestiny-0.5.4-py2.6.egg
Finished processing dependencies for mozrunner==5.5

(mozmill)│python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozprocess
>>> mozprocess.__file__
'/home/jhammel/mozmill/lib/python2.6/site-packages/mozprocess-0.1b2-py2.6.egg/mozprocess/__init__.pyc'

Comment 1

[Carl Meyer (:carljm)]

Here's what's happening:

1. There is already an egg version of mozprocess installed in this virtualenv.
2. When setup_development.py runs "python setup.py develop" for mozprocess, it successfully removes the egg path from easy-install.pth and replaces it with the path to the source checkout, but it doesn't remove the actual egg itself (setuptools never removes eggs, it only removes references to them in easy-install.pth).
3. Later when setup_development.py runs "python setup.py develop" for mozrunner, when searching for the mozprocess dependency it finds the egg file in site-packages and somehow, inexplicably, decides that this egg is a preferable way to satisfy the dependency than the already-in-place develop install of mozprocess. So then it removes the source checkout path to mozprocess from easy-install.pth and replaces it with the egg path! And does the same for the dependency on mozinfo.

So yes, this is a straight-up bug in setuptools. Fortunately, there's a simple fix; since setup_development.py is already manually handling the dependency chain (higher-level aside: why? :P), you can just add the "--no-deps" flag to each "setup.py develop" call, and it will fix this problem. 

This solution will break if any of the components of mozbase have external PyPI dependencies that setup_development.py does not explicitly handle. If that's the case, you might consider abandoning setup_development.py entirely in favor of pip.

Comment 2

[Carl Meyer (:carljm)]

Attachment: the simple fix; tell setuptools not to worry about dependencies

Comment 3

[Jeff Hammel]

Sadly, we do have pypi dependencies :( So it won't be quite this easy.  Thanks for the detective work, carljm.  This at least clarifies what is going on ...setuptools grumble grumble...

We currently don't require pip for development, fwiw, but I'd like to talk about that direction going forward

Comment 4

[Carl Meyer (:carljm)]

After a bit of experimentation, it turns out pip also needs a little more work to handle this use case correctly. I can do that work if you decide to move in that direction, but for the moment I think the simplest solution would be to use --no-deps and additionally have setup_development.py handle other external dependencies explicitly.

Comment 5

[Jeff Hammel]

Yeah, that sounds like the best plan to me too.  Shouldn't be too difficult

Comment 6

[Jeff Hammel]

Attachment: fix with external deps

Comment 7

[Jeff Hammel]

Attachment: forgot .items()

Comment 8

[Carl Meyer (:carljm)]

In the long run, I think it'd be better to make the necessary tweaks/improvements to reusable tools like pip so they can handle these cases, rather than ad-hoc reimplementations in things like setup_development.py. But for now, this works fine!

Comment 9

[Jeff Hammel]

pushed: https://github.com/mozilla/mozbase/commit/b7ed86653ae25978d830e5f27109951b92e7ad62

In general, I'd like to move to a simpler system.  Note that the next thing I'm going to do is to write a versioning script which will use some of setup_development.py.  So I think of what we have now as a system in flux which needs to work but isn't necessarily supposed to be best practices (yet).

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jeff, do we need the same for Mozmill?

Comment 11

[Jeff Hammel]

(In reply to Henrik Skupin (:whimboo) from comment #10)
> Jeff, do we need the same for Mozmill?

It would be a nice to have, yes.  The script certainly suffers from the same setuptools bug. That said, I've seen it a lot less there for whatever reasons