736086 - Make all of mozbase pip installable? setup_development.py inadequate for installation

Status: RESOLVED INCOMPLETE

(Testing :: Mozbase, defect)

Description

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

I would like to either

$ pip install mozbase
or 
$ pip install -e  git://github.com/mozilla/mozbase.git#egg=mozbase

to make it easier to reuse components in other projects (Test Pilot admin stuff, in my case)

I tend to like `requirements.txt` and (pip freeze).   Current strategy of "clone, run setup_development" makes this hard.

Comment 1

[Jeff Hammel]

maybe setup_development.py could spit out a setup.py file for upload to pypi?  I'm not sure the entireity of the problem context here.

Comment 2

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

As described, neither of the usual pip install methods work for mozbase.  It would be nice to get mozbase installed all at once.  I am not sure which part of this is confusing or unclear, so please elaborate!

Pip can install from github just fine, usually.  It would be fine to see all the pieces be uploaded to PyPi as well, but a bit annoying to have to remember all their names.

Comment 3

[Jeff Hammel]

We can make a setup.py that points to the pypi versions of all of the mozbase packages; i don't think it is possible to make a setup.py that points to the github versions of the mozbase packages since the subdirectories are not available as tarballs

Comment 4

[Jeff Hammel]

Attachment: sample code

Comment 5

[Jeff Hammel]

So there is good news and bad news.

The good news is, there is a standard entry to setup.py, package_dir, that may be used to map complex directory structures to packages.  I've uploaded an example setup.py that installs mozinfo if you use `python setup.py install` (obviously, in practice, all of the packages would be filled out and installed).  See http://docs.python.org/distutils/setupscript.html , although I'll admit the documentation leaves something to be desired

The bad news is....this doesn't work for python setup.py develop.  See e.g. https://bitbucket.org/tarek/distribute/issue/177/setuppy-develop-doesnt-support-package_dir .  Instead it silently fails (that is, it looks like it succeeds, but you won't be able to import mozinfo after running the script).

Maybe this is okay for this purpose?  If the goal of the package is to be able to install from the mozbase repo via a http(s) URL and not for development, this is probably the way to go.  Likewise, it would be fine for making pypi releases.  But it will not work for `python setup.py develop`.

It also won't work for development in general which brings me to the complicated news, but I will do that in a separate comment.

Comment 6

[Jeff Hammel]

Which brings me to the complicated news...

While its easy to imagine extending attachment 606887 [details] to cover the other mozbase packages, there are a few interesting notes related to dependencies.  Firstly, I did nothing for external dependencies.  These will need to be gathered from the various mozbase packages and included, otherwise the installed packages will not work properly and fail on import errors.

Secondly, several mozbase packages depend on other mozbase packages:

mozInstall: ['mozinfo']
mozprocess: ['mozinfo']
mozprofile: ['ManifestDestiny >= 0.5.4']
mozrunner: ['mozinfo', 'mozprocess', 'mozprofile >= 0.1']

Care will need to be taken to list the packages in the proper order or, installation via the metapackage setup.py may fail.  This is in fact why setup_development.py was written, to get this ordering right.

The good news is that setup_development.py already has the infrastructure to do both of these things.  *IFF* this is desired and all the caveats of comment 5 are understood, I would advise the following:  have setup_development.py have a command line flag that generates a setup.py for the mozbase meta-package.  This can be checked into github and will be installable from there.  (We'll also need a MANIFEST.in and other details.) We should also check in the resultant setup.py that the `develop` flag is not passed, as it will not work.

It might be a good idea to revisit the versioning discussion as part of this.  I would propose extending setup_development.py (which may need to be renamed) to bump versions and generate a new (version-bumped) mozbase setup.py each time a sub-package was bumped (we could also make it do git tags, etc).  I have more thoughts here, but want to make sure we're all on the same page here.

So do we want to do this?  Its not rocket science, but OTOH there will be a lot of corner cases.  I'm willing to take this on if people want this and the caveats from comment 5 are understood (if you don't understand the various caveats in this comment, don't worry; I'm short-handing and its mostly packaging esoterica, but comment 5 has practical implications).

Comment 7

[William Lachance (:wlach)]

(In reply to Jeff Hammel [:jhammel] from comment #6)

> So do we want to do this?  Its not rocket science, but OTOH there will be a
> lot of corner cases.  I'm willing to take this on if people want this and
> the caveats from comment 5 are understood (if you don't understand the
> various caveats in this comment, don't worry; I'm short-handing and its
> mostly packaging esoterica, but comment 5 has practical implications).

What you propose sounds okish (I'm not 100% clear on some of the details e.g. "We should also check in the resultant setup.py that the `develop` flag is not passed, as it will not work."). 

I guess the question is how much it buys us. Personally I find the combination of pip-installable individual modules + setup_development.py has worked well enough for our purposes (this bug is the first I've heard of a problem). How much time would be saved by doing this? Is it worth the development time and additional complexity?

Comment 8

[Jeff Hammel]

(In reply to William Lachance (:wlach) from comment #7)
> (In reply to Jeff Hammel [:jhammel] from comment #6)
<snip/>
> I guess the question is how much it buys us. Personally I find the
> combination of pip-installable individual modules + setup_development.py has
> worked well enough for our purposes (this bug is the first I've heard of a
> problem). How much time would be saved by doing this? Is it worth the
> development time and additional complexity?

I think this depends on who is "us". I don't think it is a time-saving measure, but an accessibility measure.  I also have found the existing system to work for my needs, but I have not only commit access to the mozbase respository, I also have ownership of the pypi packages, so if I am working on downstream software that has mozbase dependencies, I can bump the appropriate versions and release them.  I personally don't see much need for a metapackage.

That said, mozbase users may not have access to mozbase or pypi.  There is no current way of depending on trunk mozrunner (e.g.). Even with this fix, you cannot depend on trunk mozrunner -- only on mozbase, which IMHO is not a great way of doing things.  But is it better than what we have? :shrug: I don't know.  I would be more in favor of a development model where users/contributors would ask for new releases when they were needed and we would promptly respond.  On the other hand, I would much rather have people using mozbase (for appropriate problems) than not using mozbase and rolling their own.  So the real question is would doing this result in something user/contributors, such as Gregg Lind, want and result in a more positive experience for them and make them more engaged as mozbase users?  That I cant really speak to.  I would hate to turn off potential mozbase users because you can't use repository versions of these packages via e.g. pip requirements or easy_install.  A metapackage is not particularly a good way, but it is the only way I can think of doing this.

Comment 9

[Jeff Hammel]

Anyone have additional thoughts on this bug?  I'm considering a WONTFIX.  We've also considered splitting mozbase into submodules which should solve this issue

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I am 50/50 on this.  It would be really nice to do 'easy_install mozbase' and call it good.  But that makes it harder to split up development,etc..  For example working on mozdevice which might change more frequently than other 'modules', it would be frustrating working on such a larger 'module'.

The real question is, how many pieces exist in mozbase?  If we continue to add more and these pieces depend on each other, I would rather make it a single install.

Comment 11

[Jeff Hammel]

As per comment 5 and comment 6, this isn't about making mozbase a singular monolithic package, but rather having a front-end installer to all of the subpackages which can be done with a setup.py (caveats aside). Or we could use submodules.  I'm going to close this for now as it doesn't seem like anyone has any time or interest in making this work.  Really, people should be using the pypi releases which we've gotten a lot better at releasing more frequently, the should use the whole git clone if they *really* need the bleeding edge version, or if there is a feature they need for production they should poke us and ask us to fix and release. If there is any interest in reopening, please do so with comments as to what the problem actually is that isn't addressed here.

Comment 12

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: sample setup.py that could live at pypi

Comment 13

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

So, I tried this again, and most of my complaint is that pip fails *so opaquely* with no hint of what to do next.  After looking through setup_develop, it seems like a lot of what it is doing feels like the dependency checking that pip already does.

Even a 'shadow' package at pypi for 'mozbase' that just gets the others (like in the attached setup.py) would go a long way toward making this more usable.  

In the case of trying to install the 'local' versions, a failout with the suggested command:

   #to install each from local

   for x in [...]; python setup.py blah . ....  

As it stands now, none of it feels very pythonic, and feels much more like how NumPy and other `easy_install` stuff feels, which I don't mean a compliment :)

Comment 14

[Jeff Hammel]

So if we go with a setup.py it should be generated and it should probably be done with package_dir as done above.  I can do this, but we should probably talk about this on the tools list to ensure that we're all on the same page.

Comment 15

[Jeff Hammel]

See (and participate) in the discussion at http://groups.google.com/group/mozilla.tools/browse_thread/thread/8f4bc14c0b6dca79#

Comment 16

[Jeff Hammel]

So given the lack of activity here, I'm going to close as invalid.  If there are other concerns, please reopen.

As for 

"""
As it stands now, none of it feels very pythonic, and feels much more like how NumPy and other `easy_install` stuff feels, which I don't mean a compliment :)
"""

NumPy is of course a special case as it is not only compiled but relies on several dependencies.  I'm not 100% defending it, but `pip install NumPy` would be difficult even in the best of cases.

As for "other `easy_install` stuff", I'm not 100% sure what you mean here.  Both easy_install, pip, and distribute rely (currently) on the same format for setup.py files, which was written with setuptools and has many good things and many bad things that are out of scope here.

While I personally agree that having one repository for a group of packages is a minus there are a few factors here:

1. :carljm, one of the pypa folks, is working on having pip be able to deal with subpaths for e.g. git.  If this finishes, we could have a requirements.txt in mozbase and call it a day, AIUI.

2. Outside of separating all packages, which is out of scope here and a discussion which I don't particularly intend to start again, having a mozbase setup.py would allow you to install all of the packages in "install" (vs "develop") mode. Since trunk/HEAD/tip/whatever-the-cool-kids-call-it-these-days is intended for developers, I find this use-case much less compelling.

3. We (endeavor) to release good stable versions to pypi.  IMHO, the appropriate thing to do is communicate with mozbase maintainers (particularly :wlach and myself) wrt to getting to versions out.  Since our policy is that trunk should be stable, releasing new versions should be pretty straightforward.  Good software (and good software ecosystems) == communication.  The lack of further communication here is one reason I am resolving the bug.

As said before, making a setup.py that installs all of mozbase from HEAD not in develop mode is doable.  Not trivial, but doable.  If there is a usecase for the above, I am happy to invest the time to do it.  But I don't see this use-case, so I am closing.

In any case, I am (personally, I know others feel differently) having a 'mozbase' package on pypi.  Mozbase is several packages.  These are released to pypi and can be consumed as the discrete software packages that they are.  The way I see it, mozbase == a development effort to develop core testing-needed modules that happen to live in the same repo, not a swiss-army-knife package.

If someone wanted to take up the effort and have a setup.py that is dynamically generated from the mozbase directory structure, I would support them.  That said, I have no time to do it, no need, and I have bigger fish to fry.

Comment 17

[Jeff Hammel]

changing title so i can actually find this bug