805440 - Mozrunner installed from pypi when running setup_development.py

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Mark Côté [:mcote]]

When you run setup_development.py, it determines that marionette-client is a dependency (of mozb2g, I believe) that has to be installed from pypi. The script installs this first, but it relies on mozrunner, so the latter is also installed from pypi, even though it will be installed in development mode later on in the process. This is sure to cause confusion.

$ python setup_development.py 
Searching for marionette-client
Reading http://pypi.python.org/simple/marionette_client/
Reading https://wiki.mozilla.org/Auto-tools/Projects/Marionette
Best match: marionette-client 0.5.1
Downloading http://pypi.python.org/packages/source/m/marionette_client/marionette_client-0.5.1.tar.gz#md5=ff1dfcaf176ce79fa546cb622b48f7b2
Processing marionette_client-0.5.1.tar.gz
Running marionette_client-0.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-CHl_kv/marionette_client-0.5.1/egg-dist-tmp-jLFtyX
Adding marionette-client 0.5.1 to easy-install.pth file
Installing marionette script to /tmp/mozbase/bin

Installed /tmp/mozbase/lib/python2.7/site-packages/marionette_client-0.5.1-py2.7.egg
Processing dependencies for marionette-client
Searching for mozrunner==5.10
Reading http://pypi.python.org/simple/mozrunner/
Reading http://github.com/mozautomation/mozmill
Reading http://mikeal.github.com/mozrunner/
Reading http://code.google.com/p/mozrunner/
Reading https://wiki.mozilla.org/Auto-tools/Projects/MozBase
Reading https://github.com/mozilla/mozbase/tree/master/mozrunner
Best match: mozrunner 5.10
Downloading http://pypi.python.org/packages/source/m/mozrunner/mozrunner-5.10.tar.gz#md5=58c1b4121685f8ae20199b9c93c77e66

Comment 1

[Mark Côté [:mcote]]

It seems that when mozrunner is later installed in development mode, this overrides the previous regular egg installation, so this is probably not a critical bug.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jeff, this looks very similar to what I have seen on bug 804886 with mozinfo on Tuesday this week. So it's just a variation from it.

Comment 3

[Jeff Hammel]

Its a variation in that they are both setup problems.  I don't think they are related more so than that, though the original issue in bug 804886 is hard to determine based on the output available.

This is a very particular issue of a particular class of issues. To review how https://github.com/mozilla/mozbase/blob/master/setup_development.py works, it will
1. look at the dependencies for all (given) mozbase packages
2. installs dependencies from pypi
3. unrolls these dependencies in such an order that mozbase internal dependencies are installed in an order that the dependent packages are installed following their dependencies

The problem is: because we install marionette client from pypi in step 2. and then install the local mozbase, because mozrunner is a dependency of a pypi dep, it will grab mozrunner from pypi.  This *should* be overwritten when the local mozrunner is installed.  Is this the case, :mcote?

In any case, I think we can move 2. after 3. and avoid this whole problem.

Comment 4

[Jeff Hammel]

> ...it will grab mozrunner from pypi.  This *should* be overwritten when the local mozrunner is installed.  Is this the case, :mcote?

Reading comment 0 again, this does seem to be the case.

Comment 5

[Jeff Hammel]

So comment 3 is correct.  However, there is an additional problem.  Currently, mozb2g depends on marionette client which is a pypi-based dependency (i.e. it does not live in mozbase). However, marionette_client is pegged to use mozrunner == 5.10 and mozprocess == 0.5 .  This makes it impossible to have the HEAD of mozbase installed for development due to the version requirements conflicts.

If we're going to allow this pattern of things in mozbase depending on things outside of mozbase that depend on things in mozbase, then the things outside of mozbase depend on things in mozbase, we must be lenient with the versions that we require such that we don't end up making this inconsistent as is the current state.

A solution is to make marionette client use mozrunner >= mozrunner 5.10 and mozprocess >= 0.5, bump its version, release it to pypi, and bump mozb2gs version requiring it to use the new released version of marionette_client.

Comment 6

[Jeff Hammel]

:jgriffin, :ahal, :mdas, could one of you either do this or provide me pypi ownership?

Comment 7

[Jonathan Griffin (:jgriffin)]

I can do it (probably not until tomorrow), but I've added you as owner too.

Comment 8

[Jeff Hammel]

Attachment: bump version for pypi release

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 675583 [details] [diff] [review]
bump version for pypi release

Thanks Jeff.

Comment 10

[Jeff Hammel]

Pushed to inbound: http://hg.mozilla.org/integration/mozilla-inbound/rev/027bbf3ee90e
and released to pypi: http://pypi.python.org/pypi/marionette_client

Comment 11

[Jeff Hammel]

Attachment: depend on new marionette_client

Will bump version to 0.2 and re-release following this

Comment 12

[Jeff Hammel]

Not sure if we should peg mozdevice >= some_version along with this

Comment 13

[Jonathan Griffin (:jgriffin)]

Comment on attachment 675610 [details] [diff] [review]
depend on new marionette_client

thanks!  yes, we should probably >= for mozdevice too.

Comment 14

[Jeff Hammel]

Will peg to >= 0.12

Comment 15

[Jeff Hammel]

(In reply to Jonathan Griffin (:jgriffin) from comment #13)
> Comment on attachment 675610 [details] [diff] [review]
> depend on new marionette_client
> 
> thanks!  yes, we should probably >= for mozdevice too.

pushed: https://github.com/mozilla/mozbase/commit/55f09d438709202fb6d9fec8b705fed3cc4eb63f

Comment 16

[Jeff Hammel]

version bumped: https://github.com/mozilla/mozbase/commit/e627ea3de6ff80cd27823e929e4ed47298306c8d

Comment 17

[Jeff Hammel]

Attachment: install internal dependencies before external dependencies

seems to work on my system

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/027bbf3ee90e

Comment 19

[Jeff Hammel]

Sorry, I should have marked this as [leave-open]

Comment 20

[Mark Côté [:mcote]]

Comment on attachment 675642 [details] [diff] [review]
install internal dependencies before external dependencies

wfm!

Comment 21

[Jeff Hammel]

pushed: https://github.com/mozilla/mozbase/commit/1f679a8ae3ff49d1e3c96cbf3c01ad4742568e1b

Comment 22

[Jeff Hammel]

This should actually be fixed now :)

Comment 23

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/30cb831b2dc7