734235 - Track mozbase dependencies: Create dependency graph for mozbase-based projects

Status: RESOLVED WONTFIX

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

Talos, Mozmill, and possibly other projects use MozBase packages for
dependencies.  In addition, Mozbase packages depend on other mozbase
packages for dependencies.  A graph should be made that details these
dependencies letting one know what versions are being used and aiding
in version bumping.  This graph should be linked to from
https://wiki.mozilla.org/Auto-tools/Projects/MozBase

Comment 1

[Jeff Hammel]

WIP:
* http://k0s.org/mozilla/mozbase/mozbase-usage.json
* http://k0s.org/mozilla/mozbase/dependencies.html

Comment 2

[William Lachance (:wlach)]

FWIW, Eideticker uses mozdevice, mozprofile, and mozhttpd

Comment 3

[Jeff Hammel]

(In reply to William Lachance (:wlach) from comment #2)
> FWIW, Eideticker uses mozdevice, mozprofile, and mozhttpd

Thanks, updated

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Also you missed from the email thread that mozmill-automation makes use of mozinfo and manifestdestiny.

Comment 5

[Jeff Hammel]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> Also you missed from the email thread that mozmill-automation makes use of
> mozinfo and manifestdestiny.

Thanks, updated.  It might be a good idea to add a setup.py that lists these dependencies.  If I ever make an automated system that updates this list and chart, it will require a setup.py to properly determine this information.

Comment 6

[cmtalbert]

this is so cool, thanks Jeff!!

Comment 7

[Jeff Hammel]

So I'm going to leave this bug open as this would be very nice to do dynamically.  http://k0s.org/mozilla/mozbase/mozbase-usage.json is just a hand-written JSON file.  Instead, we should gather the information from setup.py files.  We can know with confidence mozbase's setup.pys due to its directory structure.  What depends on mozbase will still be a matter of manual updating: in general, a software package has no real way of knowing what uses it.  But in any case, given a list of consumer setup.pys we should be able to make a dynamic service.

I'm not sure the best way of gathering information from setup.py.  Traditionally, I have run `python setup.py egg_info` (which only works with setuptools, but then again so does `install_requires`) and reading the generated information. However, this is (fairly) slow.  So if we're doing this once day/hour this is fine, but if we're doing this as a webservice this is a bit slow.  So I was thinking of monkey-patching setuptools.setup and loading the information directly.  This isn't 100% reliable, but should work for our cases (I hope).  Any thoughts?

This will also require that all things that we care about have setup.py files.  Currently this is not the case.  Maybe worth blocking on?  Until then we have the manual JSON file...

Comment 8

[Jeff Hammel]

If we're going to get serious about mozbase this quarter, we'll need to have something live. I'm not going to start on this straight away though the sooner its done, the sooner we can reap the benefits

Comment 9

[Jeff Hammel]

https://bugzilla.mozilla.org/show_bug.cgi?id=852235#c2

Comment 10

[Jeff Hammel]

This is still outstanding;  I would estimate maybe 16-24 manhours to get this done and stable.  I am very willing to do this if this is desired; the solution should also be portable to other mozilla python library code (that is, code with setup.pys anyway).

Comment 11

[Julien Pagès (:parkouss)]

Hi,

Mozregression uses currently:

mozcommitbuilder, mozfile, mozprofile, mozrunner, mozinstall, mozinfo, mozlog, mozversion and mozdevice.

Also, not sure this is interesting here but pip now have a "show" command to list installed packages:

For example, running this command in the _virtualenv dir of mozilla-central:

pip show mozrunner
---
Name: mozrunner
Version: 6.4
Location: /home/jp/dev/mozilla-central/testing/mozbase/mozrunner
Requires: mozcrash, mozdevice, mozfile, mozinfo, mozlog, mozprocess, mozprofile

Comment 12

[Jeff Hammel]

I would doubt there is much need for this ticket.  Feel free to close.  I am unsubscribing.

Comment 13

[Andrew Halberstadt [:ahal]]

Yeah, in-tree dependencies are easy enough to search for in dxr and there really isn't any way of knowing out of tree dependencies. I'm +1 for closing, though if anyone disagrees feel free to re-open.