692279 - Talos should be a python package

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[William Lachance (:wlach)]

For Eideticker, I'm thinking of creating a virtualenv which includes a videocapture python module, talos's various dependencies, mozmill's jsbridge, and maybe a few other things (along the lines of what I did for the gofaster dashboard here: https://github.com/wlach/gofaster_dashboard/blob/master/bootstrap.sh). This should reduce the amount of manual set up required to get a working environment.

Along these lines, it would be even more convenient if talos itself were a python module so that:

1. Its dependencies could be downloaded automatically.
2. Its various scripts (remotePerfConfigurator, runTests, etc.) were callable directly from the virtualenv's bin/ directory.

jhammel asked me to file a bug about this, so here it is. :) It's not super urgent or anything, just a nice-to-have to keep in mind as things are refactored.

Comment 1

[Jeff Hammel]

See also: https://bitbucket.org/jmaher/moztalos/src/6be453cffa91/setup.py

Comment 2

[Jeff Hammel]

Attachment: makes a python package

So this patch mostly adds a setup.py and moves things around, so it is a very nominal fix.  On the other hand, since all relative paths remain the same, it shouldn't affect anything else negatively.  We can take other fixes, such as console scripts, what to export, what to do with data, as separate issues if this works

Comment 3

[William Lachance (:wlach)]

Comment on attachment 569170 [details] [diff] [review]
makes a python package

Looks pretty reasonable. I would like to have a better story for how this thing is supposed to be installed/used (something like the installation instructions for mozmill's README.md: https://github.com/mozautomation/mozmill/blob/master/README.md) before applying this. 

So overall I like where this is going, but I'd suggest iterating on it a bit to make sure it works in the big picture.

Minor nit:

+      author='Alice Nodelman',
+      author_email='[e-mail]',

anode's contribution to talos is substantial, but there are many other people who deserve credit too. Also, author_email points to a private mailing list. :) How about we say author='Mozilla Foundation' and skip author_email? That's what django seems to do.

Comment 4

[Jeff Hammel]

I would prefer to add an author email.  It can be me or any tools email address, but it is nice to have a point of contact

Comment 5

[Jeff Hammel]

wrt comment 3, I didn't enumerate all the things that should be done in connection to here, but a partial list is

- add an installation script
- make the README not horrible; preferably in a way that can be mirrored/included to the wiki
- add console_scripts for the things that should be console scripts (e.g. a 'talos' script)
- remove all need for relative paths
- move the tests and configs out of the python-package level

But I would prefer to tackle these all after the patch is landed

Comment 6

[Jeff Hammel]

Attachment: updated patch with attribution

Comment 7

[William Lachance (:wlach)]

Comment on attachment 569503 [details] [diff] [review]
updated patch with attribution

Ok, as we discussed on irc, this seems like a nice step forward with no real downsides. I'd say we can commit, though I'd be interested in what jmaher thinks as well.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 569503 [details] [diff] [review]
updated patch with attribution

Review of attachment 569503 [details] [diff] [review]:
-----------------------------------------------------------------

::: setup.py
@@ +36,5 @@
> +                           '*.rdf',
> +                           '*.sqlite',
> +                           '*.svg',
> +                           '*.xml',
> +                           '*.xul', 

does this properly capture all the file data in talos.zip?  what about *.py, *.exe, *.dll, *.jpg;  these files exist in the talos directory.

Comment 9

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #8)
> Comment on attachment 569503 [details] [diff] [review] [diff] [details] [review]

> does this properly capture all the file data in talos.zip?  what about *.py,
> *.exe, *.dll, *.jpg;  these files exist in the talos directory.

The .py files are taken care of by setuptools.  The others are legitimate misses.  Thanks for the good eye!  I used minidump_stackwalk* instead of *.exe since I was missing the other minidump_stackwalks too. other than python files setuptools has to be told what you want to include when you install it and upload to pypi.

Fixed in the patch: http://k0s.org/mozilla/hg/talos-patches/rev/0c1e55232ad9

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

ideally we would use the breakpad stuff instead of including minidump_stackwalk in the talos.zip.  But until we can depend on stuff....repeat

Comment 11

[Jeff Hammel]

So I don't have permission to push to the talos repo. :jmaher could you or :wlach push please?

Comment 12

[Jeff Hammel]

My mistake, my .hgrc file was not correctly set.  Thanks, jmaher!

Pushed: http://hg.mozilla.org/build/talos/rev/2f75acc0f8f2

I'm not sure if we should communicate this change to anyone.  Basically all of the files are moved.