694625 - talos should consume mozprocess

Status: RESOLVED DUPLICATE of bug 923770

(Testing :: Talos, defect, P1)

Description

[Jeff Hammel]

Much of Talos, including but not limited to ffprocess.py, is a
duplication of the generic mozprocess:

https://github.com/mozautomation/mozmill/tree/master/mozprocess

As much of this functionality as possible should be upstreamed and
Talos should consume mozprocess.

Comment 1

[Jeff Hammel]

Once the talos code uses mozprocess, we need to figure out how to land this. There are at least two paths forward:

1. For talos == a python package, the solution is pretty simple: add `mozprocess== someversion` to the setup.py `install_requires`
2. For talos.zip, we can write a script that packages the talos subdirectory (http://hg.mozilla.org/build/talos/file/tip/talos) with mozprocess modules (http://hg.mozilla.org/build/talos/file/tip/talos) as a subdirectory of that into the talos.zip file. We will also have to include mozinfo.py but since that is a single file, that's not really that bad.

Note that while 2 works fine for now by taking advantage of how python imports things, it will work less fine if we have a more complex chain of dependencies (say, if Talos used mozrunner and mozprofile, the former consuming the latter and mozprocess).  We need a real strategy going forward to have Talos consume python, but need to make sure we have all of the use-cases before moving to something more drastic.

However, I am not sure if 1+2 cover all of the cases here.  :bear, I believe there are some hg checkouts....for remote testing?  Anything else? Do you have any idea how to deploy python package dependencies for this/these scenarios?

Comment 2

[Jeff Hammel]

See also bug 698898. If mozrunner goes in first, mozprocess won't need to be used correctly, except possibly utilities from it (e.g. mozprocess.pid). I am going to labor under the assumption that this is not the case for the time being.

Comment 3

[Jeff Hammel]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=675688

Comment 4

[Jeff Hammel]

I also remember rumours of bcontroller.py being shelled out to for mysterious windows reasons.  Does anyone know if this is the case?  Is there any documented case for this?

Comment 5

[William Lachance (:wlach)]

(In reply to Jeff Hammel [:jhammel] from comment #4)
> I also remember rumours of bcontroller.py being shelled out to for
> mysterious windows reasons.  Does anyone know if this is the case?  Is there
> any documented case for this?

Yes, I wrote about this a bit on the newsgroup:

http://groups.google.com/group/mozilla.tools/browse_thread/thread/9161393769f4e65d#

You can find the original rationale for the decision here:

https://bugzilla.mozilla.org/show_bug.cgi?id=480413#c13

Comment 6

[Jeff Hammel]

As far as practical implementation, we can't meaningfully rely on mozprocess (in the python-dependency sense) for production until mozharness, etc, is in production and we can install talos in a virtualenv.  For the time being, adding mozprocess to the methodology for create-talos-zip for bug 707218 seems a reasonable approach.

There is a major caveat in that mozprocess requires mozinfo.  While it is, on the surface tempting to put the four files for mozprocess directly in the 'talos' subdirectory, note that this requires a different import path than if mozprocess was installed from python setup.py develop:  `from processhandler import ProcessHandler` vs. `from mozprocess import ProcessHandler` (or `from mozprocess.processhandler import ProcessHandler`). So this is a non-starter.  Since mozinfo.py is one file, we will just mirror it (again) into the mozprocess subdirectory.  While this is far from ideal, the whole create-talos-zip is just an interim solution that will ultimately go away anyway.

Comment 7

[Jeff Hammel]

see also: http://groups.google.com/group/mozilla.tools/browse_thread/thread/9161393769f4e65d?pli=1

While I was actively working on this in Q4, I'm unsure when I will be able work on this in Q1