671724 - add mozmill dependencies to test pool

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

I need:

# mercurial 1.9 will break
pip install http://mercurial.selenic.com/release/mercurial-1.7.3.tar.gz
pip install mozmill

on all test boxes for bug 498425 and bug 588398.
Since mercurial is a specific revision, we may be best served creating a virtualenv ? (not sure how this will interact with buildbot + mozharness)

I'd like this rolled out across at least one of each staging talos platform, so I can get mozmill running across nightly+release updates while verifying that this is sufficient.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Does it need to be in a particular location, or is that flexible?

Comment 2

[Aki Sasaki (not active)]

If it's a virtualenv, or if the libraries are otherwise in my python path, I don't particularly care where they are.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #0)
> pip install mozmill

You shouldn't install the latest version of Mozmill from pypi. That's something which has already been caused problems because we also push beta releases there. Using a fixed version would be preferred. For now the latest version of Mozmill we are using and which is compatible up to 8.0a1 is 1.5.4b6.

> on all test boxes for bug 498425 and bug 588398.
> Since mercurial is a specific revision, we may be best served creating a
> virtualenv ? (not sure how this will interact with buildbot + mozharness)

So you are using Mercurial 1.9 for everything else? In this case please file a bug under MozillaQA/Mozmill Automation and I will take care of it.

> I'd like this rolled out across at least one of each staging talos platform,
> so I can get mozmill running across nightly+release updates while verifying
> that this is sufficient.

Release should work without issues. For daily builds we probably would have to tweak the download script if you want to use it.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to comment #2)
> If it's a virtualenv, or if the libraries are otherwise in my python path, I
> don't particularly care where they are.

Well, how do you execute it - virtualenvs don't change PYTHONPATH, but are based on the path to the executable you run.  If we build a virtualenv at /tools/mozmill, you run /tools/mozmill/bin/mozmill (or whatever the mozmill executable is called) and it will run in the context of the virtualenv.

(In reply to comment #3)
> You shouldn't install the latest version of Mozmill from pypi. That's
> something which has already been caused problems because we also push beta
> releases there. Using a fixed version would be preferred. For now the latest
> version of Mozmill we are using and which is compatible up to 8.0a1 is
> 1.5.4b6.

Good to know.  We have a local cache of Python sdists, and virtualenvs specify specific versions.  Should I just use the 1.5.4b6 off pypi?

> > on all test boxes for bug 498425 and bug 588398.
> > Since mercurial is a specific revision, we may be best served creating a
> > virtualenv ? (not sure how this will interact with buildbot + mozharness)
> 
> So you are using Mercurial 1.9 for everything else? In this case please file
> a bug under MozillaQA/Mozmill Automation and I will take care of it.

I'll still use 1.7.3 in the virtualenv, for the moment.

Comment 5

[Aki Sasaki (not active)]

(In reply to comment #4)
> (In reply to comment #2)
> > If it's a virtualenv, or if the libraries are otherwise in my python path, I
> > don't particularly care where they are.
> 
> Well, how do you execute it - virtualenvs don't change PYTHONPATH, but are
> based on the path to the executable you run.  If we build a virtualenv at
> /tools/mozmill, you run /tools/mozmill/bin/mozmill (or whatever the mozmill
> executable is called) and it will run in the context of the virtualenv.

Hm, mozharness is going to call a mozmill-automation script that imports mozmill.
The first two repos are going to be in the workdir, but mozmill will be in this virtualenv. I can experiment with activating a venv in mozharness or in buildbot before calling mozharness.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Don't think of "activating" a virtualenv - the activate command is just syntactic sugar to make a virtualenv easy to use from an interactive shell.

A virtualenv is a python binary + libraries - you will make yourself sad if you try to use the libraries in a virtualenv with a different python binary, or try to use a virtualenv's python binary to run something not in the virtualenv.

Probably what you want to do is run
 /tools/mozmill/bin/python mozmilla-automation-script.py
but if the script is more than just a .py file, then you may need something more complex.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #4)
> Good to know.  We have a local cache of Python sdists, and virtualenvs
> specify specific versions.  Should I just use the 1.5.4b6 off pypi?

Exactly. Just use that package from pypi. It's our one and only distribution channel (except the source repo).

Comment 8

[Aki Sasaki (not active)]

(In reply to comment #6)
> Don't think of "activating" a virtualenv - the activate command is just
> syntactic sugar to make a virtualenv easy to use from an interactive shell.
> 
> A virtualenv is a python binary + libraries - you will make yourself sad if
> you try to use the libraries in a virtualenv with a different python binary,
> or try to use a virtualenv's python binary to run something not in the
> virtualenv.
> 
> Probably what you want to do is run
>  /tools/mozmill/bin/python mozmilla-automation-script.py
> but if the script is more than just a .py file, then you may need something
> more complex.

wfm.
I added a --virtualenv option to the mozharness script.
If this is set, it calls %(virtualenv)s/bin/python; otherwise it calls python.
Tested, works.

Comment 9

[Aki Sasaki (not active)]

Added a --create-virtualenv as well; we can point the urls to internal URLs to avoid the blocking-internet-access issue.  So that should reduce the need for this bug.

However, I think mercurial requires compilation, and aiui the Talos/test boxes don't have compilation tools.

Comment 10

[Aki Sasaki (not active)]

I'm going to take this until it's clear that we're doing this on test slaves, and not build slaves. (bug 588398 comment 13).

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #9)
> However, I think mercurial requires compilation, and aiui the Talos/test
> boxes don't have compilation tools.

You could run a pure install which wouldn't compile binary components and just use the plain python code. It's a bit slower but for our usage it shouldn't make a difference.

I wonder if you have found a way to get virtualenv working without xtools installed on a Mac. That's one of our highest issues we have with Mozmill Crowd at the moment.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> However, I think mercurial requires compilation, and aiui the Talos/test
> boxes don't have compilation tools.

'Scuse the fly-by question, but why do we need to compile Mercurial? We already have it on test machines...

Comment 13

[Aki Sasaki (not active)]

(In reply to comment #12)
> (In reply to comment #9)
> > However, I think mercurial requires compilation, and aiui the Talos/test
> > boxes don't have compilation tools.
> 
> 'Scuse the fly-by question, but why do we need to compile Mercurial? We
> already have it on test machines...

Hm, I was doing a virtualenv with --no-site-packages to be clean, and needed mozmill+mercurial to be on the same virtualenv.

I suppose I could try removing the --no-site-packages.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

--no-site-packages is generally a good idea.  If nothing else, it means that you can upgrade mercurial separately for this stuff than for other stuff.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Aki Sasaki [:aki] from comment #0)
> # mercurial 1.9 will break
> pip install http://mercurial.selenic.com/release/mercurial-1.7.3.tar.gz
> pip install mozmill

This is now fixed. You no longer have to install an old version of Mercurial. I hope it helps you a bit.

Comment 16

[Aki Sasaki (not active)]

I think I got around this before getting blocked by bug 646076.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Aki, just a question for you. I'm working on an updated version of our Mozmill environment. See the project page: https://wiki.mozilla.org/QA/Automation_Services/Projects/Mozmill_Automation/Environment

It ships with everything necessary by our tests. So no setup of tools outside of it would be necessary. Is it something you would consider to use in the future? Or don't you usually use pre-configured environments?