Closed Bug 1188891 Opened 9 years ago Closed 9 years ago

make sure we're not downloading moz* modules in automation

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: aus)

References

Details

Attachments

(1 file)

I happened to notice that a bunch of other modules crack the top 100 downloaded pypi modules (http://pypi-ranking.info/week). Dustin pointed out the very large upswing it downloades around March or April on this chart: http://pypi-ranking.info/module/mozrunner

Given that, and the fact that most of them are clumped together with almost exactly the same number of downloads, it seems pretty likely that something in automation is downloading them. I thought RelEng machinery was firewalled off from pypi, but that may not be the case. It could also be Taskcluster jobs that are doing it. I had a cursory look at a few b2g tests, and they seemed to pull from our own mirrors.

cc'ing some folks who may know.
[root@bld-lion-r5-002.build.releng.scl3.mozilla.com ~]# nc -vz pypi.python.org 443
Connection to pypi.python.org 443 port [tcp/https] succeeded!

so it could be from Buildbot jobs
I would imagine tests would be downloading more from pypi than builds.
True, but I don't think we restrict either one.
In general, we have --find-links specified on all mozharness jobs and in any case, we default (or used to) to the packages inside of the tests.zip.

Is there a way to determine which IP ranges are routing to pypi.python.org? (From all Mozilla networks or S3 ones we own)
Or, can we raise the firewall for few minutes? (even per pool)

If we look at one of the graphs [1], something changed in the last 6 months that made us go from less than 0.5M downloads to 2.5M.

[1] http://pypi-ranking.info/module/mozinfo
could do some log grepping to see if any of the public pypi links show up there
I wonder if on-device B2G automation is responsible for this.  Dylan, who owns that automation that we can ask?
Flags: needinfo?(doliver)
Johan can probably help here.
Flags: needinfo?(doliver) → needinfo?(jlorenzo)
It looks like we're using pypi from gaia-master: https://treeherder.mozilla.org/#/jobs?repo=gaia-master

E.g.,

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/WGh74_DSRYCnCrlvRs-Ccw/1/public/logs/live_backing.log

Installed /home/worker/git_checkout/node_modules/marionette-js-runner/venv/lib/python2.7/site-packages/mozprofile-0.27-py2.7.egg
Searching for mozrunner>=6.6
Reading https://pypi.python.org/simple/mozrunner/
Best match: mozrunner 6.9
Downloading https://pypi.python.org/packages/source/m/mozrunner/mozrunner-6.9.tar.gz#md5=e5be7b9d9b09575b14a4439aaa8c151b

This does not occur with the same jobs on gecko branches; there we use the bundled version in tests.zip.  Because of the large numbers of chunks on gaia-master, this probably explains most of the apparent requests.

The difference is that we don't use mozharness on gaia-master, but do on gecko branches.  Gareth, is there any way we can switch to using mozharness for gaia-master, or at least our internal pypi?
Flags: needinfo?(gaye)
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> I wonder if on-device B2G automation is responsible for this.

Our Jenkins instance downloads marionette_driver and marionette_client directly from pypi. But, as far as I know, we haven't changed this configuration since (at least) 2014. Asking Dave Hunt to confirm.
Flags: needinfo?(jlorenzo) → needinfo?(dave.hunt)
The on-device tests are using PyPI:

Searching for marionette-client==0.17
Reading https://pypi.python.org/simple/marionette_client/
Best match: marionette-client 0.17
Downloading https://pypi.python.org/packages/source/m/marionette_client/marionette_client-0.17.tar.gz#md5=f45c6ced461a6abea8b9910759326ac5

The packages should at least be cached on the Jenkins infrastructure to avoid repeatedly downloading them, but it looks like that's not working. It's also possible for us to switch to a mirror, however we'll need to make sure the mirror has all of the required versions of packages and is kept up to date when dependencies change.
Flags: needinfo?(dave.hunt)
(In reply to Dave Hunt (:davehunt) from comment #10)
> The on-device tests are using PyPI:
> 
> Searching for marionette-client==0.17
> Reading https://pypi.python.org/simple/marionette_client/
> Best match: marionette-client 0.17
> Downloading
> https://pypi.python.org/packages/source/m/marionette_client/
> marionette_client-0.17.tar.gz#md5=f45c6ced461a6abea8b9910759326ac5
> 
> The packages should at least be cached on the Jenkins infrastructure to
> avoid repeatedly downloading them, but it looks like that's not working.
> It's also possible for us to switch to a mirror, however we'll need to make
> sure the mirror has all of the required versions of packages and is kept up
> to date when dependencies change.

RelEng uses: http://pypi.pub.build.mozilla.org/pub/

If you can get ssh access, you can use https://hg.mozilla.org/build/braindump/file/default/utils/publish_package_our_pypi.sh to publish packages, otherwise I believe RelEng can do it for you (via buildduty).
He's right you know :)  And you know what would make a great Intern/Outreachy project?  Writing a RelengAPI component that can automatically update that directory, so we don't require SSH permissions.  If doing or mentoring that sounds interesting, get in touch with me.
:aus knows a little bit more about the TC setup
Flags: needinfo?(gaye) → needinfo?(aus)
Oh, I think this is probably because we're omitting setting this env var on the gaia jobs -- PIP_FIND_LINKS and PIP_TRUSTED_HOST (to avoid the whining...)

Pull request is up and testing right now. Should be resolved shortly. Thanks for catching this. :)
Assignee: nobody → aus
Status: NEW → ASSIGNED
Flags: needinfo?(aus)
For some reason, the env vars are not getting used when using a setup script. Bummer. :( I'm not great at this python thing so a little more info would be appreciated.

I don't think moving to mozharness is realistic in the short term, so getting pip (when called from a setup script!) to use our mirror would be best right now.
I'm looking in here:
https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=fa84b42d001ace397772cbf021e78c5c86550879

This log:
https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/0okRcgUZRbKcLU5UmzpPkw/0/public/logs/live_backing.log

It seems that this is the bash script called to set up the venv:
> ./scripts/setup_python.sh

Is this the file?
https://github.com/mozilla-b2g/gaia/blob/666d89197d4f3688f705b451ef9c18917d624cee/tests/jsmarionette/runner/marionette-js-runner/scripts/setup_python.sh

You can probably set the variables before calling it in here:
https://github.com/mozilla-b2g/gaia/blob/666d89197d4f3688f705b451ef9c18917d624cee/tests/jsmarionette/runner/marionette-js-runner/scripts/setup_python.sh#L69

You can also use --no-index --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host http://pypi.pub.build.mozilla.org/pub

You should also add pip freeze at the end to easily track what got installed.

It would also be ideal if you had a requirements.txt with fixed versions of the packages (Use "==" instead of ">=").
That way you won't have packages change under your feet when they're released.

You can also use "set -ex" in the bash script to show the commands being executed (to help debuggability).
You can do so before line 66.

Notice that the internal pypi will have missing packages which we can upload for you.
At that point just re-trigger a job on Treeherder would be sufficient (no need to push again).
:armenzg -- Unfortunately that doesn't seem to work. It would seem that python setup scripts don't actually call pip OR that there is no way to influence how pip works. I tried command line args to python itself, to virtualenv, environment vars for all of them and a pip.conf file in the venv root to no avail.

I'm guessing mozharness has a set list of python packages it installs and that it calls pip manually, yes? That would certainly seem to be the only way to avoid this problem.
Flags: needinfo?(armenzg)
What are the steps to reproduce this outside of automation?
I would like to try it on my side.
I think I can just call the bash script and see. I will try tomorrow.

I did once use Bash to set up an virtual environment [1] and instead of using source venv/bin/activate I used venv/bin/pip.
There was a reason I just can't remember why.

Also using set -ex would help us know exactly what happens within the bash script.

Could you also point us to logs or code of what you tried?

[1] https://hg.mozilla.org/build/braindump/file/default/community/setup_buildbot_environment.sh#l71
WRT to pip not being used.

I believe we would need to change this call:
> python setup.py develop
to:
> pip install -r requirements.txt (if there is one)

I think this is the runner's setup.py:
https://github.com/mozilla-b2g/gaia/blob/9dfedd7d35da00fa9b08dcffc3ab4f47da54e4f0/tests/jsmarionette/runner/marionette-js-runner/host/python/runner-service/setup.py

You would need a requirements.txt file in there with this content:
blessings==1.6
mozcrash==0.15
mozdevice==0.46
mozfile==1.2
mozinfo==0.8
mozlog==3.0
moznetwork==0.27
mozprocess==0.22
mozprofile==0.27
mozrunner==6.10

This is what gets installed with `pip install mozrunner mozprofile`

I don't think you need all those variables on your patch. Let me know if you would like me to review it.
`source venv/bin/activate` is only for interactive use; any kind of automation using a virtualenv should call paths in the virtualenv's bin directory directly.  Note that to run python, do the same, e.g., `venv/bin/python setup.py develop`
Flags: needinfo?(armenzg)
Commit (master): https://github.com/mozilla-b2g/gaia/commit/6272c9da1e61337e38da0272117deb36d2d70e7b

Fixed.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
(In reply to Aus Lacroix [:aus] from comment #22)
> Commit (master):
> https://github.com/mozilla-b2g/gaia/commit/
> 6272c9da1e61337e38da0272117deb36d2d70e7b
> 
> Fixed.

Shouldn't some kind of caching be enabled too?

Also, are we sure that was the only place this occurred? Perhaps we should reopen until someone checks the pypi stats in a week or so.
(See also the findings in bug 1172744)
Depends on: 1211972
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: