1172744 - Are automation jobs not caching mozrunner/mozdevice/mozprofile/... ?

Status: RESOLVED INCOMPLETE

(Testing :: Mozbase, defect)

Description

[Ed Morley [:emorley]]

I was looking at the Python 3 compatibility chart on http://python3wos.appspot.com/ - which lists PyPI packages by number of downloads - and noticed that the mozbase packages appear in the top N.

It seems like perhaps we're not caching these (either on build/test machines or on the internal mirror).

eg:
https://pypi.python.org/pypi/mozrunner 
 24997 downloads in the last day
127516 downloads in the last week
269507 downloads in the last month 

https://pypi.python.org/pypi/mozdevice
 24813 downloads in the last day
126322 downloads in the last week
263278 downloads in the last month 

https://pypi.python.org/pypi/mozprofile
 24768 downloads in the last day
126342 downloads in the last week
262167 downloads in the last month 

https://pypi.python.org/pypi/mozprocess
 24773 downloads in the last day
126158 downloads in the last week
262054 downloads in the last month 

https://pypi.python.org/pypi/mozinfo
 24777 downloads in the last day
126270 downloads in the last week
261702 downloads in the last month 

Is this expected?

To me this means:
1) We're presumably slowing down the job setup time, since we're not caching locally or on an internal mirror
2) Paying for more AWS transfer cost than we need
3) Putting extra load on PyPI unnecessarily

Comment 1

[Ed Morley [:emorley]]

Not sure if this is from buildbot or taskcluster jobs?

Comment 2

[Jonathan Griffin (:jgriffin)]

For buildbot jobs, we use the copy in tests.zip.  It's quite possible that some TC or other CI are using PyPI.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Regarding mozmill-ci I flipped to the internal mirror a couple of days ago. So we cannot be the cause for the latest download numbers.

Comment 4

[Armen [:armenzg]]

We can probably talk with Pypi and ask them to help us get some metrics.
We could also chat with Mozilla's network team and see if they can give us some metrics.

Releng mozharness scripts should be using internal pypi or packages from the tests.zip.

Comment 5

[Andrew Halberstadt [:ahal]]

Just to add to the confusion, even if we are hitting the external pypi index, pip should automatically do caching for us:
https://pip.pypa.io/en/latest/reference/pip_install.html#caching

Comment 6

[Andrew Halberstadt [:ahal]]

Which actually might point to taskcluster, as the cache wouldn't get saved in the docker image.

(In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #4)
> Releng mozharness scripts should be using internal pypi or packages from the
> tests.zip.

Armen, do you know where the internal pypi index gets specified? After a quick search I can't find any reference to "--index-url" or "PIP_INDEX_URL" anywhere.

Comment 7

[Ed Morley [:emorley]]

(In reply to Andrew Halberstadt [:ahal] from comment #6)
> Which actually might point to taskcluster, as the cache wouldn't get saved
> in the docker image.

That implies the docker setup we're using at the moment is pip installing them as part of some ENTRYPOINT or CMD script, rather than including in the Dockerfile build (which most of the time would be able to have the cached layer reused) - this seems suboptimal.

Comment 8

[Armen [:armenzg]]

We append --no-index if both pip_index and find-links are specified [1]
I can see some scripts not setting pip_index to False [2]

The only config that does not set both is (ignoring developer_config.py):
configs/android/android_panda_talos_releng.py

[1]
http://hg.mozilla.org/build/mozharness/file/7ba09340e043/mozharness/base/python.py#l214

[2]
MacAir mozharness hg:[default!] $ grep -r "pip_index" scripts/
scripts//push_apk.py:            'pip_index': True,
scripts//update_apk_description.py:            'pip_index': True,

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I did a query via MXR for pypi.python.org and found the following:

* Talos: http://mxr.mozilla.org/mozilla-central/source/testing/talos/mach_commands.py#68

Joel, can you have a look at this? Is that relevant and run on testing machines constantly?

Regarding taskcluster I wonder if Pete can help us here. Maybe there is a way to check the known task graphs for usage of pypi.python.org.

Comment 10

[Armen [:armenzg]]

In this case, searching for pypi.python.org is limiting since it is the default value if --no-index is not specified.

Comment 11

[Pete Moore [:pmoore][:pete]]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> I did a query via MXR for pypi.python.org and found the following:
> 
> * Talos:
> http://mxr.mozilla.org/mozilla-central/source/testing/talos/mach_commands.
> py#68
> 
> Joel, can you have a look at this? Is that relevant and run on testing
> machines constantly?
> 
> Regarding taskcluster I wonder if Pete can help us here. Maybe there is a
> way to check the known task graphs for usage of pypi.python.org.

The majority of the scripts that get run as part of a task live here:
https://hg.mozilla.org/mozilla-central/file/default/testing/taskcluster/scripts/builder

I don't know if a way to search through all task live logs - I think with papertrail we'll just get the docker worker logs rather than the logs from the tasks themselves - so no easy way to search for "pypi.python.org" in our logs.

We could consider disabling access to pypi.python.org from our workers via the security group. Might not be a bad thing. I believe releng block access.

I think this bug is a duplicate of bug 1188891. Not sure which to mark as a duplicate of the other, since the other one came first, but this one maybe has a bit more detailed information.

Comment 12

[Ed Morley [:emorley]]

Mass-closing old bugs I filed that have not had recent activity/no longer affect me.