1228544 - [firefox-ui-tests] Very slow fetches of virtualenv packages from releng PyPI mirror in SCL3 with proxxy config

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Today I have turned on mozharness support for our Firefox UI tests in our CI staging instance in SCL3. Builds on Windows and OS X seem to work pretty fine. But on Ubuntu I see a huge slowdown while installing required python packages from the RelEng PyPI mirror:

01:28:06     INFO - ##### Running create-virtualenv step.
[..]
01:53:21     INFO - ##### Running install step.

All in all the create-virtualenv step takes 25 minutes! Here an example of the installation for one of the needed packages:

01:41:12     INFO -  Ignoring indexes: https://pypi.python.org/simple/
01:41:12     INFO -  Downloading/unpacking browsermob-proxy==0.6.0 (from -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt (line 2))
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:43:12     INFO -    Using download cache from /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache/http%3A%2F%2Fpypi.pub.build.mozilla.org%2Fpub%2Fbrowsermob-proxy-0.6.0.tar.gz
01:43:12     INFO -    Running setup.py (path:/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/build/browsermob-proxy/setup.py) egg_info for package browsermob-proxy

We wait 2 minutes before using a download cache?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I assume that this is happing because pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com is used here. Creating another virtualenv outside of the mozharness script works fine as expected.

I assume that we can easily turn off the proxxy config for our tests via Jenkins?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The command as executed here looks like:

/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/bin/pip -v install --download-cache /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache --timeout 120 -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt --no-index --find-links http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub --find-links http://pypi.pub.build.mozilla.org/pub

When I remove the find-links option for proxxy1 it works just fine.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I wonder why we cannot reach this proxxy host:

$ ping pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com
PING proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46) 56(84) bytes of data.
64 bytes from proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46): icmp_seq=1 ttl=61 time=1.41 ms


The proxy config looks like and should cover everything inside of SCL3:

no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,.scl3.mozilla.com,.phx1.mozilla.com,.mozqa.com,mm-ci-production,mm-ci-staging,puppet,repos

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Is there a reason why we force setting the proxxy in install_module?

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/python.py#257

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Btw that has been introduced by bug 1017759:
https://hg.mozilla.org/build/mozharness/rev/89a6b34448ab

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So we have two options here:

1. Get our mozharness script working with proxxy and benefit from the caches
2. Don't use proxxy at the moment

I would suggest to use option 2 for now, so we can get tests running, and figuring out later why it doesn't work on the Linux hosts.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1 (disable proxxy)

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #6)
> 1. Get our mozharness script working with proxxy and benefit from the caches

Dustin, maybe you have an idea about the networking related stuff... Our Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub. But I can ping this host. The network ACLs in place are set for all nodes in qa.scl3.mozilla.com, so it should affect all platforms.

Comment 9

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/7883e81f3c30

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8692946 [details] [diff] [review]
Patch v1 (disable proxxy)

Landed on mozilla-central so it's available in mozharness via archiver. a=tomcat via IRC.

https://hg.mozilla.org/mozilla-central/rev/7883e81f3c30

Comment 11

[Jordan Lund (:jlund)]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> Is there a reason why we force setting the proxxy in install_module?
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/
> base/python.py#257

if I understand your question correctly, I think it's because we were focused on production. If you want, feel free to add a flag that skips using Proxxy for other use cases.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> Dustin, maybe you have an idea about the networking related stuff... Our
> Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from
> http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub.
> But I can ping this host. The network ACLs in place are set for all nodes in
> qa.scl3.mozilla.com, so it should affect all platforms.

Proxxy isn't intended for use outside of the releng network, and the firewall rules enforce that.

Keep in mind that "ping" is not a good network diagnostic tool -- Mozilla generally allows ping from any host to any host, regardless of firewall rules that may be in place for other TCP por UDP ports.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Dustin, so the usage of proxxy should really be optional in this case. 

Given that the workaround works perfectly for us right now I will file a new bug to get the underlying problem in mozharness fixed. I don't have time to do that myself.