Closed Bug 1228544 Opened 4 years ago Closed 4 years ago

[firefox-ui-tests] Very slow fetches of virtualenv packages from releng PyPI mirror in SCL3 with proxxy config

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

Unspecified
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: whimboo, Assigned: whimboo)

References

()

Details

Attachments

(1 file)

Today I have turned on mozharness support for our Firefox UI tests in our CI staging instance in SCL3. Builds on Windows and OS X seem to work pretty fine. But on Ubuntu I see a huge slowdown while installing required python packages from the RelEng PyPI mirror:

01:28:06     INFO - ##### Running create-virtualenv step.
[..]
01:53:21     INFO - ##### Running install step.

All in all the create-virtualenv step takes 25 minutes! Here an example of the installation for one of the needed packages:

01:41:12     INFO -  Ignoring indexes: https://pypi.python.org/simple/
01:41:12     INFO -  Downloading/unpacking browsermob-proxy==0.6.0 (from -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt (line 2))
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:43:12     INFO -    Using download cache from /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache/http%3A%2F%2Fpypi.pub.build.mozilla.org%2Fpub%2Fbrowsermob-proxy-0.6.0.tar.gz
01:43:12     INFO -    Running setup.py (path:/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/build/browsermob-proxy/setup.py) egg_info for package browsermob-proxy

We wait 2 minutes before using a download cache?
I assume that this is happing because pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com is used here. Creating another virtualenv outside of the mozharness script works fine as expected.

I assume that we can easily turn off the proxxy config for our tests via Jenkins?
Flags: needinfo?(armenzg)
The command as executed here looks like:

/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/bin/pip -v install --download-cache /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache --timeout 120 -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt --no-index --find-links http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub --find-links http://pypi.pub.build.mozilla.org/pub

When I remove the find-links option for proxxy1 it works just fine.
I wonder why we cannot reach this proxxy host:

$ ping pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com
PING proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46) 56(84) bytes of data.
64 bytes from proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46): icmp_seq=1 ttl=61 time=1.41 ms


The proxy config looks like and should cover everything inside of SCL3:

no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,.scl3.mozilla.com,.phx1.mozilla.com,.mozqa.com,mm-ci-production,mm-ci-staging,puppet,repos
Is there a reason why we force setting the proxxy in install_module?

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/python.py#257
Flags: needinfo?(jlund)
So we have two options here:

1. Get our mozharness script working with proxxy and benefit from the caches
2. Don't use proxxy at the moment

I would suggest to use option 2 for now, so we can get tests running, and figuring out later why it doesn't work on the Linux hosts.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Attachment #8692946 - Flags: review?(armenzg)
(In reply to Henrik Skupin (:whimboo) from comment #6)
> 1. Get our mozharness script working with proxxy and benefit from the caches

Dustin, maybe you have an idea about the networking related stuff... Our Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub. But I can ping this host. The network ACLs in place are set for all nodes in qa.scl3.mozilla.com, so it should affect all platforms.
Flags: needinfo?(dustin)
Attachment #8692946 - Flags: review?(armenzg) → review+
Flags: needinfo?(armenzg)
Comment on attachment 8692946 [details] [diff] [review]
Patch v1 (disable proxxy)

Landed on mozilla-central so it's available in mozharness via archiver. a=tomcat via IRC.

https://hg.mozilla.org/mozilla-central/rev/7883e81f3c30
Attachment #8692946 - Flags: checked-in+
(In reply to Henrik Skupin (:whimboo) from comment #4)
> Is there a reason why we force setting the proxxy in install_module?
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/
> base/python.py#257

if I understand your question correctly, I think it's because we were focused on production. If you want, feel free to add a flag that skips using Proxxy for other use cases.
Flags: needinfo?(jlund)
See Also: → 1228922
See Also: → 1228928
See Also: 1228922
See Also: 1228928
(In reply to Henrik Skupin (:whimboo) from comment #8)
> Dustin, maybe you have an idea about the networking related stuff... Our
> Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from
> http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub.
> But I can ping this host. The network ACLs in place are set for all nodes in
> qa.scl3.mozilla.com, so it should affect all platforms.

Proxxy isn't intended for use outside of the releng network, and the firewall rules enforce that.

Keep in mind that "ping" is not a good network diagnostic tool -- Mozilla generally allows ping from any host to any host, regardless of firewall rules that may be in place for other TCP por UDP ports.
Flags: needinfo?(dustin)
Thanks Dustin, so the usage of proxxy should really be optional in this case. 

Given that the workaround works perfectly for us right now I will file a new bug to get the underlying problem in mozharness fixed. I don't have time to do that myself.
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Depends on: 1229008
You need to log in before you can comment on or make changes to this bug.