[firefox-ui-tests] Very slow fetches of virtualenv packages from releng PyPI mirror in SCL3 with proxxy config

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
2 years ago
3 months ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
Today I have turned on mozharness support for our Firefox UI tests in our CI staging instance in SCL3. Builds on Windows and OS X seem to work pretty fine. But on Ubuntu I see a huge slowdown while installing required python packages from the RelEng PyPI mirror:

01:28:06     INFO - ##### Running create-virtualenv step.
[..]
01:53:21     INFO - ##### Running install step.

All in all the create-virtualenv step takes 25 minutes! Here an example of the installation for one of the needed packages:

01:41:12     INFO -  Ignoring indexes: https://pypi.python.org/simple/
01:41:12     INFO -  Downloading/unpacking browsermob-proxy==0.6.0 (from -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt (line 2))
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com has it available
01:41:12     INFO -    http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
01:43:12     INFO -    Using download cache from /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache/http%3A%2F%2Fpypi.pub.build.mozilla.org%2Fpub%2Fbrowsermob-proxy-0.6.0.tar.gz
01:43:12     INFO -    Running setup.py (path:/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/build/browsermob-proxy/setup.py) egg_info for package browsermob-proxy

We wait 2 minutes before using a download cache?
(Assignee)

Updated

2 years ago
Blocks: 1221044
(Assignee)

Comment 1

2 years ago
I assume that this is happing because pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com is used here. Creating another virtualenv outside of the mozharness script works fine as expected.

I assume that we can easily turn off the proxxy config for our tests via Jenkins?
Flags: needinfo?(armenzg)
(Assignee)

Comment 2

2 years ago
The command as executed here looks like:

/home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/bin/pip -v install --download-cache /home/mozauto/jenkins/workspace/mozilla-central_functional/build/venv/cache --timeout 120 -r /home/mozauto/jenkins/workspace/mozilla-central_functional/build/firefox_ui_tests/requirements.txt --no-index --find-links http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub --find-links http://pypi.pub.build.mozilla.org/pub

When I remove the find-links option for proxxy1 it works just fine.
(Assignee)

Comment 3

2 years ago
I wonder why we cannot reach this proxxy host:

$ ping pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com
PING proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46) 56(84) bytes of data.
64 bytes from proxxy1.srv.releng.scl3.mozilla.com (10.26.48.46): icmp_seq=1 ttl=61 time=1.41 ms


The proxy config looks like and should cover everything inside of SCL3:

no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,.scl3.mozilla.com,.phx1.mozilla.com,.mozqa.com,mm-ci-production,mm-ci-staging,puppet,repos
(Assignee)

Comment 4

2 years ago
Is there a reason why we force setting the proxxy in install_module?

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/python.py#257
Flags: needinfo?(jlund)
(Assignee)

Comment 6

2 years ago
So we have two options here:

1. Get our mozharness script working with proxxy and benefit from the caches
2. Don't use proxxy at the moment

I would suggest to use option 2 for now, so we can get tests running, and figuring out later why it doesn't work on the Linux hosts.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
(Assignee)

Comment 7

2 years ago
Created attachment 8692946 [details] [diff] [review]
Patch v1 (disable proxxy)
Attachment #8692946 - Flags: review?(armenzg)
(Assignee)

Comment 8

2 years ago
(In reply to Henrik Skupin (:whimboo) from comment #6)
> 1. Get our mozharness script working with proxxy and benefit from the caches

Dustin, maybe you have an idea about the networking related stuff... Our Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub. But I can ping this host. The network ACLs in place are set for all nodes in qa.scl3.mozilla.com, so it should affect all platforms.
Flags: needinfo?(dustin)
Attachment #8692946 - Flags: review?(armenzg) → review+
Flags: needinfo?(armenzg)
(Assignee)

Comment 10

2 years ago
Comment on attachment 8692946 [details] [diff] [review]
Patch v1 (disable proxxy)

Landed on mozilla-central so it's available in mozharness via archiver. a=tomcat via IRC.

https://hg.mozilla.org/mozilla-central/rev/7883e81f3c30
Attachment #8692946 - Flags: checked-in+
(In reply to Henrik Skupin (:whimboo) from comment #4)
> Is there a reason why we force setting the proxxy in install_module?
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/
> base/python.py#257

if I understand your question correctly, I think it's because we were focused on production. If you want, feel free to add a flag that skips using Proxxy for other use cases.
Flags: needinfo?(jlund)

Updated

2 years ago
See Also: → bug 1228922

Updated

2 years ago
See Also: → bug 1228928

Updated

2 years ago
See Also: bug 1228922
(Assignee)

Updated

2 years ago
See Also: bug 1228928
(In reply to Henrik Skupin (:whimboo) from comment #8)
> Dustin, maybe you have an idea about the networking related stuff... Our
> Ubuntu nodes in qa.scl3.mozilla.com cannot download python packages from
> http://pypi.pub.build.mozilla.org.proxxy1.srv.releng.scl3.mozilla.com/pub.
> But I can ping this host. The network ACLs in place are set for all nodes in
> qa.scl3.mozilla.com, so it should affect all platforms.

Proxxy isn't intended for use outside of the releng network, and the firewall rules enforce that.

Keep in mind that "ping" is not a good network diagnostic tool -- Mozilla generally allows ping from any host to any host, regardless of firewall rules that may be in place for other TCP por UDP ports.
Flags: needinfo?(dustin)
(Assignee)

Comment 13

2 years ago
Thanks Dustin, so the usage of proxxy should really be optional in this case. 

Given that the workaround works perfectly for us right now I will file a new bug to get the underlying problem in mozharness fixed. I don't have time to do that myself.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Depends on: 1229008
You need to log in before you can comment on or make changes to this bug.