Open Bug 1387039 Opened 8 years ago Updated 2 years ago

test installs from pypi.python.org, fails in mdc1

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: arich, Unassigned)

References

Details

We shouldn't be installing modules directly from github, we should have them as part of our internal pypi mirror or in tooltool, depending. The test machines in mdc1 do not allow external connections to install from github directly. See: https://tools.taskcluster.net/groups/M6YmCZUETJ2jman6S44TUg/tasks/GSC_kOB-RpePrx_OHtntXg/runs/0/logs/public%2Flogs%2Flog_raw.log Running setup.py (path:/Users/cltbld/tasks/task_1501761780/build/venv/build/twisted/setup.py) egg_info for package twisted Downloading/unpacking txws==0.9.1 (from -r /Users/cltbld/tasks/task_1501761780/build/tests/mochitest/websocketprocessbridge/websocketprocessbridge_requirements.txt (line 4))
05:16:45 ERROR - Traceback (most recent call last): 05:16:45 INFO - File "<string>", line 17, in <module> 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/build/txws/setup.py", line 15, in <module> 05:16:45 INFO - url="http://github.com/MostAwesomeDude/txWS", 05:16:45 INFO - File "/tools/python27/lib/python2.7/distutils/core.py", line 112, in setup 05:16:45 INFO - _setup_distribution = dist = klass(attrs) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/dist.py", line 221, in __init__ 05:16:45 INFO - self.fetch_build_eggs(attrs.pop('setup_requires')) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs 05:16:45 INFO - parse_requirements(requires), installer=self.fetch_build_egg 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 576, in resolve 05:16:45 INFO - dist = best[req.key] = env.best_match(req, self, installer) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 821, in best_match 05:16:45 INFO - return self.obtain(req, installer) # try and download/install 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 833, in obtain 05:16:45 INFO - return installer(requirement) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/dist.py", line 294, in fetch_build_egg 05:16:45 INFO - return cmd.easy_install(req) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/command/easy_install.py", line 576, in easy_install 05:16:45 INFO - self.local_index 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 491, in fetch_distribution 05:16:45 INFO - self.find_packages(requirement) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 325, in find_packages 05:16:45 INFO - self.scan_url(self.index_url + requirement.unsafe_name+'/') 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 668, in scan_url 05:16:45 INFO - self.process_url(url, True) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 201, in process_url 05:16:45 INFO - f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 610, in open_url 05:16:45 INFO - return open_with_auth(url) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 753, in _socket_timeout 05:16:45 INFO - return func(*args, **kwargs) 05:16:45 INFO - File "/Users/cltbld/tasks/task_1501761780/build/venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/setuptools/package_index.py", line 779, in open_with_auth 05:16:45 INFO - fp = urllib2.urlopen(request) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen 05:16:45 INFO - return _opener.open(url, data, timeout) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open 05:16:45 INFO - response = self._open(req, data) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open 05:16:45 INFO - '_open', req) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain 05:16:45 INFO - result = func(*args) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 1207, in http_open 05:16:45 INFO - return self.do_open(httplib.HTTPConnection, req) 05:16:45 INFO - File "/tools/python27/lib/python2.7/urllib2.py", line 1180, in do_open 05:16:45 INFO - r = h.getresponse(buffering=True) 05:16:45 INFO - File "/tools/python27/lib/python2.7/httplib.py", line 1030, in getresponse 05:16:45 INFO - response.begin() 05:16:45 INFO - File "/tools/python27/lib/python2.7/httplib.py", line 407, in begin 05:16:45 INFO - version, status, reason = self._read_status() 05:16:45 INFO - File "/tools/python27/lib/python2.7/httplib.py", line 365, in _read_status 05:16:45 INFO - line = self.fp.readline() 05:16:45 INFO - File "/tools/python27/lib/python2.7/socket.py", line 447, in readline 05:16:45 INFO - data = self._sock.recv(self._rbufsize) 05:16:45 INFO - socket.error: [Errno 54] Connection reset by peer the requirements file lists txWS=0.9.1, which is in the pypi mirror, and is being fetched successfully -- its setup.py is running when the failure occurs. Both of its requirements -- vcversioner and six -- are also in the pypi mirror. It looks like the problem is that txWS uses setup_requires, and that is parsed internally and fetched by distribute within the invocation of `setup.py egg_info`. Adding some debug prints to distribute and running this manually, I see that the open_with_auth call is trying to open http://pypi.python.org/simple/six/. I think that's because it's using the ancient "easy_install" process for setup_requires, so it doesn't know about the pip command-line arguments. I can fetch that URL though: >>> urllib2.urlopen('http://pypi.python.org/simple/six/').read() '<!DOCTYPE html><html>.....</html>' Note that http redirects to https. This is looking more complicated than I thought: * the requirement itself and its deps are in the pypi mirror, but it's not using them; and * it's not clear why the HTTP request to pypi.python.org is failing
Traffic to pypi.python.org during the pip invocation: 09:17:35.381579 IP (tos 0x0, ttl 64, id 45677, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->8575)!) t-yosemite-r7-472.test.releng.mdc1.mozilla.com.55641 > 151.101.40.223.http: Flags [S], cksum 0x0308 (incorrect -> 0xfdfe), seq 2828324974, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 684112565 ecr 0,sackOK,eol], length 0 0x0000: b40c 25e0 4010 a860 b624 7eda 0800 4500 ..%.@..`.$~...E. 0x0010: 0040 b26d 4000 4006 0000 0a31 3860 9765 .@.m@.@....18`.e 0x0020: 28df d959 0050 a894 d06e 0000 0000 b002 (..Y.P...n...... 0x0030: ffff 0308 0000 0204 05b4 0103 0305 0101 ................ 0x0040: 080a 28c6 bab5 0000 0000 0402 0000 ..(........... 09:17:35.382649 IP (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 151.101.40.223.http > t-yosemite-r7-472.test.releng.mdc1.mozilla.com.55641: Flags [S.], cksum 0xeaed (correct), seq 381086945, ack 2828324975, win 28960, options [mss 1460,sackOK,TS val 2145464167 ecr 684112565,nop,wscale 9], length 0 0x0000: a860 b624 7eda b40c 25e0 4010 0800 4500 .`.$~...%.@...E. 0x0010: 003c 0000 4000 3a06 3de7 9765 28df 0a31 .<..@.:.=..e(..1 0x0020: 3860 0050 d959 16b6 ece1 a894 d06f a012 8`.P.Y.......o.. 0x0030: 7120 eaed 0000 0204 05b4 0402 080a 7fe1 q............... 0x0040: 2f67 28c6 bab5 0103 0309 /g(....... 09:17:35.382666 IP (tos 0x0, ttl 64, id 18625, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->ef2d)!) t-yosemite-r7-472.test.releng.mdc1.mozilla.com.55641 > 151.101.40.223.http: Flags [.], cksum 0x02fc (incorrect -> 0x7ac6), seq 1, ack 1, win 4117, options [nop,nop,TS val 684112566 ecr 2145464167], length 0 0x0000: b40c 25e0 4010 a860 b624 7eda 0800 4500 ..%.@..`.$~...E. 0x0010: 0034 48c1 4000 4006 0000 0a31 3860 9765 .4H.@.@....18`.e 0x0020: 28df d959 0050 a894 d06f 16b6 ece2 8010 (..Y.P...o...... 0x0030: 1015 02fc 0000 0101 080a 28c6 bab6 7fe1 ..........(..... 0x0040: 2f67 /g 09:17:35.382707 IP (tos 0x0, ttl 64, id 11166, offset 0, flags [DF], proto TCP (6), length 199, bad cksum 0 (->bbe)!) t-yosemite-r7-472.test.releng.mdc1.mozilla.com.55641 > 151.101.40.223.http: Flags [P.], cksum 0x038f (incorrect -> 0x015a), seq 1:148, ack 1, win 4117, options [nop,nop,TS val 684112566 ecr 2145464167], length 147 0x0000: b40c 25e0 4010 a860 b624 7eda 0800 4500 ..%.@..`.$~...E. 0x0010: 00c7 2b9e 4000 4006 0000 0a31 3860 9765 ..+.@.@....18`.e 0x0020: 28df d959 0050 a894 d06f 16b6 ece2 8018 (..Y.P...o...... 0x0030: 1015 038f 0000 0101 080a 28c6 bab6 7fe1 ..........(..... 0x0040: 2f67 4745 5420 2f73 696d 706c 652f 7369 /gGET./simple/si 0x0050: 782f 2048 5454 502f 312e 310d 0a41 6363 x/.HTTP/1.1..Acc 0x0060: 6570 742d 456e 636f 6469 6e67 3a20 6964 ept-Encoding:.id 0x0070: 656e 7469 7479 0d0a 486f 7374 3a20 7079 entity..Host:.py 0x0080: 7069 2e70 7974 686f 6e2e 6f72 670d 0a43 pi.python.org..C 0x0090: 6f6e 6e65 6374 696f 6e3a 2063 6c6f 7365 onnection:.close 0x00a0: 0d0a 5573 6572 2d41 6765 6e74 3a20 5079 ..User-Agent:.Py 0x00b0: 7468 6f6e 2d75 726c 6c69 622f 322e 3720 thon-urllib/2.7. 0x00c0: 7365 7475 7074 6f6f 6c73 2f30 2e36 6331 setuptools/0.6c1 0x00d0: 310d 0a0d 0a 1.... 09:17:35.383509 IP (tos 0x0, ttl 63, id 6834, offset 0, flags [none], proto TCP (6), length 40) 151.101.40.223.http > t-yosemite-r7-472.test.releng.mdc1.mozilla.com.55641: Flags [R.], cksum 0x460c (correct), seq 1, ack 148, win 4117, length 0 0x0000: a860 b624 7eda b40c 25e0 4010 0800 4500 .`.$~...%.@...E. 0x0010: 0028 1ab2 0000 3f06 5e49 9765 28df 0a31 .(....?.^I.e(..1 0x0020: 3860 0050 d959 16b6 ece2 a894 d102 5014 8`.P.Y........P. 0x0030: 1015 460c 0000 0000 db32 9801 ..F......2..
Summary: test tries to install http://github.com/MostAwesomeDude/txWS → test installs from pypi.python.org, fails in mdc1
Ah, something is filtering on user-agent: [root@t-yosemite-r7-472.test.releng.mdc1.mozilla.com venv]# curl -v --user-agent 'Python-urllib/2.7 setuptools/0.6c11' http://pypi.python.org/simple/six/ * Trying 151.101.40.223... * Connected to pypi.python.org (151.101.40.223) port 80 (#0) > GET /simple/six/ HTTP/1.1 > Host: pypi.python.org > User-Agent: Python-urllib/2.7 setuptools/0.6c11 > Accept: */* > * Recv failure: Connection reset by peer * Closing connection 0 curl: (56) Recv failure: Connection reset by peer [root@t-yosemite-r7-472.test.releng.mdc1.mozilla.com venv]# curl -v http://pypi.python.org/simple/six/ * Trying 151.101.40.223... * Connected to pypi.python.org (151.101.40.223) port 80 (#0) > GET /simple/six/ HTTP/1.1 > Host: pypi.python.org > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Server: Varnish < Retry-After: 0 < Location: https://pypi.python.org/simple/six/ < Content-Length: 0 < Accept-Ranges: bytes < Date: Thu, 03 Aug 2017 16:22:20 GMT < Via: 1.1 varnish < Connection: close < X-Served-By: cache-sjc3628-SJC < X-Cache: HIT < X-Cache-Hits: 0 < X-Frame-Options: deny < X-XSS-Protection: 1; mode=block < X-Content-Type-Options: nosniff < X-Permitted-Cross-Domain-Policies: none < * Closing connection 0
I can't reproduce this outside of mdc1, not even in scl3. Amy, is there a hidden proxy in this DC or something that would otherwise be limiting such requests? Alternately, it's possible that pypi has blacklisted some of our IPs or we've gotten tripped up by some DOS protection. Maybe they whitelisted the scl3 NAT IPs and need to do the same for the new mdc1 IPs? Anyway, repro recipe is above, and it looks like netops is hte next step to try to solve this.
Flags: needinfo?(arich)
james, would the mdc1 firewall be blocking these connections? We need to make sure that all 80/443 are allowed (there was a resolved bug for that, but I know that the firewall was looking at things other than port number).
Flags: needinfo?(arich) → needinfo?(jbarnell)
added python-easy-install, subversion-base, and ftp to the rule set. please test again.
Flags: needinfo?(jbarnell) → needinfo?(arich)
I don't see a need for subversion or ftp. If python-easy-install is necessary, then ports 80 and 443 are not really open -- and tests will expect those ports to be completely open. I don't think we can play whack-a-mole with every HTTP user-agent we want to allow outbound access for. Please change the rule to the equivalent of "if destination not in 01.0.0.0/8 and port == 80 or port == 443: allow"
I concur, we want these to be open based on port not type of traffic. The tests will request a variety of different things that we aren't going to be able to know or anticipate. Please just allow everything outbound on 80 and 443.
Flags: needinfo?(arich) → needinfo?(jbarnell)
That's not how it works in PAN. Please go ahead and test.
Flags: needinfo?(jbarnell)
Even if it works for one test now, there's no way we can anticipate what tests will be added by developers later. We do not want to falsely break firefox tests because of a filtering rule on the firewall. How can we open this up for all types of traffic on 80 and 443?
Flags: needinfo?(jbarnell)
We should be grabbing Python packages from vendored source or hitting our internal PyPI mirror for all critical automation. We typically fix issues like this by adding a missing package to our internal PyPI mirror. We don't want our CI hammering PyPI and we don't want to have to close the trees when PyPI isn't available. I'd prefer we toe the line and keep the firewall closed. Not sure if that is reasonable.
Yeah, that's why I originally filed the bug here, but in this case the files are already vendored, but easy_install isn't looking for them in the right place. I doubt there's any non-fragile way to convince setup.py's setup_requires to use the URL options given to pip. In previous seasons of "let's shut off external access from testers", we've uncovered layer upon layer of interesting behavior, not all of it obviously wrong, and much of it difficult to fix or replace. With everything else going on right now, I don't think this is the time to premiere a new season. All of these tasks run fine in scl3, so mdc1 should be configured such that they run fine there, too.
So setup_requires in txWS's setup.py is triggering a URL search for a package at setup.py time. Fantastic. I noticed that testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt doesn't have an entry for vcversioner. It would be a good practice to add one. If we move vcversioner and six in testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt to before txWS, pip will install those first and *might* avoid a URL search during txWS's setup.py or it may treat it as non-fatal since the package is already installed. But I'm not 100% sure on this because it still appears to do a PyPI lookup: Collecting vcversioner==2.16.0.0 (from -r testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt (line 5)) 1 location(s) to search for versions of vcversioner: * https://pypi.python.org/simple/vcversioner/ Getting page https://pypi.python.org/simple/vcversioner/ Looking up "https://pypi.python.org/simple/vcversioner/" in the cache No cache entry available "GET /simple/vcversioner/ HTTP/1.1" 200 965 Updating cache with response from "https://pypi.python.org/simple/vcversioner/" Caching b/c date exists and max-age > 0 Analyzing links from page https://pypi.python.org/simple/vcversioner/ Found link https://pypi.python.org/packages/19/df/0886516b7fccac96f5e9733cd7a20af6bf10874d698c4953bbe0c20ac677/vcversioner-1.13.0.0.tar.gz#md5=07ad32ac279161b4a036324834096079 (from https://pypi.python.org/simple/vcversioner/), version: 1.13.0.0 Found link https://pypi.python.org/packages/26/8d/cb1bd2f9c4c44ddd5824aa0b536d054e7a8898db44188002a43f375bf3a7/vcversioner-0.13.0.tar.gz#md5=3a3eb170400fc7bc29231ec811a87bda (from https://pypi.python.org/simple/vcversioner/), version: 0.13.0 Found link https://pypi.python.org/packages/51/ab/26ccd65f36cc84835761872f122f40a80b149f872c4d2b83bb1a3aba7756/vcversioner-1.14.0.0.tar.gz#md5=91d20625f9823f6b565b41e77d2de968 (from https://pypi.python.org/simple/vcversioner/), version: 1.14.0.0 Found link https://pypi.python.org/packages/5a/6b/6f5da157648cadbaf83f625c395cd23ff6be3421268b7bf54523b8d9aaab/vcversioner-2.16.0.0-py2-none-any.whl#md5=afbe817dbd60d2f70724ce920e492590 (from https://pypi.python.org/simple/vcversioner/), version: 2.16.0.0 Found link https://pypi.python.org/packages/77/3a/4f93edf09f2cc8925eedfd9ad286b50909cf71f132f56a32e10ece5a283c/vcversioner-0.13.0.2.tar.gz#md5=fedfad15168149a19da6bea9df568fd3 (from https://pypi.python.org/simple/vcversioner/), version: 0.13.0.2 Found link https://pypi.python.org/packages/88/2d/2846726385dacc0b227f8f5521b7dec673977946f0abba0f4474fe095ab2/vcversioner-1.14.1.1.tar.gz#md5=3680f1df3982edebf0e2417439cdacf4 (from https://pypi.python.org/simple/vcversioner/), version: 1.14.1.1 Found link https://pypi.python.org/packages/a6/e4/e1b993b191e3022cd31b81953e86e24673ddf060e4309294c1f991f78530/vcversioner-0.13.1.0.tar.gz#md5=866fa9840a9034d6530c98923d7356d0 (from https://pypi.python.org/simple/vcversioner/), version: 0.13.1.0 Found link https://pypi.python.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz#md5=aab6ef5e0cf8614a1b1140ed5b7f107d (from https://pypi.python.org/simple/vcversioner/), version: 2.16.0.0 Found link https://pypi.python.org/packages/c8/bf/a850e9924d42d52e8677703338c56559da2c54166fa6a813f517a6b80161/vcversioner-2.14.0.0.tar.gz#md5=7848a365ced9941053bc25d9a9f8f4b4 (from https://pypi.python.org/simple/vcversioner/), version: 2.14.0.0 Found link https://pypi.python.org/packages/d5/68/c8b167951ffd6665e8bf7bb3d6b690b3b802bb1dce3dc361e6c118f2e283/vcversioner-0.13.0.1.tar.gz#md5=9007e3a05fe02916fb5e6500f56706f7 (from https://pypi.python.org/simple/vcversioner/), version: 0.13.0.1 Found link https://pypi.python.org/packages/f9/6a/76da913e539de4e7adbf57cde814dc2e749244000f250ea151274bd6fdb5/vcversioner-1.14.1.0.tar.gz#md5=d229c0f4d0e61bdd10cca868b9649f97 (from https://pypi.python.org/simple/vcversioner/), version: 1.14.1.0 Using version 2.16.0.0 (newest of versions: 2.16.0.0) Looking up "https://pypi.python.org/packages/5a/6b/6f5da157648cadbaf83f625c395cd23ff6be3421268b7bf54523b8d9aaab/vcversioner-2.16.0.0-py2-none-any.whl" in the cache No cache entry available "GET /packages/5a/6b/6f5da157648cadbaf83f625c395cd23ff6be3421268b7bf54523b8d9aaab/vcversioner-2.16.0.0-py2-none-any.whl HTTP/1.1" 200 13934 Downloading vcversioner-2.16.0.0-py2-none-any.whl Downloading from URL https://pypi.python.org/packages/5a/6b/6f5da157648cadbaf83f625c395cd23ff6be3421268b7bf54523b8d9aaab/vcversioner-2.16.0.0-py2-none-any.whl#md5=afbe817dbd60d2f70724ce920e492590 (from https://pypi.python.org/simple/vcversioner/) Updating cache with response from "https://pypi.python.org/packages/5a/6b/6f5da157648cadbaf83f625c395cd23ff6be3421268b7bf54523b8d9aaab/vcversioner-2.16.0.0-py2-none-any.whl" Caching due to etag Collecting txws==0.9.1 (from -r testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt (line 8)) 1 location(s) to search for versions of txws: * https://pypi.python.org/simple/txws/ Getting page https://pypi.python.org/simple/txws/ Looking up "https://pypi.python.org/simple/txws/" in the cache No cache entry available "GET /simple/txws/ HTTP/1.1" 200 850 Updating cache with response from "https://pypi.python.org/simple/txws/" Caching b/c date exists and max-age > 0 Analyzing links from page https://pypi.python.org/simple/txws/ Found link https://pypi.python.org/packages/40/d1/6b73300f089da0555c73dfceed06b601aa9d5a88c6222bacc68c40ccbba8/txWS-0.9.tar.gz#md5=b2c023774e89ed758225e2959e2ffa23 (from https://pypi.python.org/simple/txws/), version: 0.9 Found link https://pypi.python.org/packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz#md5=d113910af0521ea62db8a0f3d7c63abb (from https://pypi.python.org/simple/txws/), version: 0.9.1 Found link https://pypi.python.org/packages/57/24/997db7ce7388f58fd4d2fa3c75f6382921c1c174eb2f7caeea5289dd0e6f/txWS-0.6.1.tar.gz#md5=ec0635ead35323a58d2a17154b4c86a4 (from https://pypi.python.org/simple/txws/), version: 0.6.1 Found link https://pypi.python.org/packages/58/f6/d4a5ca71b8394317677fde2e85ece3a73113bb61a9b74e02a94b45cdce67/txWS-0.7.tar.gz#md5=e8f5fb03c189d83b47b21176c7574126 (from https://pypi.python.org/simple/txws/), version: 0.7 Found link https://pypi.python.org/packages/70/d6/004a201735e21e5cd8331e4bba57e83274decf0f3c738a4bb58640542207/txWS-0.8.tar.gz#md5=c5584da7c653a74f442c1b1e8bc636d9 (from https://pypi.python.org/simple/txws/), version: 0.8 Found link https://pypi.python.org/packages/71/89/9a058a193558fdce92917f5f1c3e813733df560c7a62664cba7598bf3daa/txWS-0.7.1.1.tar.gz#md5=f88b215c569d98d3666bbfcc306c4171 (from https://pypi.python.org/simple/txws/), version: 0.7.1.1 Found link https://pypi.python.org/packages/9b/b2/e48413260f7c9f2d750b7edae636bed429527d2bb7e494f6ac80d8a44706/txWS-0.5.tar.gz#md5=7b6f7a595a2af30463801f9a82a0c722 (from https://pypi.python.org/simple/txws/), version: 0.5 Found link https://pypi.python.org/packages/a4/e5/6645105dc57b7146b4be6d62c2a7ec7b10228842d4bf98e17f830d3256db/txWS-0.6.2.tar.gz#md5=a653b00eb1cdaca0fb3209dae31e966d (from https://pypi.python.org/simple/txws/), version: 0.6.2 Found link https://pypi.python.org/packages/c6/1c/9cbada53da231f2355e0ce3a47cb2945fc8141a291f94c3504d6e295cbfe/txWS-0.6.tar.gz#md5=b8d48035cef6d48d16eaa4d7a90e2d8c (from https://pypi.python.org/simple/txws/), version: 0.6 Found link https://pypi.python.org/packages/d2/cf/33c703663e65eee7dbfd8724c1a34ae4692511767670876e73eda6519689/txWS-0.7.1.tar.gz#md5=0663015b62528a94107b9b434755f887 (from https://pypi.python.org/simple/txws/), version: 0.7.1 Using version 0.9.1 (newest of versions: 0.9.1) Looking up "https://pypi.python.org/packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz" in the cache No cache entry available "GET /packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz HTTP/1.1" 200 9899 Downloading txWS-0.9.1.tar.gz Downloading from URL https://pypi.python.org/packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz#md5=d113910af0521ea62db8a0f3d7c63abb (from https://pypi.python.org/simple/txws/) Updating cache with response from "https://pypi.python.org/packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz" Caching due to etag Running setup.py (path:/tmp/pip-build-BHDfuN/txws/setup.py) egg_info for package txws Running command python setup.py egg_info no previously-included directories found matching 'documentation/_build' zip_safe flag not set; analyzing archive contents... six: module references __path__ Installed /tmp/pip-build-BHDfuN/txws/.eggs/six-1.10.0-py2.7.egg Searching for vcversioner Reading https://pypi.python.org/simple/vcversioner/ Downloading https://pypi.python.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz#md5=aab6ef5e0cf8614a1b1140ed5b7f107d Best match: vcversioner 2.16.0.0 Processing vcversioner-2.16.0.0.tar.gz Writing /tmp/easy_install-dpFcdV/vcversioner-2.16.0.0/setup.cfg Running vcversioner-2.16.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-dpFcdV/vcversioner-2.16.0.0/egg-dist-tmp-oxzAXz zip_safe flag not set; analyzing archive contents... Moving vcversioner-2.16.0.0-py2.7.egg to /tmp/pip-build-BHDfuN/txws/.eggs Installed /tmp/pip-build-BHDfuN/txws/.eggs/vcversioner-2.16.0.0-py2.7.egg running egg_info creating pip-egg-info/txWS.egg-info writing pip-egg-info/txWS.egg-info/PKG-INFO writing top-level names to pip-egg-info/txWS.egg-info/top_level.txt writing dependency_links to pip-egg-info/txWS.egg-info/dependency_links.txt writing manifest file 'pip-egg-info/txWS.egg-info/SOURCES.txt' reading manifest file 'pip-egg-info/txWS.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pip-egg-info/txWS.egg-info/SOURCES.txt' Source in /tmp/pip-build-BHDfuN/txws has version 0.9.1, which satisfies requirement txws==0.9.1 from https://pypi.python.org/packages/4f/ee/72101b853a7dc16e5bc8b784e253cdd016ba00ab9df31a4a1dec44cd0e1c/txWS-0.9.1.tar.gz#md5=d113910af0521ea62db8a0f3d7c63abb (from -r testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt (line 8)) Building wheels for collected packages: txws I dunno. tarek: can you recommend a way to stop this URL lookup during setup_requires?
Flags: needinfo?(tarek)
James, we're still looking for an answer about allowing all HTTP/HTTPS traffic out of this network -- the subsequent discussion of pip and easy_install does not remove that requirement.
that's already in place as discussed on Friday. If something is broken its not the policy.
Flags: needinfo?(jbarnell) → needinfo?(dustin)
OK, I wasn't part of that conversation. Thanks!
not sure why I was needinfoed -- if there's info needed from me somewhere here, please ask again.
Flags: needinfo?(dustin)
It's a hack I have not tried but you can block any call to PyPI made by easy_install (which do those calls) during the installation of the package by routing easy_install index to a local dir https://pip.pypa.io/en/latest/reference/pip_install/#controlling-setup-requires If that does not work we can do something at the pip level to block this behavior (and ask pypa to add it as a new option, I don't see that) email me or ping me on irc if you need more help on that
Flags: needinfo?(tarek)
See Also: → 1415655
Product: Core → Firefox Build System
I'm the project manager for Warehouse https://github.com/pypa/warehouse/ (serving pypi.org, the new PyPI https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html that's replacing pypi.python.org on Monday April 16th). Your CI running a lot of "pip install" commands is, I presume, mostly hitting our CDN (Fastly) so that's not too bad on our side. What code is your internal PyPI mirror running? Bandersnatch or something else?
The internal mirror is just an Apache directory listing. That has the advantage of being simple and thus not failure-prone. It works well enough, even if it is a bit annoying to have to scp packages there to add them. We do want the control of only allowing a select few people to put packages on the mirror. So it's not really a mirror :) I think the concern here is what Tarek is referring to - ensuring that installs do not ever talk to pypi.python.org. Not because of a concern with load on that site, but because it might have packages on it that we do not want to install (e.g., a newer version of some library). Maybe there's something we can discuss here, though? I'm probably not the right person to be involved, at least alone..
We should be using hash pinning in pip requirements files for all index installs. Full stop. It's a security vulnerability and non-determinism issue if we don't. In addition to that, CI should be using a Mozilla-hosted index so we don't have a dependency on 3rd party services. That's a CI platform reliability issue. These are the ideals, not what we're currently practicing universally.
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.