756129 - add multiple no-index python package server support to mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect, P3)

Description

[Aki Sasaki (not active)]

Currently, if you specify a --pypi-url, VirtualenvMixin goes there and searches for its packages.

We should make --pypi-url an extendable list that VirtualenvMixin uses.  It should go through the list until it finds a working server, and fail out if none of them work.

This is essentially to create a client-side HA.

Also, we seem to be going away from the pypi index format to a directory of tarballs; the pip call should read

  pip install --no-index --find-links URL PACKAGE

Comment 1

[Aki Sasaki (not active)]

http://jacobian.org/writing/when-pypi-goes-down/ tells you how to use pypi mirrors; that might only be useful if you want to go to the official mirrors, though.

Comment 2

[Aki Sasaki (not active)]

find_links is now an available config list; we can avoid hitting pypi by setting pypi_url elsewhere, per https://bugzilla.mozilla.org/show_bug.cgi?id=766706#c4 .

Marking resolved.

Comment 3

[Jeff Hammel]

Its less of my concern as to how you want to run mozharness in production, but AIUI specifying a pypi_url of, say, build.m.o/blank_page will still hit the page and look for links ala http://pypi.python.org/simple/ .  We can leave this, as desired, but IMHO it is better to explicitly disable --pypi-url