740147 - VirtualenvMixin default python package server option

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Aki Sasaki (not active)]

If we were able to point at a server other than http://pypi.python.org/, we wouldn't have to explicitly create PACKAGENAME_url config items for every package.

We'd need to define the failure case when PACKAGENAME isn't on the python package server specified.

Comment 1

[Jeff Hammel]

with pip and easy_install this is 

 -i URL, --index-url=URL, --pypi-url=URL
                        Base URL of Python Package Index (default
                        http://pypi.python.org/simple/)

I believe this will fall back to pypi.python.org if a package is not found (though I should check this).

Comment 2

[Aki Sasaki (not active)]

(bb08)deathduck:~/src/jhammel/mozharness [14:23:13] (default)
589$ pip --version
pip 0.8.1 from /Users/asasaki/wrk/virtualenv/bb08/lib/python2.6/site-packages/pip-0.8.1-py2.6.egg (python 2.6)
(bb08)deathduck:~/src/jhammel/mozharness [14:23:15] (default)
590$ pip -i
Usage: pip COMMAND [OPTIONS]

pip: error: no such option: -i


pip --help doesn't show any -i.  However, I've got

  "pip_url": "http://build.mozilla.org/talos/zips/pip-1.0.2.tar.gz",

in some of the peptest config files, so maybe we're ok as long as we specify which version of pip needs to be installed.

Comment 3

[[e-mail] (:bhearsum)]

With pip you can use --find-links, which is what we do in our Puppet manifests that make use of virtualenv:
https://github.com/mozilla/puppet-manifests/blob/master/modules/python/manifests/virtualenv.pp#L260
http://guide.python-distribute.org/pip.html#add-urls-to-search-for-links

Comment 4

[Jeff Hammel]

I set up a pypi directory structure of talos and its dependencies for experimentation:

http://people.mozilla.com/~jhammel/pypi/

Comment 5

[Jeff Hammel]

(In reply to Aki Sasaki [:aki] from comment #2)
> (bb08)deathduck:~/src/jhammel/mozharness [14:23:13] (default)
> 589$ pip --version
> pip 0.8.1 from
> /Users/asasaki/wrk/virtualenv/bb08/lib/python2.6/site-packages/pip-0.8.1-py2.
> 6.egg (python 2.6)
> (bb08)deathduck:~/src/jhammel/mozharness [14:23:15] (default)
> 590$ pip -i
> Usage: pip COMMAND [OPTIONS]
> 
> pip: error: no such option: -i
> 
> 
> pip --help doesn't show any -i.  However, I've got
> 
>   "pip_url": "http://build.mozilla.org/talos/zips/pip-1.0.2.tar.gz",
> 
> in some of the peptest config files, so maybe we're ok as long as we specify
> which version of pip needs to be installed.

Newer versions of virtualenv come bundled with modern pip.

Comment 6

[Aki Sasaki (not active)]

(In reply to Jeff Hammel [:jhammel] from comment #5)
> Newer versions of virtualenv come bundled with modern pip.

I doubt we have newer versions on the production slaves, but we have workarounds.

Comment 7

[Jeff Hammel]

Attachment: allow specification of an index url to virtualenvmixin

Comment 8

[Jeff Hammel]

Oh, and it seems if you don't have the package available you get

"""
(tmp)│pip install --index-url http://people.mozilla.com/~jhammel/pypidfas/ http://hg.mozilla.org/build/talos/archive/tip.tar.gz -v
Downloading/unpacking http://hg.mozilla.org/build/talos/archive/tip.tar.gz
  Downloading tip.tar.gz (unknown size): 5.9Mb downloaded
  Running setup.py egg_info for package from http://hg.mozilla.org/build/talos/archive/tip.tar.gz
    running egg_info
    creating pip-egg-info/talos.egg-info
    writing requirements to pip-egg-info/talos.egg-info/requires.txt
    writing pip-egg-info/talos.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/talos.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/talos.egg-info/dependency_links.txt
    writing entry points to pip-egg-info/talos.egg-info/entry_points.txt
    writing manifest file 'pip-egg-info/talos.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found
    
    reading manifest file 'pip-egg-info/talos.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/talos.egg-info/SOURCES.txt'
Downloading/unpacking PyYAML (from talos==0.0)
  Could not fetch URL http://people.mozilla.com/~jhammel/pypidfas/PyYAML: HTTP Error 404: Not Found
  Will skip URL http://people.mozilla.com/~jhammel/pypidfas/PyYAML when looking for download links for PyYAML (from talos==0.0)
  Could not fetch URL http://people.mozilla.com/~jhammel/pypidfas/: HTTP Error 404: Not Found
  Will skip URL http://people.mozilla.com/~jhammel/pypidfas/ when looking for download links for PyYAML (from talos==0.0)
  Cannot fetch index base URL http://people.mozilla.com/~jhammel/pypidfas/
  Could not fetch URL http://people.mozilla.com/~jhammel/pypidfas/PyYAML/: HTTP Error 404: Not Found
  Will skip URL http://people.mozilla.com/~jhammel/pypidfas/PyYAML/ when looking for download links for PyYAML (from talos==0.0)
  Could not find any downloads that satisfy the requirement PyYAML (from talos==0.0)
No distributions at all found for PyYAML (from talos==0.0)
Exception information:
Traceback (most recent call last):
  File "/home/jhammel/tmp/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/home/jhammel/tmp/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg/pip/commands/install.py", line 223, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/jhammel/tmp/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg/pip/req.py", line 954, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/home/jhammel/tmp/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg/pip/index.py", line 152, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for PyYAML (from talos==0.0)

Storing complete log in /home/jhammel/.pip/pip.log
"""

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 611891 [details] [diff] [review]
allow specification of an index url to virtualenvmixin

First, --index-url doesn't necessarily scream "this is a virtualenv/pypi option" to me, and we don't know which scripts will be using these options.  Maybe --pypi-url? Maybe --python-index-url?

Other than that nit, I think the patch in general is good.
However, I:

virtualenv --no-site-packages ~/wrk/virtualenv/test
. ~/wrk/virtualenv/test/bin/activate
which pip
pip --version
pip -i
pip --help

pip 1.0.1 doesn't seem to have -i .
Unless we're going to very strictly control what version(s) of pip we're going to be compatible with, it seems fairly significant that pip 0.8.1 through 1.0.1 don't support this option.

Comment 10

[Aki Sasaki (not active)]

Also, I'd much rather we catch the exception and fatal() with an error message.
But I don't think we can take the patch as is unless we solve the missing -i option anyway.

Comment 11

[Jeff Hammel]

(In reply to Aki Sasaki [:aki] from comment #10)
> Also, I'd much rather we catch the exception and fatal() with an error
> message.
> But I don't think we can take the patch as is unless we solve the missing -i
> option anyway.

It won't be an exception since we're shelling out.

Comment 12

[Jeff Hammel]

(In reply to Aki Sasaki [:aki] from comment #9)
> Comment on attachment 611891 [details] [diff] [review]
> allow specification of an index url to virtualenvmixin
> 
> First, --index-url doesn't necessarily scream "this is a virtualenv/pypi
> option" to me, and we don't know which scripts will be using these options. 
> Maybe --pypi-url? Maybe --python-index-url?
> 
> Other than that nit, I think the patch in general is good.
> However, I:
> 
> virtualenv --no-site-packages ~/wrk/virtualenv/test
> . ~/wrk/virtualenv/test/bin/activate
> which pip
> pip --version
> pip -i
> pip --help
> 
> pip 1.0.1 doesn't seem to have -i .
> Unless we're going to very strictly control what version(s) of pip we're
> going to be compatible with, it seems fairly significant that pip 0.8.1
> through 1.0.1 don't support this option.

Do you have any idea what you want to do here?  easy_install has had -i since time immemorial (or at least the 2.4 days).  I'm actually surprised older versions of pip do not.

Comment 13

[Jeff Hammel]

Installing pip 0.8.2 locally seems to have both -i and the --index-url options:

(tmp)│pip --version
pip 0.8.2 from /home/jhammel/tmp/lib/python2.7/site-packages (python 2.7)
(tmp)│pip install --help
Usage: /home/jhammel/tmp/bin/pip install [OPTIONS] PACKAGE_NAMES...

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -E DIR, --environment=DIR
                        virtualenv environment to run pip in (either give the
                        interpreter or the environment base directory)
  -s, --enable-site-packages
                        Include site-packages in virtualenv if one is to be
                        created. Ignored if --environment is not used or the
                        virtualenv already exists.
  -v, --verbose         Give more output
  -q, --quiet           Give less output
  --log=FILENAME        Log file where a complete (maximum verbosity) record
                        will be kept
  --proxy=PROXY         Specify a proxy in the form
                        user:[e-mail]:port. Note that the
                        user:password@ is optional and required only if you
                        are behind an authenticated proxy.  If you provide
                        [e-mail]:port then you will be prompted for a
                        password.
  --timeout=SECONDS, --default-timeout=SECONDS
                        Set the socket timeout (default 15 seconds)
  -e VCS+REPOS_URL[@REV]#egg=PACKAGE, --editable=VCS+REPOS_URL[@REV]#egg=PACKAGE
                        Install a package directly from a checkout. Source
                        will be checked out into src/PACKAGE (lower-case) and
                        installed in-place (using setup.py develop). You can
                        run this on an existing directory/checkout (like pip
                        install -e src/mycheckout). This option may be
                        provided multiple times. Possible values for VCS are:
                        svn, git, hg and bzr.
  -r FILENAME, --requirement=FILENAME
                        Install all the packages listed in the given
                        requirements file.  This option can be used multiple
                        times.
  -f URL, --find-links=URL
                        URL to look for packages at
  -i URL, --index-url=URL, --pypi-url=URL
                        Base URL of Python Package Index (default
                        http://pypi.python.org/simple/)
  --extra-index-url=URL
                        Extra URLs of package indexes to use in addition to
                        --index-url
  --no-index            Ignore package index (only looking at --find-links
                        URLs instead)
  -M, --use-mirrors     Use the PyPI mirrors as a fallback in case the main
                        index is down.
  --mirrors=URL         Specific mirror URLs to query when --use-mirrors is
                        used
  -b DIR, --build=DIR, --build-dir=DIR, --build-directory=DIR
                        Unpack packages into DIR (default
                        /home/jhammel/tmp/build) and build from there
  -d DIR, --download=DIR, --download-dir=DIR, --download-directory=DIR
                        Download packages into DIR instead of installing them
  --download-cache=DIR  Cache downloaded packages in DIR
  --src=DIR, --source=DIR, --source-dir=DIR, --source-directory=DIR
                        Check out --editable packages into DIR (default
                        /home/jhammel/tmp/src)
  -U, --upgrade         Upgrade all packages to the newest available version
  -I, --ignore-installed
                        Ignore the installed packages (reinstalling instead)
  --no-deps, --no-dependencies
                        Ignore package dependencies
  --no-install          Download and unpack all packages, but don't actually
                        install them
  --no-download         Don't download any packages, just install the ones
                        already downloaded (completes an install run with
                        --no-install)
  --install-option=INSTALL_OPTIONS
                        Extra arguments to be supplied to the setup.py install
                        command (use like --install-option="--install-
                        scripts=/usr/local/bin").  Use multiple --install-
                        option options to pass multiple options to setup.py
                        install.  If you are using an option with a directory
                        path, be sure to use absolute path.
  --global-option=GLOBAL_OPTIONS
                        Extra global options to be supplied to the
                        setup.pycall before the install command
  --user                Install to user-site

Not sure why you don't see this.  Does --index-url work for you?

Comment 14

[Aki Sasaki (not active)]

I'm using py26.
We should block this on the upgrade-everything-to-py27 bugs, but not block talos work on this bug.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 611891 [details] [diff] [review]
allow specification of an index url to virtualenvmixin

Deferring review until we decide to drop pre-2.7 support, or otherwise solve this.

Comment 16

[Jeff Hammel]

I used python 2.4 on people.  Installing pip still gave me -i, --index-url

Comment 17

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #16)
> I used python 2.4 on people.  Installing pip still gave me -i, --index-url

with pip 0.8.2 btw

we should probably figure out what's actually going on here

Comment 18

[Jeff Hammel]

Ah, wait, I see the confusion here I hope.  -i, --install-url, --pypi-url is an option to the `pip install` command, not a global option, as is done in the patch  :aki, can you run `pip help install` and let me know if this is still an issue and if we can lose the blocking bug?

Comment 19

[Aki Sasaki (not active)]

That's precisely it.  Sorry about that.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 611891 [details] [diff] [review]
allow specification of an index url to virtualenvmixin

I think i'd prefer --pypi-url, which pip also groks

  -i URL, --index-url=URL, --pypi-url=URL

which will help people understand the purpose of the option and avoid conflicts with future options.  Otherwise I think this will work.

Comment 21

[Jeff Hammel]

Attachment: rename --index-url -> --pypi-url

carrying aki's r+ forward

Comment 22

[Jeff Hammel]

pushed: http://hg.mozilla.org/build/mozharness/rev/247a0442ac9e