885780 - mozharness should use pip.conf

Status: RESOLVED INCOMPLETE

(Release Engineering :: Applications: MozharnessCore, defect, P3)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

https://mxr.mozilla.org/build/search?string=http%3A%2F%2Frepos%2F&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=build indicates that lots of mozharness configs set up pypi to point to http://repos/.  That's fragile, since that CNAME points to only one host, which might be down at the time.

Instead, mozharness should use the existing pip config for cltbld.  I'm not sure if that config carries over into mock environments.  Puppet configures that pip config to point to all puppet masters, so that if one is down, pip will try another.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This just caused a bunch of builds to burn.

Comment 2

[[e-mail] (:bhearsum)]

Aki, any idea how to do this? Maybe we can have one of the interns look at it if it's clear what the implementation looks like.

Comment 3

[Aki Sasaki (not active)]

In a perfect world, it's:

* remove all instances of pypi_url and find_links from mozharness configs
* find a way to deal with distribute_url (still needed once python is a recent version?) and simplejson_url (probably create a virtualenv with simplejson as a virtualenv_modules dependency).
* find a way to deal with pywin32, which requires a specific package name and may require a specific url to deal with

In reality, it's also

* verify that every single build/test type has the proper configuration+behavior, including mock environments, otherwise we'll default to hitting pypi over the internet for python packages.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

ideally, pip will be using a mirror list in pip.conf.  That's installed for the buidler user and root on osx and linux, and we could do something similar on windows.

Comment 5

[Aki Sasaki (not active)]

Does that work inside a mock env?
We also need to make sure all current and future mock envs have that file.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

It wonn't work inside a mock env without some additional care to make sure ~cltbld/pip/pip.conf is accessible, so yes, that will need some attention.

Comment 7

[Chris AtLee [:catlee]]

Does pip support something like /etc/pip.conf? That may be easier to manage with puppet/mock. Our other option is to copy in ~cltbld/pip/pip.conf for all the builds that require it.

What's the magic required to get pip to look at a pip.conf file?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

It only looks in ~/.pip/pip.conf, although it does so by default.

 52 user_dir = os.path.expanduser('~')
 53 if sys.platform == 'win32':
...
 61 else:
 63     default_storage_dir = os.path.join(user_dir, '.pip')
 64     default_config_file = os.path.join(default_storage_dir, 'pip.conf')

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

This keeps closing the trees.  Please fix.

Comment 10

[Aki Sasaki (not active)]

We don't have a universal pip.conf solution yet, so I'm going to try to fix this in a different way.

pip -v install --download-cache ~/wrk/virtualenv/test/cache simplejson --find-links http://localhost --find-links http://repos/python/packages/ --no-index

works.

If I replace the --find-links URL lists with the actual list of servers we want to hit, that will remove the repos dependency for generic |pip install| commands.

I still have to deal with pointing at specific URLs in a different way.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

That list of servers is available in /etc/puppet/puppetmasters.txt on slaves, if you need a source.

Comment 12

[Aki Sasaki (not active)]

Trying: https://tbpl.mozilla.org/?tree=Ash&rev=b7f636fada9f

Pretty sure http://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness/rev/19e76dbd858a is good.

I have strong doubts about http://hg.mozilla.org/users/asasaki_mozilla.com/ash-mozharness/rev/9d9bb9762a53 ; that'll blow up windows talos if it's bad.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

The first is good for working around the tree-closing issues, but embedding that list in several places in a different repo is not good long-term.  So let's keep this open until we find a way to had that information off from puppet to mozharness.

Comment 14

[Aki Sasaki (not active)]

Attachment: repos

This patch:

* makes all virtualenv configs use a multi-server find_links, with a pip_index of None.  This makes us search our internal servers for the package, and ignore pypi.
* removes the unnecessary pypi_url
* removes the unnecessary {pip,distribute,pywin32}_url from the windows talos config: it looks like our new python installs handle this without these workarounds!
* removes the simplejson_url, and specifies the find_links instead.
* adds the ability to make |pip install| verbose
* gives a default virtualenv cache dir
* removes peptest configs since the script was removed in http://hg.mozilla.org/build/mozharness/rev/b12e5b16c32b .

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 792929 [details] [diff] [review]
repos

If pip uses these sources in order, then releng-puppet2.srv.releng.use1 is going to get pounded if any other puppetmaster goes down.  Leaving http://repos in the list helps with this - otherwise every slave would use use1.

The puppetmasters.txt files are different per host and per DC, with local puppetmasters first in random order, followed by remote puppetmasters in random order.  Even more reason to get to a point where puppet is providing the mirror list.

Comment 16

[Aki Sasaki (not active)]

(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> Comment on attachment 792929 [details] [diff] [review]
> repos
> 
> If pip uses these sources in order, then releng-puppet2.srv.releng.use1 is
> going to get pounded if any other puppetmaster goes down.  Leaving
> http://repos in the list helps with this - otherwise every slave would use
> use1.

... did you look at this patch?
repos is at the beginning of every list.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

I did look at the patch - that's why I said that.  I'd have said "don't commit it like this!" otherwise.

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 792929 [details] [diff] [review]
repos

Review of attachment 792929 [details] [diff] [review]:
-----------------------------------------------------------------

we may want to investigate shuffling and/or ordering the puppet masters in order of network distance. that can be done later though.

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 792929 [details] [diff] [review]
repos

https://hg.mozilla.org/build/mozharness/rev/4a47d92c2824

Should hopefully keep this from being a blocker bug, at least.


Discussed future approaches:

* wait for pip.conf to be available everywhere, including windows, and then test and use that.
* read puppetmasters.txt for the list, and if that doesn't exist, fall back to the find_links list.

Comment 20

[Aki Sasaki (not active)]

Will merge to production if this goes well:
https://tbpl.mozilla.org/?tree=Cedar&rev=0966feea2ada

Comment 21

[Aki Sasaki (not active)]

In production.
Is there a bug to get pip.conf everywhere?

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

That will be part of bug 798657.  If you'd like to add a dep bug for pip.conf specifically, go ahead.

Comment 23

[Aki Sasaki (not active)]

It looks like we still want a solution like this. We debated pipfiles via pipenv, but they're still fairly new and not quite as stable as we'd like.

We've moved from http://repos/ and the puppet servers to using pypi.{pub,pvt}.build.mozilla.org. This host has a bad ssl cert (bug 1399926; related: bug 1390946). We also point a bunch of unpinned installs at this host, so new uploads to the pub/ root tend to break unexpected things.

If we still want pip.conf usage, I see a few issues that need resolving:

* Bug 1399926 - we need a new ssl cert that covers the CNAMEs we'll be using for the host
* We probably want to leave pub/ alone, and start populating a new root or root(s) that require pinning.
* Since pip, setuptools, etc change their behavior across major releases, we need to standardize on a set of supported python and pip versions, and keep these up to date. 
  * We've started a releng doc for best python practices; those standards live there.
  * Releng puppet installs python 2.7.3 by default on CentOS. We have 2.7.12 for a small allowlist of hostnames.
    * We should flip this allowlist: by default install 2.7.12+, allowlist for 2.7.3; and we should ramp down this list to zero.
    * 2.7.12 is already out of date; we should package up 2.7.14 and 3.6.4 and prefer py3 wherever possible.
* We need pip.conf everywhere.
  * Bug 798657 was wontfixed, so we need to see if relops-maintained windows hosts have pip.conf.
    * It's possible all these hosts are running tools from in-tree, so something like bug 1437593 may solve a lot of these issues
  * We install python files in taskcluster - we need a pip.conf audit here as well, of both docker-image tasks and other tasks that run pip.
* Audit of all python processes
* Expand and pin all dependency lists, ideally with hashes; use those pinned deps. Update the pinned deps regularly.

Related: bug 1387039, bug 1448113, bug 1408199, bug 1341154, bug 1447057, probably more.

Comment 24

[Aki Sasaki (not active)]

>  * Bug 798657 was wontfixed, so we need to see if relops-maintained windows hosts have pip.conf.
>    * It's possible all these hosts are running tools from in-tree, so something like bug 1437593 may solve a lot of these issues

markco - Do you know if we install a pip.conf in our relops-maintained windows hosts?

Comment 25

[Mark Cornmesser [:markco]]

I don't believe we do. 

grenade: Do you know if pip.conf comes from anywhere in tree?

Comment 26

[Rob Thijssen [:grenade (EET/UTC+0300)]]

so i had a look and we do install a pip.ini file on test worker types [1] (but not on build worker types, i'm not sure why, or if there is a reason at all). The file is installed at: C:\ProgramData\pip\pip.ini. The content of the file is [2]:

[global]
disable-pip-version-check = true
trusted-host = pypi.pub.build.mozilla.org
download-cache = y:\pip-cache

if the content of this file is valid, maybe we should consider adding this file to builders (win 2012) as well?

[1]: https://github.com/mozilla-releng/OpenCloudConfig/search?utf8=%E2%9C%93&q=C%3A\\ProgramData\\pip\\pip.ini
[2]: https://github.com/mozilla-releng/OpenCloudConfig/blob/master/userdata/Configuration/pip.conf

Comment 27

[Aki Sasaki (not active)]

I think we're doing the right thing with python deps these days - largely through ./mach python and machine configurations.
I think this would be a non-mozharness bug if it were filed today.

Going to resolve incomplete for now. We should probably follow up with specific issues if they are known.