1148087 - Windows pulls packages from pypi.python.org

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Michael Shal [:mshal]]

We have an internal pypi.pvt.build.mozilla.org for python packages, but some Windows builds appear to be pulling packages from pypi.python.org instead. Everything should be pulling from pypi.pvt.build.mozilla.org. I'm not entirely sure where the fix for this is - maybe a mozharness parameter or something we need to add to the Windows builders.

Comment 1

[Michael Shal [:mshal]]

Looks like this is for all windows builds, not just some.

Comment 2

[Michael Shal [:mshal]]

Q, is this something we'd need to add to GPO or puppet somewhere? OSX has this for example:

$ cat .pip/pip.conf 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
[install]
no-index = true
find-links = 
    http://pypi.pvt.build.mozilla.org/pub
    http://pypi.pub.build.mozilla.org/pub


Which comes from puppet: http://mxr.mozilla.org/build/source/puppet/modules/python/templates/user-pip-conf.erb

Comment 3

[Q]

Would need to be done in both gpo and puppett while we are keeping parity. It shouldn't be too hard to drop in

Comment 5

[Mark Cornmesser [:markco] OOO 2024/04/15]

This file is already on the Windows builders at C:\users\cltbld\.pip\pip.conf. 
Contents:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
[install]
no-index = true
disable-pip-version-check = true
find-links = 
    http://pypi.pvt.build.mozilla.org/pub
    http://pypi.pub.build.mozilla.org/pub
trusted-host = 
    pypi.pvt.build.mozilla.org
    pypi.pub.build.mozilla.org



mshal: Does this need to go to a different directory?

Comment 6

[Justin Wood (:Callek)]

> On Windows, the configuration file is: %HOME%\pip\pip.ini

https://pip.readthedocs.io/en/1.4/configuration.html

OR for a newer pip:

https://pip.readthedocs.io/en/stable/user_guide/#config-file

> On Windows the configuration file is %APPDATA%\pip\pip.ini.

Though ""There are also a legacy per-user configuration file which is also respected, these are located at:""  which matches the first location

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Retriggers are coming back green, so I'm going to reopen the trees, but this should still get fixed at some point soon, hopefully.

Comment 8

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: Bug1148087-correct-pip-conf-dir.patch

Comment 9

[Mark Cornmesser [:markco] OOO 2024/04/15]

Comment on attachment 8760955 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

If r+, I will land this post-London.

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 8760955 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

Review of attachment 8760955 [details] [diff] [review]:
-----------------------------------------------------------------

Even if you have a reason not to prefer the legacy location, this isn't correct because it shouldn't be '.pip' but rather the bareword 'pip' for directory name.

::: modules/python/manifests/user_pip_conf.pp
@@ +22,2 @@
>                  file {
> +                    "$homedir_/AppData/Roaming/.pip":

I'd actually prefer the "legacy" directory from that same ref, since it works for both older and newer pip versions.

$homedir_/pip/

@@ +24,2 @@
>                          ensure => directory;
> +                    "$homedir_/AppData/Roaming/.pip/pip.conf":

and here: $homedir/pip/pip.conf

Comment 11

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: Bug1148087-correct-pip-conf-dir.patch

Comment 12

[Justin Wood (:Callek)]

Comment on attachment 8760961 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

Review of attachment 8760961 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/python/manifests/user_pip_conf.pp
@@ +24,2 @@
>                          ensure => directory;
> +                    "$homedir_/pip/pip.conf":

User "Kwan" on IRC spotted this issue.

 <Kwan> Callek, markco: I might be missing something obvious, but why doesn't the patch for bug 1148087 also change the file ext to ini?

It absolutely should have an ini extension.

Comment 13

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: Bug1148087-correct-pip-conf-dir.patch

Comment 14

[Mark Cornmesser [:markco] OOO 2024/04/15]

Comment on attachment 8761646 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

My bad. I was trying to be quick in getting this done, and missed the obvious.

Comment 15

[Justin Wood (:Callek)]

Comment on attachment 8761646 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

Review of attachment 8761646 [details] [diff] [review]:
-----------------------------------------------------------------

No worries.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

markco, anything blocking the patch landing ?

Comment 17

[Mark Cornmesser [:markco] OOO 2024/04/15]

No, this just fail off my radar. I am going NI myself just as reminder and land it Monday.

Comment 18

[Mark Cornmesser [:markco] OOO 2024/04/15]

Comment on attachment 8761646 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

https://hg.mozilla.org/build/puppet/rev/d947598c0bc848c773ebe989295f0735574ba608

Comment 19

[Mark Cornmesser [:markco] OOO 2024/04/15]

This has broken a some things: 

https://treeherder.mozilla.org/logviewer.html#?job_id=38202986&repo=mozilla-inbound#L638

Comment 20

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: bug1148087revert.patch

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8804479 [details] [diff] [review]
bug1148087revert.patch

Review of attachment 8804479 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/python/manifests/user_pip_conf.pp
@@ +25,1 @@
>                          content => template("python/user-pip-conf.erb");

I'm confused by this. If we're aiming for %APPDATA%\pip\pip.ini (per the doc in comment #6), and I assume %APPDATA% expands to $homedir_/AppData/Roaming, then why the . in .pip ? To put it another way - when you tested, did you double check it was pulling from our pypi, rather than just not paying attention to the file and using pypi.python.org ?

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Oops, I was referring to this diff:

-                    "$homedir_/pip/pip.ini":
+                    "$homedir_/AppData/Roaming/.pip/pip.conf":

so pip.ini instead of pip.conf too.

Comment 23

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: bug1148087revert2.patch

Full revert, so we can get back to a working state,

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8761646 [details] [diff] [review]
Bug1148087-correct-pip-conf-dir.patch

Backed out to prevent failures after the next AMI generation run:
 https://hg.mozilla.org/build/puppet/rev/828c3b247deff1d2d31695d305380aaa0bc57bc3
 https://hg.mozilla.org/build/puppet/rev/8c4751e03a2fbd7475a108c3950ea79b37dd51b9

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8804493 [details] [diff] [review]
bug1148087revert2.patch

I just backed out the original patch, which we generally do without further review.

Comment 26

[Mark Cornmesser [:markco] OOO 2024/04/15]

mshal: What is the impact of this? And does this cause any security concerns? 

grenade: Do you know if this is an issue in task cluster as well? Has this been something you have already solved there?

Overall this does not seem to have a trivial fix. In the current state, with USERDIR\.pip\pip.conf, the file is not being used by python. When the file corrected to USERDIR\pip\pip.ini python uses the file but it causes additional issues. Particularly, python starts looking for the prerequisite files in the same directory. Output while running "c:\mozilla-build\python27\python.exe c:/mozilla-build/buildbotve/virtualenv.py -v --no-site-packages --distribute c:\builds\10":

Installing pip...
  Processing pip
  error: Couldn't find a setup script in C:\Users\Administrator\pip

Comment 27

[Michael Shal [:mshal]]

It's been a while, but I think I filed this bug because I added something to our internal pypi, and everything except Windows worked because it pulls from a different location. I don't know the exact reasons why we setup an internal pypi in the first place, but if I had to speculate I would guess that the goal is to avoid relying on external services being available for our builds to work. IOW, if pypi.python.org goes down, we suddenly can't do Windows builds and have no control in getting them working again. I could see security concerns being a reason as well. Might be worth finding someone more familiar with the history here to get some better answers :)

Comment 28

[Amy Rich [:arr] [:arich]]

It was to avoid pulling frmo external sites for stability and security reasons, but considering the amount of work likely needed to get this functional (aka, it doesn't appear to be easy), I think we should just fix it in taskcluster and move on.

Comment 29

[Rob Thijssen [:grenade (EET/UTC+0300)]]

taskcluster windows builders and testers pull packages from pypi.pub.build.mozilla.org and can additionally pull packages from hosts configured in mozharness configs (find_links). this is currently also limited to pypi.pub.build.mozilla.org.

tc workers use global config from C:\ProgramData\pip\pip.ini (downloaded from https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/master/userdata/Configuration/pip.conf during ami creation)

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

It'd be fantastic if we can pick up the taskcluster fix, but we'd have to watch out for version differences in pip and virtualenv which might result in different behaviour.

Comment 31

[Mark Cornmesser [:markco] OOO 2024/04/15]

We are being held back by the version, 1.5.5, of pip that is part of the buildbot virtual env. From the testing I have done, it seems like this version does not pick up the configuration in the ini or the conf file regardless of the location. An updated version will pick up the configuration from C:\ProgramData\pip\pip.ini.

If we want to have pip pull from our pypi repo we will need to update pip. Which I don't know if this will cause other issues.