1297515 - Use modern pip when installing packages

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Andrew Halberstadt [:ahal]]

We want to start using --require-hashes to install packages everywhere for security and repeatability. However this feature was introduced in pip 8, while it looks like our automation is still using pip 1.5!

Mozharness should make sure the most recent version of pip gets set up. I think this probably just means updating virtualenv to something recent.

Comment 1

[Jordan Lund (:jlund)]

I had to do some investigating but I think I got it figured out. I'll document my findings for future reference.

the virtualenv used at https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/unittests/linux_unittest.py#36

comes from: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/virtualenv/prerequisites.pp#16

which uses this logic to get file contents: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/misc_python_file.pp#16

in the above misc_python_file.pp#16 you can see "python::virtualenv::settings::packages_dir_source" and that points to the url of the contents: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/virtualenv/settings.pp#12

in other words, it turns out that this virtualenv.py is downloaded from the same location as our python packages: http://puppetagain.pub.build.mozilla.org/data/python/packages/

that means to update it, we do: https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Other_Duties#Python_packages

ahal: we will need someone from releng to manually upload the new virtualenv.py to relengwebadm.private.scl3.mozilla.com. could you attach the one you want in this bug and provide expected shasum? Then needinfo me or someone from releng to upload it. I *think* that's all we have to do.

note to releng: for completeness, we should upload a copy of the current virtualenv.py and rename it to virtualenv-1.7.1.2.py or something in case we need to roll back or track history.

Comment 2

[Andrew Halberstadt [:ahal]]

Thanks for digging Jordan!

But I actually just noticed that we apparently vendor in virtualenv 15.0.1 in mozilla-central under python/virtualenv. Do you think there would be any issues if we instead tried to use that in mozharness? We could kill two birds with one stone, upgrade the package while moving more stuff in-tree. I'll look into this a bit more.

Comment 3

[Andrew Halberstadt [:ahal]]

(Also sorry.. hope I didn't end up making you dig too long for nothing :/)

Comment 4

[Andrew Halberstadt [:ahal]]

Did some digging. I think this is easy to do with taskcluster, because we can just package the in-tree virtualenv with the mozharness.zip generated by the build. However, it appears that buildbot does not use mozharness.zip. Using the in-tree virtualenv there looks like it needs a step added to buildbotcustom similar to this:
https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4349

Luckily python/virtualenv has existed for many many releases (at least back to esr-31), so we shouldn't run into any "doesn't exist on older branches" problems. But I guess we should ask whether doing this is worth it or not. Keeping in mind, we could potentially just update the puppetagain virtualenv.py for buildbot, and perhaps just switch to using the in-tree one for taskcluster. On the other hand, the buildbotcustom patch doesn't look particularly hard either..

Jordan, what do you think?

Comment 5

[Jordan Lund (:jlund)]

I am wary to use archiver for bundling this up and checking it out on buildbot workers like we do with mozharness. Archiver was meant to be a stopgap and some dev (me!) wrote it without much robustness.

I'm wondering, if all the tests that need virtualenv.py are driven by mozharness itself then we can safely assume the buildbot and tc workers have mozharness checked out already right?

In which case, we have option 3: put a copy of virtualenv.py in: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/external_tools/

in fact, I believe once upon a time we even did that. Maybe if we were to go with this route, we could use something like `hg copy python/virtualenv/virtualenv.py testing/mozharness/external_tools/virtualenv.py` or equivalent so a change to one file, updates both copies.

Comment 6

[Andrew Halberstadt [:ahal]]

Yeah, makes sense. I think I'm just going to stick to option 1 for now (update it on puppetagain). Using the in-tree copy of virtualenv would be nice, but doesn't really solve any major problems.. so I should probably just stay focused on fixing the task at hand. I'll post the file and shasum shortly.

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: virtualenv.py - 15.0.1

shasum:    a6d38e1bc4a8311c072893091080c4804ae89368
sha256sum: 46a69fc0ea759433fed0d7b084c83607b7992da972ae00260820f5f8071333cd

I grabbed this directly from python/virtualenv/virtualenv.py in-tree.

Comment 8

[Jordan Lund (:jlund)]

as discussed over irc, this change may be more invasive than we want as it would affect every worker in our infra. Though we should update virtualenv and pip, I think solving this bug will be easier if we:

1) put a copy of virtualenv.py in mh
2) put a copy of virtualenv.py in tooltool

Comment 9

[Gregory Szorc [:gps]]

I've started doing some of this work in bug 1304176. It is a massive rabbit hole.

Comment 10

[Andrew Halberstadt [:ahal]]

This can probably be duped then. Using the in-tree copy was the ideal scenario and only wasn't chosen because of the rabbit hole you mention.

Comment 11

[Andrew Halberstadt [:ahal]]

Even after bug 1304176 I still see pip 1.5 being used in automation, so I guess there are some more steps to do. Possibly it only got fixed for taskcluster.

Comment 12

[Gregory Szorc [:gps]]

It only got fixed where there is a source checkout. As we drive more things to source checkouts, this should naturally fix itself.

Comment 13

[Ed Morley [:emorley]]

I see pip 1.5 still being used here:

03:35:04     INFO - [mozharness: 2017-02-22 11:35:04.029089Z] Running create-virtualenv step.
03:35:04     INFO - Running pre-action listener: _install_mozbase
03:35:04     INFO - Running pre-action listener: _pre_create_virtualenv
03:35:04     INFO - Running pre-action listener: _resource_record_pre_action
03:35:04     INFO - Running main action method: create_virtualenv
03:35:04     INFO - Creating virtualenv /builds/slave/test/build/venv
03:35:04     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/builds/slave/test/build/venv'] in /builds/slave/test/build
03:35:04     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /builds/slave/test/build/venv
03:35:04     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
03:35:04     INFO -  Using real prefix '/tools/python27'
03:35:04     INFO -  New python executable in /builds/slave/test/build/venv/bin/python
03:35:04     INFO -  Installing distribute.............................................................................................................................................................................................done.
03:35:05     INFO -  Installing pip..........
03:35:05     INFO -    Complete output from command /builds/slave/test/b...env/bin/easy_install /tools/misc-python/pip-1.5.5.tar.gz:
03:35:05     INFO -    Processing pip-1.5.5.tar.gz

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=79319808&lineNumber=492

OS X 10.10 debug Mochitest Mochitest M(2)
t-yosemite-r7-0354

Which presumably comes from:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#300

Comment 14

[Ed Morley [:emorley]]

Same here, pip 1.5.5 being used and likely a cause of this intermittent failure:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=105456030&lineNumber=491

We really need to be using 8.x / 9.x everywhere...

Comment 15

[Ed Morley [:emorley]]

Ancient pip / still using `distribute` is probably the cause of bug 1387970 fwiw.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As what I have seen when doing some checks for bug 1387970, we seem to only use pip 1.5.5 on OS X but not for Linux nor Windows TC workers.

Wouldn't it be more helpful to just update pip in general on the workers, without having to upgrade pip for each job?

Comment 17

[Gregory Szorc [:gps]]

In general, yes, we should update pip on the workers.

In reality, relying on the system's pip is difficult. When a modern pip is needed, we tend to use the in-tree copy that's part of the virtualenv package in third_party/python/virtualenv. But there can be a chicken-and-egg problem because a) many tasks don't have the VCS checkout at all b) many tasks need to do Python things requiring pip to obtain a VCS checkout. Choose your rabbit hole.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

At least there is bug 1388816 now to also get pip upgraded on OS X workers.

Comment 19

[Andrew Halberstadt [:ahal]]

I think this just needs to be fixed, even if it means having two copies of virtualenv in-tree. We can file a follow-up to remove the second copy once everything is off of buildbot.

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1297515 - Make a copy of third_party/python/virtualenv under testing/mozharness,

This adds a second copy of virtualenv to the tree. We need to do this because
buildbot is downloading the 'testing/mozharness' directory directly from
hg.m.o. It would be hard and risky to also get it to download
'third_party/python/virtualenv'. So instead, we simply copy it to avoid the
issue.

Bug 1408051 is filed to remove this extra copy once everything is using
taskcluster.

Review commit: https://reviewboard.mozilla.org/r/188848/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/188848/

Comment 21

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

This removes the option to pass in a custom virtualenv binary and gets
mozharness to always use the copy in 'external_tools'. Because this is now
deterministic, we don't need to do things like check what version of pip is
being used.

Review commit: https://reviewboard.mozilla.org/r/188850/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/188850/

Comment 23

[Jordan Lund (:jlund)]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195022

::: testing/mozharness/mozharness/base/python.py
(Diff revision 1)
>          "action": "store",
>          "dest": "virtualenv_path",
>          "default": "venv",
>          "help": "Specify the path to the virtualenv top level directory"
>      }],
> -    [["--virtualenv"], {

huh, I guess this isn't used?

::: testing/mozharness/mozharness/base/python.py:364
(Diff revision 1)
>          dirs = self.query_abs_dirs()
>          venv_path = self.query_virtualenv_path()
>          self.info("Creating virtualenv %s" % venv_path)
>  
> -        # If running from a source checkout, use the virtualenv that is
> -        # vendored since that is deterministic.
> +        # Always use the virtualenv that is vendored since that is deterministic.
> +        # TODO Bug 1408051 - Use the copy of virtualenv under /third_party/python/virtualenv once everything is on buildbot

s/on/off/

Comment 24

[Jordan Lund (:jlund)]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

lgtm. Have you tested this anywhere?

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8917958 [details]
Bug 1297515 - Make a copy of third_party/python/virtualenv under testing/mozharness,

https://reviewboard.mozilla.org/r/188848/#review195028

Comment 26

[Jordan Lund (:jlund)]

ahal, thanks for taking this and providing a sensible, least invasive stopgap

Comment 27

[Andrew Halberstadt [:ahal]]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

Yeah, I ran a preliminary try push that had some buildbot tasks in it and things passed. I'll do another try run now with even more jobs.

Comment 28

[Andrew Halberstadt [:ahal]]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195022

> huh, I guess this isn't used?

Oh yeah, I meant to ask you about that. It definitely isn't being used in mozilla-central anywhere. Is there a chance it's being used somewhere under hg.m.o/build?

Comment 29

[Andrew Halberstadt [:ahal]]

re: comment 28

Comment 30

[Jordan Lund (:jlund)]

(In reply to Andrew Halberstadt [:ahal] from comment #28)
> Comment on attachment 8917959 [details]
> Bug 1297515 - Always use the mozharness vendored copy of virtualenv,
> 
> https://reviewboard.mozilla.org/r/188850/#review195022
> 
> > huh, I guess this isn't used?
> 
> Oh yeah, I meant to ask you about that. It definitely isn't being used in
> mozilla-central anywhere. Is there a chance it's being used somewhere under
> hg.m.o/build?

I looked but couldn't find any instances.

Comment 31

[Andrew Halberstadt [:ahal]]

Comment on attachment 8917958 [details]
Bug 1297515 - Make a copy of third_party/python/virtualenv under testing/mozharness,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/188848/diff/1-2/

Comment 32

[Andrew Halberstadt [:ahal]]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/188850/diff/1-2/

Comment 33

[Andrew Halberstadt [:ahal]]

Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

Here's the new try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f91c95cfbd3135d4a38c3f0dca3c416430494b7a

I'll wait for osx results before pushing, but looks good so far.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da7ac021e120
Make a copy of third_party/python/virtualenv under testing/mozharness, r=jlund
https://hg.mozilla.org/integration/autoland/rev/5a44c9c16c21
Always use the mozharness vendored copy of virtualenv, r=jlund

Comment 35

[Gregory Szorc [:gps]]

Words cannot express how happy this bug makes me. A whole class of problems related to running an ancient virtualenv and pip in CI just went away. Good riddance.

Comment 36

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/da7ac021e120
https://hg.mozilla.org/mozilla-central/rev/5a44c9c16c21

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/073a2a43a15c
https://hg.mozilla.org/releases/mozilla-beta/rev/c5ed62b37c2e

Comment 38

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/dab5c013eb9c
https://hg.mozilla.org/releases/mozilla-release/rev/be45e488b8b2

Comment 39

[Ryan VanderMeulen [:RyanVM]]

Follow-up for Windows buildbot jobs:
https://hg.mozilla.org/releases/mozilla-beta/rev/4868b6953a04
https://hg.mozilla.org/releases/mozilla-release/rev/978e942a3778

Comment 40

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-esr52/rev/b62dbb5c238933a1d13d9b9788de51c456d03392
https://hg.mozilla.org/releases/mozilla-esr52/rev/66104b2976058c495bcbe9f126c8642da94a5877

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/36e7393f09871a9aa05cf23bdf05ecf2ab1d2f4b