Closed Bug 1297515 Opened 3 years ago Closed 2 years ago

Use modern pip when installing packages

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

defect
Not set

Tracking

(firefox-esr52 fixed, firefox56 fixed, firefox57 fixed, firefox58 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr52 --- fixed
firefox56 --- fixed
firefox57 --- fixed
firefox58 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

(Blocks 2 open bugs)

Details

Attachments

(3 files)

We want to start using --require-hashes to install packages everywhere for security and repeatability. However this feature was introduced in pip 8, while it looks like our automation is still using pip 1.5!

Mozharness should make sure the most recent version of pip gets set up. I think this probably just means updating virtualenv to something recent.
I had to do some investigating but I think I got it figured out. I'll document my findings for future reference.

the virtualenv used at https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/unittests/linux_unittest.py#36

comes from: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/virtualenv/prerequisites.pp#16

which uses this logic to get file contents: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/misc_python_file.pp#16

in the above misc_python_file.pp#16 you can see "python::virtualenv::settings::packages_dir_source" and that points to the url of the contents: https://dxr.mozilla.org/build-central/source/puppet/modules/python/manifests/virtualenv/settings.pp#12

in other words, it turns out that this virtualenv.py is downloaded from the same location as our python packages: http://puppetagain.pub.build.mozilla.org/data/python/packages/

that means to update it, we do: https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Other_Duties#Python_packages

ahal: we will need someone from releng to manually upload the new virtualenv.py to relengwebadm.private.scl3.mozilla.com. could you attach the one you want in this bug and provide expected shasum? Then needinfo me or someone from releng to upload it. I *think* that's all we have to do.

note to releng: for completeness, we should upload a copy of the current virtualenv.py and rename it to virtualenv-1.7.1.2.py or something in case we need to roll back or track history.
Flags: needinfo?(ahalberstadt)
Thanks for digging Jordan!

But I actually just noticed that we apparently vendor in virtualenv 15.0.1 in mozilla-central under python/virtualenv. Do you think there would be any issues if we instead tried to use that in mozharness? We could kill two birds with one stone, upgrade the package while moving more stuff in-tree. I'll look into this a bit more.
(Also sorry.. hope I didn't end up making you dig too long for nothing :/)
Did some digging. I think this is easy to do with taskcluster, because we can just package the in-tree virtualenv with the mozharness.zip generated by the build. However, it appears that buildbot does not use mozharness.zip. Using the in-tree virtualenv there looks like it needs a step added to buildbotcustom similar to this:
https://dxr.mozilla.org/build-central/source/buildbotcustom/process/factory.py#4349

Luckily python/virtualenv has existed for many many releases (at least back to esr-31), so we shouldn't run into any "doesn't exist on older branches" problems. But I guess we should ask whether doing this is worth it or not. Keeping in mind, we could potentially just update the puppetagain virtualenv.py for buildbot, and perhaps just switch to using the in-tree one for taskcluster. On the other hand, the buildbotcustom patch doesn't look particularly hard either..

Jordan, what do you think?
Flags: needinfo?(ahalberstadt) → needinfo?(jlund)
I am wary to use archiver for bundling this up and checking it out on buildbot workers like we do with mozharness. Archiver was meant to be a stopgap and some dev (me!) wrote it without much robustness.

I'm wondering, if all the tests that need virtualenv.py are driven by mozharness itself then we can safely assume the buildbot and tc workers have mozharness checked out already right?

In which case, we have option 3: put a copy of virtualenv.py in: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/external_tools/

in fact, I believe once upon a time we even did that. Maybe if we were to go with this route, we could use something like `hg copy python/virtualenv/virtualenv.py testing/mozharness/external_tools/virtualenv.py` or equivalent so a change to one file, updates both copies.
Flags: needinfo?(jlund)
Yeah, makes sense. I think I'm just going to stick to option 1 for now (update it on puppetagain). Using the in-tree copy of virtualenv would be nice, but doesn't really solve any major problems.. so I should probably just stay focused on fixing the task at hand. I'll post the file and shasum shortly.
Attached file virtualenv.py - 15.0.1
shasum:    a6d38e1bc4a8311c072893091080c4804ae89368
sha256sum: 46a69fc0ea759433fed0d7b084c83607b7992da972ae00260820f5f8071333cd

I grabbed this directly from python/virtualenv/virtualenv.py in-tree.
Flags: needinfo?(jlund)
as discussed over irc, this change may be more invasive than we want as it would affect every worker in our infra. Though we should update virtualenv and pip, I think solving this bug will be easier if we:

1) put a copy of virtualenv.py in mh
2) put a copy of virtualenv.py in tooltool
Flags: needinfo?(jlund)
I've started doing some of this work in bug 1304176. It is a massive rabbit hole.
This can probably be duped then. Using the in-tree copy was the ideal scenario and only wasn't chosen because of the rabbit hole you mention.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1304176
Even after bug 1304176 I still see pip 1.5 being used in automation, so I guess there are some more steps to do. Possibly it only got fixed for taskcluster.
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
It only got fixed where there is a source checkout. As we drive more things to source checkouts, this should naturally fix itself.
Blocks: 1315693
I see pip 1.5 still being used here:

03:35:04     INFO - [mozharness: 2017-02-22 11:35:04.029089Z] Running create-virtualenv step.
03:35:04     INFO - Running pre-action listener: _install_mozbase
03:35:04     INFO - Running pre-action listener: _pre_create_virtualenv
03:35:04     INFO - Running pre-action listener: _resource_record_pre_action
03:35:04     INFO - Running main action method: create_virtualenv
03:35:04     INFO - Creating virtualenv /builds/slave/test/build/venv
03:35:04     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/builds/slave/test/build/venv'] in /builds/slave/test/build
03:35:04     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /builds/slave/test/build/venv
03:35:04     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
03:35:04     INFO -  Using real prefix '/tools/python27'
03:35:04     INFO -  New python executable in /builds/slave/test/build/venv/bin/python
03:35:04     INFO -  Installing distribute.............................................................................................................................................................................................done.
03:35:05     INFO -  Installing pip..........
03:35:05     INFO -    Complete output from command /builds/slave/test/b...env/bin/easy_install /tools/misc-python/pip-1.5.5.tar.gz:
03:35:05     INFO -    Processing pip-1.5.5.tar.gz

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=79319808&lineNumber=492

OS X 10.10 debug Mochitest Mochitest M(2)
t-yosemite-r7-0354

Which presumably comes from:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#300
Same here, pip 1.5.5 being used and likely a cause of this intermittent failure:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=105456030&lineNumber=491

We really need to be using 8.x / 9.x everywhere...
Ancient pip / still using `distribute` is probably the cause of bug 1387970 fwiw.
As what I have seen when doing some checks for bug 1387970, we seem to only use pip 1.5.5 on OS X but not for Linux nor Windows TC workers.

Wouldn't it be more helpful to just update pip in general on the workers, without having to upgrade pip for each job?
In general, yes, we should update pip on the workers.

In reality, relying on the system's pip is difficult. When a modern pip is needed, we tend to use the in-tree copy that's part of the virtualenv package in third_party/python/virtualenv. But there can be a chicken-and-egg problem because a) many tasks don't have the VCS checkout at all b) many tasks need to do Python things requiring pip to obtain a VCS checkout. Choose your rabbit hole.
At least there is bug 1388816 now to also get pip upgraded on OS X workers.
Depends on: 1388816
Depends on: 1399926
See Also: → 1399926
Blocks: 1407769
I think this just needs to be fixed, even if it means having two copies of virtualenv in-tree. We can file a follow-up to remove the second copy once everything is off of buildbot.
Assignee: nobody → ahalberstadt
Status: REOPENED → ASSIGNED
Blocks: 1408051
Duplicate of this bug: 1406173
Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195022

::: testing/mozharness/mozharness/base/python.py
(Diff revision 1)
>          "action": "store",
>          "dest": "virtualenv_path",
>          "default": "venv",
>          "help": "Specify the path to the virtualenv top level directory"
>      }],
> -    [["--virtualenv"], {

huh, I guess this isn't used?

::: testing/mozharness/mozharness/base/python.py:364
(Diff revision 1)
>          dirs = self.query_abs_dirs()
>          venv_path = self.query_virtualenv_path()
>          self.info("Creating virtualenv %s" % venv_path)
>  
> -        # If running from a source checkout, use the virtualenv that is
> -        # vendored since that is deterministic.
> +        # Always use the virtualenv that is vendored since that is deterministic.
> +        # TODO Bug 1408051 - Use the copy of virtualenv under /third_party/python/virtualenv once everything is on buildbot

s/on/off/
Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

lgtm. Have you tested this anywhere?
Attachment #8917959 - Flags: review?(jlund) → review+
Comment on attachment 8917958 [details]
Bug 1297515 - Make a copy of third_party/python/virtualenv under testing/mozharness,

https://reviewboard.mozilla.org/r/188848/#review195028
Attachment #8917958 - Flags: review?(jlund) → review+
ahal, thanks for taking this and providing a sensible, least invasive stopgap
Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

Yeah, I ran a preliminary try push that had some buildbot tasks in it and things passed. I'll do another try run now with even more jobs.
Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195022

> huh, I guess this isn't used?

Oh yeah, I meant to ask you about that. It definitely isn't being used in mozilla-central anywhere. Is there a chance it's being used somewhere under hg.m.o/build?
re: comment 28
Flags: needinfo?(jlund)
(In reply to Andrew Halberstadt [:ahal] from comment #28)
> Comment on attachment 8917959 [details]
> Bug 1297515 - Always use the mozharness vendored copy of virtualenv,
> 
> https://reviewboard.mozilla.org/r/188850/#review195022
> 
> > huh, I guess this isn't used?
> 
> Oh yeah, I meant to ask you about that. It definitely isn't being used in
> mozilla-central anywhere. Is there a chance it's being used somewhere under
> hg.m.o/build?

I looked but couldn't find any instances.
Flags: needinfo?(jlund)
Comment on attachment 8917959 [details]
Bug 1297515 - Always use the mozharness vendored copy of virtualenv,

https://reviewboard.mozilla.org/r/188850/#review195026

Here's the new try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f91c95cfbd3135d4a38c3f0dca3c416430494b7a

I'll wait for osx results before pushing, but looks good so far.
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/da7ac021e120
Make a copy of third_party/python/virtualenv under testing/mozharness, r=jlund
https://hg.mozilla.org/integration/autoland/rev/5a44c9c16c21
Always use the mozharness vendored copy of virtualenv, r=jlund
Words cannot express how happy this bug makes me. A whole class of problems related to running an ancient virtualenv and pip in CI just went away. Good riddance.
Blocks: 1413341
Duplicate of this bug: 1162230
Duplicate of this bug: 1328927
Depends on: 1478467
You need to log in before you can comment on or make changes to this bug.