1421124 - fix py27 virtualenv in puppet

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Aki Sasaki (not active)]

Looks like we need a newer virtualenv.py and pip.
See https://bugzilla.mozilla.org/show_bug.cgi?id=1417575#c6 for details.

Comment 1

[Aki Sasaki (not active)]

Attachment: bug 1421124 - create a new python27 module.

We're creating a new module here so we don't upgrade the world. Instead, each module can point at the python27 module when they're ready.

This patch is intended to upgrade virtualenv and pip from our ancient pip 1.5.5 to virtualenv 15.0.2 + pip 8.1.2. I chose this version because virtualenv 15.1.0 includes pip 9.0.1; pip 9.x requires SSL; and our puppet servers don't have valid SSL certs. pip 8.1.2 allows for --trusted-host which allows us to fix the puppet bustage in this bug, but avoids requiring new SSL certs on the puppet servers.

Review commit: https://reviewboard.mozilla.org/r/203330/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/203330/

Comment 2

[Aki Sasaki (not active)]

Attachment: bug 1421124 - start using python27::virtualenv in balrog scriptworker.

Review commit: https://reviewboard.mozilla.org/r/203332/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/203332/

Comment 3

[Tom Prince [:tomprince]]

Comment on attachment 8932296 [details]
bug 1421124 - create a new python27 module.

https://reviewboard.mozilla.org/r/203330/#review208722

::: commit-message-6a752:5
(Diff revision 1)
> +bug 1421124 - create a new python27 module. r=callek
> +
> +We're creating a new module here so we don't upgrade the world. Instead, each module can point at the python27 module when they're ready.
> +
> +This patch is intended to upgrade virtualenv and pip from our ancient pip 1.5.5 to virtualenv 15.0.2 + pip 8.1.2. I chose this version because virtualenv 15.1.0 includes pip 9.0.1; pip 9.x requires SSL; and our puppet servers don't have valid SSL certs. pip 8.1.2 allows for --trusted-host which allows us to fix the puppet bustage in this bug, but avoids requiring new SSL certs on the puppet servers.

As far as I can tell, pip 9.0.1 supports --trusted-host as well.

Also, if the puppet servers have a valid, but not globaly trusted certificate, then pip has a --cert option to pass a ca-bundle too, which can include the puppet cert or CA for it.

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 8932296 [details]
bug 1421124 - create a new python27 module.

https://reviewboard.mozilla.org/r/203330/#review208964

I think this is fine, but I'd love it to get a run through travis before landing (via a fork+enabling of travis on your fork) so we can ensure it passes for linting at least.

::: modules/python27/manifests/virtualenv.pp
(Diff revision 1)
>                      group  => $ve_group;
>              }
> -            python::virtualenv::package {
> -                "${virtualenv}||pip==${python::virtualenv::settings::pip_version}":
> -                    user => $ve_user;
> -            }

Why are we no longer pinning pip here?

::: modules/python27/manifests/virtualenv/prerequisites.pp
(Diff revision 1)
> +            filename    => 'virtualenv_support',
> +            recurse     => true;
> +    } -> Anchor['python27::virtualenv::prerequisites::end']
>  
> -    # these two need to be in the same dir as virtualenv.py, or it will
> -    # want to download them from pypi

no longer true?

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 8932297 [details]
bug 1421124 - start using python27::virtualenv in balrog scriptworker.

https://reviewboard.mozilla.org/r/203332/#review208968

Comment 6

[Aki Sasaki (not active)]

(In reply to Justin Wood (:Callek) from comment #4)
> Comment on attachment 8932296 [details]
> bug 1421124 - create a new python27 module.
> 
> https://reviewboard.mozilla.org/r/203330/#review208964
> 
> I think this is fine, but I'd love it to get a run through travis before
> landing (via a fork+enabling of travis on your fork) so we can ensure it
> passes for linting at least.

Already done :)
https://github.com/escapewindow/build-puppet/tree/pip-py27

> ::: modules/python27/manifests/virtualenv.pp
> (Diff revision 1)
> >                      group  => $ve_group;
> >              }
> > -            python::virtualenv::package {
> > -                "${virtualenv}||pip==${python::virtualenv::settings::pip_version}":
> > -                    user => $ve_user;
> > -            }
> 
> Why are we no longer pinning pip here?

We download a pinned pip from the virtualenv_support dir with virtualenv.

> ::: modules/python27/manifests/virtualenv/prerequisites.pp
> (Diff revision 1)
> > +            filename    => 'virtualenv_support',
> > +            recurse     => true;
> > +    } -> Anchor['python27::virtualenv::prerequisites::end']
> >  
> > -    # these two need to be in the same dir as virtualenv.py, or it will
> > -    # want to download them from pypi
> 
> no longer true?

Distribute is old and obsolete. We have everything we need in the virtualenv_support dir.

Comment 7

[Aki Sasaki (not active)]

(In reply to Tom Prince [:tomprince] from comment #3)
> Comment on attachment 8932296 [details]
> bug 1421124 - create a new python27 module.
> 
> https://reviewboard.mozilla.org/r/203330/#review208722
> 
> ::: commit-message-6a752:5
> (Diff revision 1)
> > +bug 1421124 - create a new python27 module. r=callek
> > +
> > +We're creating a new module here so we don't upgrade the world. Instead, each module can point at the python27 module when they're ready.
> > +
> > +This patch is intended to upgrade virtualenv and pip from our ancient pip 1.5.5 to virtualenv 15.0.2 + pip 8.1.2. I chose this version because virtualenv 15.1.0 includes pip 9.0.1; pip 9.x requires SSL; and our puppet servers don't have valid SSL certs. pip 8.1.2 allows for --trusted-host which allows us to fix the puppet bustage in this bug, but avoids requiring new SSL certs on the puppet servers.
> 
> As far as I can tell, pip 9.0.1 supports --trusted-host as well.
> 
> Also, if the puppet servers have a valid, but not globaly trusted
> certificate, then pip has a --cert option to pass a ca-bundle too, which can
> include the puppet cert or CA for it.

Sounds like that's definitely worth a shot. I'd prefer to use ssl verification. However,
- I'm very aware we have limited time this quarter, and a lot of things we need to get done,
- upgrading from pip 1.5.5 -> pip 8.1.2 is a big improvement, even if it's not fully up to date, and
- this unblocks mtabara's tc relpro migration work, which was the impetus for this patch.

I'm leaning towards landing as-is and leaving virtualenv 15.1.0 as a followup.

Comment 8

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/190765e1620b
https://hg.mozilla.org/build/puppet/rev/bba38167655b