1405177 - release failure when boto3 and botocore present in our pypi mirrors

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P1)

Description

[Tom Prince [:tomprince]]

This also affects the other release scripts in the same directory when running on buildbot, which is true for thunderbird.

+++ This bug was initially created as a clone of Bug #1400746 +++

Bug 1361930 added boto3 and botocore to our mirrors, then the checksums task in the release automation makes some bad life choices:
  Using version 3-1.4.7 (newest of versions: 3-1.4.7, 3-1.4.7, 2.38.0, 2.38.0, 2.32.1, 2.32.1, 2.17.0, 2.17.0, core-1.7.7, core-1.7.7)

Notice it's just trimming the leading 'boto' off boto3-1.4.7 and botocore-1.7.7, and ends up installing boto3. This was fixed in pip 1.2 via https://github.com/pypa/pip/issues/510, but we're still on 1.1 because this is a job which runs via bbb on a linux host, and /tools/python27-virtualenv/lib/python2.7/site-packages/virtualenv_support/pip-1.1.tar.gz. Hrmph!

Comment 1

[Tom Prince [:tomprince]]

Attachment: Bug 1405177: Install (slightly) newer pip to install boto.

This is to avoid hitting https://github.com/pypa/pip/issues/510.

Review commit: https://reviewboard.mozilla.org/r/185884/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/185884/

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8914559 [details]
Bug 1405177: Install (slightly) newer pip to install boto.

https://reviewboard.mozilla.org/r/185884/#review190858

r+ if you remove the beet_mover.py change.

::: testing/mozharness/scripts/release/antivirus.py:83
(Diff revision 1)
>          BaseScript.__init__(self,
>              config_options=self.config_options,
>              require_config_file=False,
>              config={
>                  "virtualenv_modules": [
> +                    "pip==1.5.5",

This is not used by Firefox so the change is fine for TB.

::: testing/mozharness/scripts/release/beet_mover.py:151
(Diff revision 1)
>              # Default configuration
>              'config': {
>                  # base index url where to find taskcluster artifact based on taskid
>                  "artifact_base_url": 'https://queue.taskcluster.net/v1/task/{taskid}/artifacts/public/{subdir}',
>                  "virtualenv_modules": [
> +                    "pip==1.5.5",

This isn't used for TB, but is for Firefox in an environment where pip v.1.5.6 is available. Lets drop this hunk.

::: testing/mozharness/scripts/release/push-candidate-to-releases.py:74
(Diff revision 1)
>          BaseScript.__init__(self,
>              config_options=self.config_options,
>              require_config_file=False,
>              config={
>                  "virtualenv_modules": [
> +                    "pip==1.5.5",

This is shared by Firefox and Thunderbird, but Firefox runs it on a system with pip v1.5.6. I think that's OK because the pip changelog has only 
* Upgrade requests to 2.3.0 to fix an issue with proxies on Python 3.4.1 (PR #1821).
in 1.5.6, which is irrelevant to us. And mozharness should downgrade it OK, certainly 'pip install pip==1.5.5' works locally with 1.5.6 in place.

Comment 3

[Tom Prince [:tomprince]]

Comment on attachment 8914559 [details]
Bug 1405177: Install (slightly) newer pip to install boto.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/185884/diff/1-2/

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/706ffd585a89
Install (slightly) newer pip to install boto. r=nthomas

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/706ffd585a89

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/747911087c22

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/5ede87e18e66

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

We could probably back this out now that we have pip 8.1.2 in tree via bug 1297515.