1003729 - Update version of pip installed on automation machines from 0.8.2 to 1.5.5

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ed Morley [:emorley]]

From this log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=38772224&tree=Fx-Team#error3
Ubuntu VM 12.04 x64 fx-team pgo test mochitest-e10s-4
tst-linux64-spot-625

I happened to notice we're still using pip 0.8.2 which was released sometime before 2011.

We should update to something newer - current latest version is 1.5.4:
http://www.pip-installer.org/en/latest/news.html

Comment 1

[Ed Morley [:emorley]]

http://mxr.mozilla.org/build-central/source/puppet-manifests/modules/python/manifests/virtualenv.pp#78
78     $pip_req = Python::Package_dir_file["pip-0.8.2.tar.gz"]

http://mxr.mozilla.org/build-central/source/puppet/modules/python/manifests/virtualenv/prerequisites.pp#23
21     Anchor['python::virtualenv::prerequisites::begin'] ->
22     python::misc_python_file {
23         "pip-0.8.2.tar.gz": ;
24     } -> Anchor['python::virtualenv::prerequisites::end']

Comment 2

[Ed Morley [:emorley]]

Meant to add:
Updating to newer pip would mean we get https://github.com/mihneadb/pip/commit/e879b02416168bfed0ec65f5c09e530d6c368a4b which would fix the noisy tracebacks that we see occasionally in the logs & that currently aggravates the logparser on TBPL (see tbpl brief log in comment 0).

Comment 3

[Armen [:armenzg]]

Hi dustin, is this something that you can help us with? or guide us with?

Thanks!

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like Ed has pretty much solved it already..

Comment 5

[Armen [:armenzg]]

Where do we place the newer tar ball?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Sorry, that shoudl go in PuppetAgain's python packages repository (releng-puppet2.srv.releng.scl3:/data/python/packages, then let it replicate out to the other masters).

Comment 7

[Justin Wood (:Callek)]

...also make sure its on the public pypi for us on our webcluster

Comment 8

[Armen [:armenzg]]

I've uploaded 1.5.5:
[[e-mail] packages]# ls -l pip-*
-rw-r--r-- 1 puppetsync puppetsync  106126 Mar  7  2012 pip-0.8.2.tar.gz
-rw-r--r-- 1 puppetsync puppetsync 1084356 May  2 23:26 pip-1.5.5.tar.gz

[[e-mail]3 ~]$ ls -l /mnt/netapp/relengweb/pypi/pub/pip-*
-rw-r--r-- 1 root root     106126 Mar  7  2012 /mnt/netapp/relengweb/pypi/pub/pip-0.8.2.tar.gz
-rw-r--r-- 1 root armenzg 1084356 May  2 23:26 /mnt/netapp/relengweb/pypi/pub/pip-1.5.5.tar.gz

Comment 9

[Armen [:armenzg]]

Attachment: pip.diff

I can write the patch but I won't be deploying it before Friday in the early EDT morning.

edmorley: if you want to deploy this you can do it after checking with someone on releng on the early mornings of EDT. All it is needed is to merge to the production branch.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8421893 [details] [diff] [review]
pip.diff

Lgtm, but I'd want some testing before landing this!

Comment 11

[Armen [:armenzg]]

Grabbing or I will forget.

Comment 12

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=40549225&tree=Mozilla-Inbound

Comment 13

[Armen [:armenzg]]

I've tested that the tar ball gets deployed [1] to the machine and that creating a venv will have the newer pip [2].

dustin, I have tried reverting, however, it does not undo the deployment. As it is, once deployed there's no way back.


[1]
[root@talos-linux64-ix-002 ~]# ls -lrt /tools/misc-python/ | grep "pip"
-rw-r--r-- 1 root root  106126 Mar 14  2013 pip-0.8.2.tar.gz
-rw-r--r-- 1 root root    1226 Mar 14  2013 pip-check.py
-rw-r--r-- 1 root root 1084356 Jun  6 13:00 pip-1.5.5.tar.gz

[2]
[root@talos-linux64-ix-002 ~]# python /tools/misc-python/virtualenv.py test
[root@talos-linux64-ix-002 ~]# ls test/lib/python2.7/site-packages/ | grep "pip"
pip-1.5.5-py2.7.egg
[root@talos-linux64-ix-002 ~]# source test/bin/activate
(test)[root@talos-linux64-ix-002 ~]# pip --version
pip 1.5.5 from /root/test/local/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg (python 2.7)

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, that pip package gets installed into the virtualenv, so just removing the package doesn't change the version in the virtualenv.  It will be the same for upgrades: an existing virtualenv will continue to use the old pip.

If this is problematic, then the python::virtualenv class could be modified to specify a specific version of pip along with the other packages for which it manages versions.  That would basically mean adding

  python::virtualenv::package {
      "pip=${pip_version}":
          user => $ve_user;
  }

where $pip_version is a new argument to python::virtualenv, with a default of 1.5.5., allowing us to change it easily in the future.

Comment 15

[Armen [:armenzg]]

Attachment: puppet.diff

It takes me much longer to figure this out than somebody who knows puppet.
I don't know what I'm doing and it is derailing me from the projects I need to get done.
I was just trying to help in here.

Notice: /Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
Notice: /Stage[main]/Python::Virtualenv::Prerequisites/Python::Misc_python_file[pip-1.5.5.tar.gz]/File[/tools/misc-python/pip-1.5.5.tar.gz]/ensure: defined content as '{md5}7520581ba0687dec1ce85bd15496537b'
Notice: Finished catalog run in 20.34 seconds
[root@talos-linux64-ix-002 ~]# ls /tools/misc-python/
distribute-0.6.24.tar.gz  pip-0.8.2.tar.gz  pip-1.5.5.tar.gz  pip-check.py  virtualenv.py  virtualenv.pyc
[root@talos-linux64-ix-002 ~]# rm -rf test && python /tools/misc-python/virtualenv.py test && ls test/lib/python2.7/site-packages/ | grep "pip"
New python executable in test/bin/python
Installing setuptools................................done.
Installing pip.................done.
pip-1.5.5-py2.7.egg

Comment 16

[Armen [:armenzg]]

dustin, do you think you could help us find someone that could help in here? Thank you.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

As soon as the company hires me some minions, I'll get one of them right on it! :)

The patch as it stands looks like it's pretty close.

Comment 18

[Armen [:armenzg]]

coop: anyone that could look into this?

Comment 19

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #18)
> coop: anyone that could look into this?

At present, no.

As always, we're treating Windows like the red-headed stepchild. The bug is marked as all platforms: does the pip version on Windows need to change too? Has anyone done any investigation there?

Comment 20

[Ed Morley [:emorley]]

(In reply to Chris Cooper [:coop] from comment #19)
> (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #18)
> > coop: anyone that could look into this?
> 
> At present, no.
> 
> As always, we're treating Windows like the red-headed stepchild. The bug is
> marked as all platforms: does the pip version on Windows need to change too?
> Has anyone done any investigation there?

The reason for the urgency for this, is to fix bug 1007230. That is currently only occurring on !Windows. There isn't a platform selection for "several but not all platforms".

Comment 21

[Justin Wood (:Callek)]

I am taking this as a "kick it out" style thing to breakup my work with something different, per coop's ok.

My target goal here is Windows and Mac by next week.

:ed, to circle back to full scope of requirements, am I correct in that:

Required to suit your needs
* All Linux [CentOS] Build Machines
* All Linux [Ubuntu] Test Machines
* All Mac OSX Build machines
* All Mac OSX Test machines
* No existing virtual environments need to be forcefully recreated (normal clobbers, etc will do this for us)

Useful but not a requirement for this bug:
* Foopies (Tegra/Panda tests)
* All Linux [CentOS] Server Type machines
 ^ e.g. AWS Manager, Buildbot masters, ... etc
* Windows Build Machines
* Windows Physical testers
* Windows AWS Testers
* Production Systems not in direct Releng Control:
  ^ e.g. relengweb cluster (buildata, buildapi, relengapi, tbpl)
* Anything not inclusively mentioned in above two lists.

Comment 22

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #21)
> I am taking this as a "kick it out" style thing to breakup my work with
> something different, per coop's ok.
> 
> My target goal here is Windows and Mac by next week.

s/Windows/Linux/ --- that was an odd typo

Comment 23

[Justin Wood (:Callek)]

Attachment: [puppet] v1 - pin current pip ver

This has been tested.

It also has been tested that it can downgrade an existing pip==1.5.5 to 0.8.4 as present. So rollback is easy, *even for existing venvs*.

Thanks armen for the stepping stones here

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8441730 [details] [diff] [review]
[puppet] v1 - pin current pip ver

me gusta

Comment 25

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #21)
> I am taking this as a "kick it out" style thing to breakup my work with
> something different, per coop's ok.
> 
> My target goal here is Linux and Mac by next week.

That's great - thank you :-)

> :ed, to circle back to full scope of requirements, am I correct in that:

Yup both lists look good & happy with letting clobbers do their thing gradually over time :-)

Comment 26

[Justin Wood (:Callek)]

Attached patch landed:

http://hg.mozilla.org/build/puppet/rev/9a0ece91bbea
http://hg.mozilla.org/build/puppet/rev/36a56eac1348 (merge)

That didn't spur any errors/problems so landed the bump:

http://hg.mozilla.org/build/puppet/rev/158dc4354ed3
http://hg.mozilla.org/build/puppet/rev/e3b01e7333eb (merge)


All the following from c#21, including the foopies and Linux server-type things (we got those for free)

(In reply to Justin Wood (:Callek) from comment #21)
> Required to suit your needs
> * All Linux [CentOS] Build Machines
> * All Linux [Ubuntu] Test Machines
> * All Mac OSX Build machines
> * All Mac OSX Test machines
> * No existing virtual environments need to be forcefully recreated (normal
> clobbers, etc will do this for us)
> 
> Useful but not a requirement for this bug:
> * Foopies (Tegra/Panda tests)
> * All Linux [CentOS] Server Type machines
>  ^ e.g. AWS Manager, Buildbot masters, ... etc

That said, any venv's on said machines that are not "managed" by puppet may need to be recreated, but buildbot is managed, so I think we're relatively safe here.

Production Jobs may want/need a clobber to take advantage of this, I'll leave that to sheriff hands though

Comment 27

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #26)
> Production Jobs may want/need a clobber to take advantage of this, I'll
> leave that to sheriff hands though

Have clobbered trunk trees since we had another occurrence of bug 1007230 with 0.8.2 shown in the traceback paths. Thank you for sorting this! :-)

Comment 28

[Ed Morley [:emorley]]

The logs in bug 1007230 unfortunately show pip 0.8.2 still being in use:

b2g_emulator_vm mozilla-inbound opt test reftest-4 on 2014-06-19 02:46:53 PDT for push 9bd1ca579412
https://tbpl.mozilla.org/php/getParsedLog.php?id=42037877&tree=Mozilla-Inbound
02:48:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/pip-0.8.2-py2.7.egg/pip/download.py", line 84, in __call__
02:48:53     INFO -      response = urllib2.urlopen(self.get_request(url))

Would it be possible to refresh the venvs at builds/slave/test/build/venv/ since this is on a test machine and so the clobbering I did earlier won't help?

Comment 29

[Ed Morley [:emorley]]

Though it seems we regenerate the venv every test run (as one would hope):
04:02:42     INFO - ##### Running create-virtualenv step.
04:02:42     INFO - #####
04:02:42     INFO - Running pre-action listener: _pre_create_virtualenv
04:02:42     INFO - Running pre-action listener: _resource_record_pre_action
04:02:42     INFO - Running main action method: create_virtualenv
04:02:42     INFO - Creating virtualenv /builds/slave/test/build/venv
04:02:42     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/builds/slave/test/build/venv'] in /builds/slave/test/build
04:02:42     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /builds/slave/test/build/venv
04:02:44     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
04:02:44     INFO -  Using real prefix '/usr'
04:02:44     INFO -  New python executable in /builds/slave/test/build/venv/bin/python
04:02:47     INFO -  Installing distribute.............................................................................................................................................................................................done.
04:02:55     INFO -  Installing pip.................done.
04:02:55     INFO - Return code: 0

That seems to be done from:
http://mxr.mozilla.org/build-central/source/mozharness/mozharness/base/python.py#318

Comment 30

[Justin Wood (:Callek)]

So, aki,

What am I missing here:

http://pypi.pub.build.mozilla.org/pub/ has pip 1.5.5
http://puppetagain.pub.build.mozilla.org/data/python/packages/ has pip 1.5.5

the test (e.g. log in c#28) seems to show the test dir being clobbered
The log also shows the venv being created
This patch added pip 1.5.5 to the virtualenv.py path.

Afaict it should have installed pip 1.5.5 automatically, or do I need to add the self.install_module thing to mozharness directly?

Comment 31

[Aki Sasaki (not active)]

I don't think we actually install pip from pypi.p.b.m.o.
I think pip gets automatically gets installed when you call virtualenv:

(mh)deathduck:/src [12:01:47]
514$ virtualenv foo
New python executable in foo/bin/python
Installing setuptools............done.
Installing pip...............done.
(mh)deathduck:/src [12:01:51]
515$ foo/bin/pip --version
pip 1.2.1 from /src/foo/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg (python 2.7)

I think upgrading virtualenv on the box will upgrade pip; this seems like the cleanest option, but would require puppet + windows changes.


We can also upgrade pip by specifying version, but that would require doing so for every config file that we want to run the new pip.  Because of that I think upgrading virtualenv is preferable.

(mh)deathduck:/src [12:03:05]
520$ foo/bin/pip install pip==1.5.5
Downloading/unpacking pip==1.5.5
  Downloading pip-1.5.5.tar.gz (1.1MB): 1.1MB downloaded
  Running setup.py egg_info for package pip

    warning: no files found matching 'pip/cacert.pem'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip
  Found existing installation: pip 1.2.1
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching 'pip/cacert.pem'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /src/foo/bin
    Installing pip2.7 script to /src/foo/bin
    Installing pip2 script to /src/foo/bin
Successfully installed pip
Cleaning up...
(mh)deathduck:/src [12:03:35]
521$ foo/bin/pip --version
pip 1.5.5 from /src/foo/lib/python2.7/site-packages (python 2.7)


Not specifying the version would result in not actually fixing this bug:
(mh)deathduck:/src [12:04:27]
524$ foo/bin/pip install pip
Requirement already satisfied (use --upgrade to upgrade): pip in ./foo/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg
Cleaning up...

Comment 32

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #31)
> I think upgrading virtualenv on the box will upgrade pip; this seems like
> the cleanest option, but would require puppet + windows changes.

We should verify that it's upgrading virtualenv that does it, and not upgrading python itself (or both).

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

Virtualenv contains an embedded copy of pip.  Mozharness appears to already have some flexibility to support this:

https://github.com/mozilla/build-mozharness/blob/master/mozharness/base/python.py#L341
        for module in ('distribute', 'pip'):
            if c.get('%s_url' % module):
                self.download_file(c['%s_url' % module],
                                   parent_dir=dirs['abs_work_dir'])

which will leave a pip-*.tar.gz in the cwd for virtualenv to find.  So, one option is to just pick a useful default value for pip_url.

The other option is what Callek's done for puppet: *after* the virtualenv is installed, install a specific version of pip using 'pip install -U pip==....'.  This would end up being another config variable, but at least it doesn't have to be a URL, and can default to "latest I can find" (so just 'pip install -U pip').

Comment 34

[Aki Sasaki (not active)]

Ah, right, pip_url.
And the pip install -U can be done via adding pip==1.5.5 to the virtualenv_modules list.  I still think upgrading virtualenv would be preferable than having to specify this value in every config file in perpetuity, but both will work.

Comment 35

[Justin Wood (:Callek)]

:edmorley, :aki,

so how do we want to approach this, I'm not groking the difference here.

Taking the full log https://tbpl.mozilla.org/php/getParsedLog.php?id=42037877&full=1&branch=mozilla-inbound

* we clobber fully
* We create a venv from /tools/misc-python/virtualenv.py
*** virtualenv.py has logic to also grab a pip version thats beside it if one exists! (pip 1.5.5.tar.gz in this case)


In my test just now I *confirmed* that works:
[cltbld@talos-linux64-ix-071 ~]$ /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute ./t
est-venv
The --no-site-packages flag is deprecated; it is now the default behavior.
Using real prefix '/usr'
New python executable in ./test-venv/bin/python
Installing distribute............................................................................................................
.................................................................................done.
Installing pip.................done.
[cltbld@talos-linux64-ix-071 ~]$ ./test-venv/bin/pip --version
pip 1.5.5 from /home/cltbld/test-venv/local/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg (python 2.7)
[cltbld@talos-linux64-ix-071 ~]$  ls ./test-venv/local/lib/python2.7/site-packages/
distribute-0.6.24-py2.7.egg  easy-install.pth  pip-1.5.5-py2.7.egg  setuptools.pth
[cltbld@talos-linux64-ix-071 ~]$ ls -la /tools/misc-python/
total 1972
drwxr-xr-x 2 root root    4096 Jun 20 12:54 .
drwxr-xr-x 7 root root    4096 Jan 17 11:39 ..
-rw-r--r-- 1 root root  620771 Aug 30  2013 distribute-0.6.24.tar.gz
-rw-r--r-- 1 root root  106126 Aug 30  2013 pip-0.8.2.tar.gz
-rw-r--r-- 1 root root 1084356 Jun 18 12:30 pip-1.5.5.tar.gz
-rw-r--r-- 1 root root    1226 Aug 30  2013 pip-check.py
-rw-r--r-- 1 root root  101937 Aug 30  2013 virtualenv.py
-rw-r--r-- 1 root root   86432 Jun 20 12:54 virtualenv.pyc

(So this is additionally *with* pip 0.8.2 also present, fwiw)


..... unless we're silently failing 
02:47:07     INFO - #####
02:47:07     INFO - ##### Running clobber step.
02:47:07     INFO - #####
02:47:07     INFO - Running pre-action listener: _resource_record_pre_action
02:47:07     INFO - Running main action method: clobber
02:47:07     INFO - rmtree: /builds/slave/test/build
02:47:07     INFO - retry: Calling <function rmtree at 0x262c5f0> with args: ('/builds/slave/test/build',), kwargs: {}, attempt #1
02:47:13     INFO - Running post-action listener: _resource_record_post_action

Comment 36

[Aki Sasaki (not active)]

Please try upgrading virtualenv.

Comment 37

[Justin Wood (:Callek)]

(In reply to Aki Sasaki [:aki] from comment #36)
> Please try upgrading virtualenv.

Based on my investigations I'm more worried that we'll just be papering over a real problem, the lack of clobber here.... can you confirm that its not silently broken, and/or how I can verify it based on the log?

Or can you explain why mozharness is *not* working as expected vs my manual testing here, I can't see *any* difference between what mozharness is doing and what I'm testing except the different outcome.

If upgrading virtualenv is the only viable solution, great, I just don't think ignoring the differences in testing and actual expectations is a good thing.

Comment 38

[Aki Sasaki (not active)]

The absolute path of the virtualenv and the absolute path of the clobber are in the log.

Comment 39

[Justin Wood (:Callek)]

So, sorry for churn everyone ---- due to a pileup of things, most of which are relatively new things in releng; heres what I think the case is above:

* AWS (spot) instances don't run puppet on every boot
 -- We instead once-a-day build brand-new instances from puppet and at-boot we merely check the AMI version we are booting with, and if its wrong the instance gets recreated.
* This patch landed mid-day on wednesday
* We *started* building the new AMI (with this patch) at 01:20 am PT
 -- I have seen this take up to an hour and a half to complete.
* The job referenced above was started at 02:46:53 PT

So its VERY likely it was just bad timing.

Leaving needinfo to edmorley to see if he sees this again in the next 1-2 days. If so I'll need to dive back in, otherwise I'll call it fixed.

Comment 40

[Ed Morley [:emorley]]

The logs in bug 1007230 all show 1.5.5 now - guessing it was just timing per comment 39.
Thank you for this! :-)