602908 - Deploy python 2.7.3 to all build machines

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

If we decide to use the patch in Bug 593585 as is, we'll need python 2.6 to use client.mk.  We could scope that down to Windows only, in which case this reduces to upgrading the mozilla-build version, but I'd prefer to keep the configure logic the same across the board.

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This is low priority until I have a patch for 593585 that works.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We're going to need this (at least on Windows) for Bug 563318.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

We're also about to upgrade the Buildbot and Twisted version on slaves, and make it consistent across the pool, so this could fit in nicely.

Comment 4

[Jesse Ruderman]

I guess we're not jumping straight to Python 2.7 because of bug 659881 / http://bugs.python.org/issue9516.

Comment 5

[LegNeato]

We don't need to track this for Firefox 8 as bug 563318 hasn't landed in that version

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Jesse Ruderman from comment #4)
> I guess we're not jumping straight to Python 2.7 because of bug 659881 /
> http://bugs.python.org/issue9516.

Can we, now? MozillaBuild 1.6 (addressing some MSVC issues) has been released and has Python 2.7.2, Lion has 2.7 by default, as do some of the recent Linux distributions (e.g. Ubuntu 11.10).

And bug 659881 has been fixed.

Comment 7

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

I did some probing into the build slaves and realized we're on a combination of 2.5, 2.5.1 and 2.5.2 for Windows (2003 Server?) / Mac Leopard / Linux.

Mac Snow Leopard is on 2.6.4.

It would be really nice to get us all on at least Python 2.6.x, or 2.7.x soon. Python 2.7.3 is coming out soon, too, 2.7.3 RC 1 was released on Feb 23, 2012.

This will also aid bug 636155 and bug 724191.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

historically we've done this sort of thing by installing a Python version to /tools/python-x.y.z with puppet and then using that path explicitly either to run apps or to set up virtualenvs.

Comment 9

[William Lachance (:wlach)]

After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I think we're all agreed that 2.7 is the desired target here. That's what will really help us get where we want to go (bug 724191).

Comment 10

[[e-mail] (:bhearsum)]

This would be fixed for Linux with bug 670707.

Comment 11

[Rain [:rain1] (inactive)]

According to #build, all our mozilla-central Windows builds are on Win64 machines, all of which have Python 2.6 by default. l10n repacks happen on Win32 machines, which have older python.

For pymake we need at least 2.6, which means that as long as we can use MSYS make to run l10n repacks we should be good on Windows.

Comment 12

[Chris Cooper [:coop] (he/him)]

Most recent version of python installed on each class of build machines:

bld-centos6-hp: 2.7.2 (/tools/python27/bin/python)
bld-lion-r5:    2.7.2 (/tools/python-2.7.2/bin/python)
w64-ix:         2.7.3 (/c/mozilla-build/python27/python.exe)

Notes:
* I've ignored the mw32-ix machines. We'll be off of them soon.
* I've also ignored the linux-ix/linux64-ix slaves because we'll be moving those builds to the hp builders/AWS (using mock) shortly.

Is that sufficiently consistent to start targeting python 2.7 on the builders? Should I work to get 2.7.3 (or greater) installed everywhere?

Comment 13

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> Is that sufficiently consistent to start targeting python 2.7 on the
> builders? Should I work to get 2.7.3 (or greater) installed everywhere?

Synchronizing to 2.7.3 would be ideal, though I can't think of anything off the top of my head that would differ from 2.7.2 to 2.7.3 at the moment.

(There has been historical cases of minor point release divergence causing issues and wasted developer time, e.g. during 2.5.x.)

Comment 14

[Rain [:rain1] (inactive)]

2.7.3 would be good, since it fixes at least one regression that might affect Pymake. http://bugs.python.org/issue11583

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Siddharth Agarwal [:sid0] from comment #14)
> 2.7.3 would be good, since it fixes at least one regression that might
> affect Pymake. http://bugs.python.org/issue11583

Sid, do you know of more Python bugs that might affect pymake that have landed in 2.7 dev branch but not made it into a release yet? (i.e. will be fixed in the future 2.7.4)

If not, 2.7.3 should be good to go as of this point in time.

coop, will the default python command be these 2.7.3 versions? Or does one have to specify the full directory to these binaries?

e.g.

Entering `python` gives default python which may not be 2.7

Entering `/tools/python-2.7.2/bin/python` gives 2.7

Comment 16

[Chris Cooper [:coop] (he/him)]

(In reply to Gary Kwong [:gkw, :nth10sd] from comment #15)
> coop, will the default python command be these 2.7.3 versions? Or does one
> have to specify the full directory to these binaries?
> 
> e.g.
> 
> Entering `python` gives default python which may not be 2.7
> 
> Entering `/tools/python-2.7.2/bin/python` gives 2.7

Not yet, no. For now if you require 2.7, you'll need to call it directly until I can run tests on each platform to make sure changing the default python version doesn't break the build process in some weird way.

I will try to get some linux/mac packages built for 2.7.3 this week. I think we (releng) would prefer to keep them identical anyway.

Comment 17

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> Not yet, no. For now if you require 2.7, you'll need to call it directly
> until I can run tests on each platform to make sure changing the default
> python version doesn't break the build process in some weird way.

Do they have an environment variable set (e.g. PYTHON27=<path>) so a script can access them across multiple platforms?

Also, is there a bug for changing the default python version?

Comment 18

[(not currently active) Ted Mielczarek]

I don't think there's a bug on increasing the version of Python required to build Mozilla yet, no. Seems preemptive until we get this rolled out.

Comment 19

[Chris Cooper [:coop] (he/him)]

I have a new 2.7.3 DMG ready for testing in staging, but the instructions for building new python packages for mock are somewhat involved. It may take a while there.

Comment 20

[Chris Cooper [:coop] (he/him)]

Python DMG creation instructions:

curl -LO http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
tar jxf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --enable-framework=/tools/python-2.7.3 \
  --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk/ \
  --prefix=/tools/python-2.7.3 --with-universal-archs=intel
make -j8
make install DESTDIR=$PWD/installroot
sh create-dmg.sh installroot/. python-2.7.3 python27 /

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

see also http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/mozilla/python27-dmg.sh

Comment 22

[Chris Cooper [:coop] (he/him)]

Attachment: Install python 2.7.3 on Lion (puppet-manifests)

Comment 23

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #21)
> see also
> http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/
> mozilla/python27-dmg.sh

Hrmm, do given the contents of https://hg.mozilla.org/build/puppet/file/dee2f60946dd/modules/packages/manifests/mozilla, should I also be building new DMGs for py27_mercurial and py27_virtualenv?

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

I believe so, yes.

Comment 25

[Chris Cooper [:coop] (he/him)]

Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Canceling review until I get the other packages ready too.

Comment 26

[Justin Wood (:Callek)]

Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Review of attachment 660843 [details] [diff] [review]:
-----------------------------------------------------------------

Note that the files dustin pointed at are "PuppetAgain" and for the 10.8 Mac machine, and Cent6 B2G builders, which are already using py2.7[.2]

I'm happy to update there as well if you like, since its simple, but I don't think we need to update the virtualenv/mercurial packages for 10.7/10.6/10.5 for this (I'm not sure if 10.7 has an hg/virtualenv tied to this version of python, doesn't look like it per: http://scl3-production-puppet.srv.releng.scl3.mozilla.com/production/darwin11-x86_64/build/DMGs/)

Comment 27

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> I'm happy to update there as well if you like, since its simple

Yes, please synchronize to Python 2.7.3.

Comment 28

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

fwiw, bug 804865 is the bug that requires Python 2.7 to build the tree. We now require Python 2.6 per bug 800614.

Comment 29

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Gregory mentions in bug 800614 comment 0 that releng builders all run Python 2.7. Coop, should this bug be updated?

Comment 30

[Chris Cooper [:coop] (he/him)]

They're still not all running 2.7.3 - it's a mix of 2.7.2 and 2.7.3. I'd still like to standardize on a single version.

Comment 31

[Chris Cooper [:coop] (he/him)]

Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

https://hg.mozilla.org/build/puppet-manifests/rev/b95fc6e3e5ef

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

I'm confused at the disconnect between this and bug 820238..

Comment 33

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> I'm confused at the disconnect between this and bug 820238..

Disconnect? Not sure I understand. 

Recall that lion builders aren't on PuppetAgain (yet), and we're deploying the same version in both places (2.7.3).

Comment 34

[Dustin J. Mitchell [:dustin] (he/him)]

In this bug we're looking to have the same version on all slaves; in the other bug, that seems to be not a concern.

Comment 35

[Chris Cooper [:coop] (he/him)]

Python 2.7.3 has replaced 2.7.2 on the Lion machines now.

Comment 36

[Mike Hommey [:glandium]]

Apparently, the windows builders on elm are not using python 2.7 at all, which a) prevents any merge from m-c b) broke my landing of the last version of bug 780561.

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Mike Hommey [:glandium] from comment #36)

The build logs have the path to the python27 install present (and early). The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the python 2.7 switch merged from m-c or a clobber might help.

Comment 38

[Mike Hommey [:glandium]]

(In reply to Nick Thomas [:nthomas] from comment #37)
> (In reply to Mike Hommey [:glandium] from comment #36)
> 
> The build logs have the path to the python27 install present (and early).
> The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the
> python 2.7 switch merged from m-c or a clobber might help.

I did clobber and retrigger, and that still used python2.6. AFAICT, the python 2.7 path is *not* in PATH, so it doesn't find python2.7, and finds python2.6. (the patch removing 2.6 support has not been merged yet)

Comment 39

[Justin Wood (:Callek)]

(In reply to Mike Hommey [:glandium] from comment #38)
> (the patch removing 2.6 support has not been merged yet)

Which needs to be done.

gps discovered (after initial backout of py2.7 req) that we don't have python2.7 binary in the PATH, so it still finds python2.6 by default :(

So you do need to land the remove 2.6 patch and clobber.

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

Which builders is this about?  w64-ix-slave*?

Comment 41

[Mike Hommey [:glandium]]

(In reply to Justin Wood (:Callek) from comment #39)
> (In reply to Mike Hommey [:glandium] from comment #38)
> > (the patch removing 2.6 support has not been merged yet)
> 
> Which needs to be done.
> 
> gps discovered (after initial backout of py2.7 req) that we don't have
> python2.7 binary in the PATH, so it still finds python2.6 by default :(
> 
> So you do need to land the remove 2.6 patch and clobber.

It worked. Thanks. Any reason we're not changing the PATH?

Comment 42

[[e-mail] (:bhearsum)]

I think this may have caused bug 858040.

Comment 43

[Gregory Szorc [:gps]]

I see from https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb we still have a few builders on Python 2.7.2. Now that MozillaBuild 1.7 is out and shipping with Python 2.7.4, I'd *really* like to bump our build requirements to 2.7.3 so we don't have to continue working around numerous bugs in older Python releases.

Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or 2.7.4)?

Comment 44

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #43) 
> Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or
> 2.7.4)?

Work is happening in bug 839560.

Comment 45

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like this was done in PuppetAgain, but for linux only.  So the Lion machines on puppetagain are using 2.7.2.  We should (a) pin the version of puppet on linux and (b) upgrade Darwin to 2.7.3.

Comment 46

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #45)
> It looks like this was done in PuppetAgain, but for linux only.  So the Lion
> machines on puppetagain are using 2.7.2.  We should (a) pin the version of
> puppet on linux and (b) upgrade Darwin to 2.7.3.

Is there a new bug for the python 2.7.3 deploy on Mac, or are we going to track that here?

Comment 47

[Dustin J. Mitchell [:dustin] (he/him)]

Between here and bug 882869 (two birds with one stone).

Comment 48

[Dustin J. Mitchell [:dustin] (he/him)]

OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:

  10.7/python27-2.7.3-1.dmg
  10.8/python27-2.7.3-1.dmg

All that remains is changing the versions in the puppet manifests, and verifying on a test system.

Comment 49

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #48)
> OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:
> 
>   10.7/python27-2.7.3-1.dmg
>   10.8/python27-2.7.3-1.dmg
> 
> All that remains is changing the versions in the puppet manifests, and
> verifying on a test system.

I've grabbed bld-lion-r5-043 if we want to try the 10.7 package there.

We can handle 10.8 (and other test platforms) in follow-up bugs.

Comment 50

[Dustin J. Mitchell [:dustin] (he/him)]

How do you want to work that?  Probably the easiest way is to pin that host to your environment, and change the version number in that environment.

Comment 51

[Dustin J. Mitchell [:dustin] (he/him)]

We should get this going - we're about to introduce a bunch of other mac silos on puppetagain, and it'd be great to just bring them up with 2.7.3.

Comment 52

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #50)
> How do you want to work that?  Probably the easiest way is to pin that host
> to your environment, and change the version number in that environment.

The slave is already attached to my buildbot dev master, or are you talking about puppet-specific changes here?

Comment 53

[Dustin J. Mitchell [:dustin] (he/him)]

Puppet-specific changes.  You can "pin" a host to a particular puppet server and environment on that server (similar to allocating a slave to a master).  So the suggestion is to set up a puppet environment for yourself on a server of your choice, change the version number in that environment without checking in, and then pin a host or two to that environment.

Comment 54

[Chris Cooper [:coop] (he/him)]

I have a build running in staging now to test this. The PATH already tells me that it's using 2.7.3, but I want to make sure we're not missing a module or something.

Comment 55

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #54)
> I have a build running in staging now to test this. The PATH already tells
> me that it's using 2.7.3, but I want to make sure we're not missing a module
> or something.

Build ran green. I'm happy here.

Comment 56

[Dustin J. Mitchell [:dustin] (he/him)]

Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+ from me assuming the patch is just a version-number change.

Comment 57

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #56)
> Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+
> from me assuming the patch is just a version-number change.

https://hg.mozilla.org/build/puppet/rev/401e2c4bfc39

Comment 58

[Chris Cooper [:coop] (he/him)]

Deployed to the bld-lion-r5 builders.

Comment 59

[Ed Morley [:emorley]]

Just be aware that ESR17 is still using Python 2.5 (though we'll be rid of it in a few months at least) - hence bugs like bug 914965.

Comment 60

[Ed Morley [:emorley]]

And bug 915075.

Comment 61

[Jordan Lund (:jlund)]

Looks like some build machines are still running <2.7.

I hit failed builds while trying to do dict comprehensions on slaves running 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build slaves) 

maybe for for the logical reasons we had a while back: https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up upgrading these slaves.

Comment 62

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #61)
> Looks like some build machines are still running <2.7.
> 
> I hit failed builds while trying to do dict comprehensions on slaves running
> 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build
> slaves) 
> 
> maybe for for the logical reasons we had a while back:
> https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up
> upgrading these slaves.

It's there, but it's not the default.

[[e-mail] ~]$ which python2.7
/usr/local/bin/python2.7
[[e-mail] ~]$ python2.7 --version
Python 2.7.3

Comment 63

[Chris Cooper [:coop] (he/him)]

Jordan: are the dict comprehensions you're doing happening inside of mock, or outside as part of the factory? Can you target 2.7 directly? 

I believe the mock env itself relies on python 2.6 which is why we've opted to install 2.7 in parallel here rather than simply upgrading.

Comment 64

[Jordan Lund (:jlund)]

this was patch for script.py (used everywhere in mozharness). I used the following syntax since I assume we want to stay compatible with <2.7:
    dict((key_expr, value_expr) for targets in iterable)

but to answer, this was outside of mock. Some slaves have 2.6 as default, as you mentioned above. This will be what mozharness uses when the script is called from bbot (I think buildbot uses `which python` and is not explicit)

I am not sure what dep mock_mozilla has but if it needs 2.6 then that makes sense. I do know that after mock is setup, if you supply 'python' in list of mock_packages, it will install 2.6.6. I think that is the default in whatever cache/dir the mock packages come from (maybe it's yum or some local file cache).

At any rate, it sounds like we need to have both. I could reach 2.7 if I need but to not mess with mozharness requirements, I won't use it.

closing since we do have 2.7 installed (but not as default)