Deploy python 2.7.3 to all build machines

RESOLVED FIXED

Status

Release Engineering
Buildduty
P2
normal
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: khuey, Assigned: coop)

Tracking

(Blocks: 1 bug)

Dependency tree / graph

Firefox Tracking Flags

(firefox8-)

Details

(Whiteboard: [buildslaves][puppet][buildfaster:p2])

Attachments

(1 attachment)

If we decide to use the patch in Bug 593585 as is, we'll need python 2.6 to use client.mk.  We could scope that down to Windows only, in which case this reduces to upgrading the mozilla-build version, but I'd prefer to keep the configure logic the same across the board.

Updated

7 years ago
OS: Windows 7 → All
Whiteboard: [buildslaves][puppet]
This is low priority until I have a patch for 593585 that works.
Priority: -- → P5
We're going to need this (at least on Windows) for Bug 563318.
Blocks: 563318
Priority: P5 → --
We're also about to upgrade the Buildbot and Twisted version on slaves, and make it consistent across the pool, so this could fit in nicely.

Comment 4

6 years ago
I guess we're not jumping straight to Python 2.7 because of bug 659881 / http://bugs.python.org/issue9516.
(Assignee)

Updated

6 years ago
tracking-firefox8: --- → ?
Priority: -- → P3

Comment 5

6 years ago
We don't need to track this for Firefox 8 as bug 563318 hasn't landed in that version
tracking-firefox8: ? → -

Updated

6 years ago
No longer blocks: 563318
(In reply to Jesse Ruderman from comment #4)
> I guess we're not jumping straight to Python 2.7 because of bug 659881 /
> http://bugs.python.org/issue9516.

Can we, now? MozillaBuild 1.6 (addressing some MSVC issues) has been released and has Python 2.7.2, Lion has 2.7 by default, as do some of the recent Linux distributions (e.g. Ubuntu 11.10).

And bug 659881 has been fixed.

Updated

6 years ago
Blocks: 724191
I did some probing into the build slaves and realized we're on a combination of 2.5, 2.5.1 and 2.5.2 for Windows (2003 Server?) / Mac Leopard / Linux.

Mac Snow Leopard is on 2.6.4.

It would be really nice to get us all on at least Python 2.6.x, or 2.7.x soon. Python 2.7.3 is coming out soon, too, 2.7.3 RC 1 was released on Feb 23, 2012.

This will also aid bug 636155 and bug 724191.
historically we've done this sort of thing by installing a Python version to /tools/python-x.y.z with puppet and then using that path explicitly either to run apps or to set up virtualenvs.
After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I think we're all agreed that 2.7 is the desired target here. That's what will really help us get where we want to go (bug 724191).
Summary: Deploy python 2.6 to all build machines → Deploy python 2.7 to all build machines

Updated

5 years ago
Blocks: 740147

Updated

5 years ago
No longer blocks: 740147
Depends on: 746723
This would be fixed for Linux with bug 670707.
According to #build, all our mozilla-central Windows builds are on Win64 machines, all of which have Python 2.6 by default. l10n repacks happen on Win32 machines, which have older python.

For pymake we need at least 2.6, which means that as long as we can use MSYS make to run l10n repacks we should be good on Windows.
No longer blocks: 593585
(Assignee)

Updated

5 years ago
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: P3 → P2
(Assignee)

Comment 12

5 years ago
Most recent version of python installed on each class of build machines:

bld-centos6-hp: 2.7.2 (/tools/python27/bin/python)
bld-lion-r5:    2.7.2 (/tools/python-2.7.2/bin/python)
w64-ix:         2.7.3 (/c/mozilla-build/python27/python.exe)

Notes:
* I've ignored the mw32-ix machines. We'll be off of them soon.
* I've also ignored the linux-ix/linux64-ix slaves because we'll be moving those builds to the hp builders/AWS (using mock) shortly.

Is that sufficiently consistent to start targeting python 2.7 on the builders? Should I work to get 2.7.3 (or greater) installed everywhere?
> Is that sufficiently consistent to start targeting python 2.7 on the
> builders? Should I work to get 2.7.3 (or greater) installed everywhere?

Synchronizing to 2.7.3 would be ideal, though I can't think of anything off the top of my head that would differ from 2.7.2 to 2.7.3 at the moment.

(There has been historical cases of minor point release divergence causing issues and wasted developer time, e.g. during 2.5.x.)
2.7.3 would be good, since it fixes at least one regression that might affect Pymake. http://bugs.python.org/issue11583
(In reply to Siddharth Agarwal [:sid0] from comment #14)
> 2.7.3 would be good, since it fixes at least one regression that might
> affect Pymake. http://bugs.python.org/issue11583

Sid, do you know of more Python bugs that might affect pymake that have landed in 2.7 dev branch but not made it into a release yet? (i.e. will be fixed in the future 2.7.4)

If not, 2.7.3 should be good to go as of this point in time.

coop, will the default python command be these 2.7.3 versions? Or does one have to specify the full directory to these binaries?

e.g.

Entering `python` gives default python which may not be 2.7

Entering `/tools/python-2.7.2/bin/python` gives 2.7
(Assignee)

Comment 16

5 years ago
(In reply to Gary Kwong [:gkw, :nth10sd] from comment #15)
> coop, will the default python command be these 2.7.3 versions? Or does one
> have to specify the full directory to these binaries?
> 
> e.g.
> 
> Entering `python` gives default python which may not be 2.7
> 
> Entering `/tools/python-2.7.2/bin/python` gives 2.7

Not yet, no. For now if you require 2.7, you'll need to call it directly until I can run tests on each platform to make sure changing the default python version doesn't break the build process in some weird way.

I will try to get some linux/mac packages built for 2.7.3 this week. I think we (releng) would prefer to keep them identical anyway.
> Not yet, no. For now if you require 2.7, you'll need to call it directly
> until I can run tests on each platform to make sure changing the default
> python version doesn't break the build process in some weird way.

Do they have an environment variable set (e.g. PYTHON27=<path>) so a script can access them across multiple platforms?

Also, is there a bug for changing the default python version?

Updated

5 years ago
Depends on: 788908
I don't think there's a bug on increasing the version of Python required to build Mozilla yet, no. Seems preemptive until we get this rolled out.

Updated

5 years ago
Blocks: 784841
(Assignee)

Updated

5 years ago
Whiteboard: [buildslaves][puppet] → [buildslaves][puppet][buildfaster:p2]
(Assignee)

Comment 19

5 years ago
I have a new 2.7.3 DMG ready for testing in staging, but the instructions for building new python packages for mock are somewhat involved. It may take a while there.
(Assignee)

Comment 20

5 years ago
Python DMG creation instructions:

curl -LO http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
tar jxf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --enable-framework=/tools/python-2.7.3 \
  --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk/ \
  --prefix=/tools/python-2.7.3 --with-universal-archs=intel
make -j8
make install DESTDIR=$PWD/installroot
sh create-dmg.sh installroot/. python-2.7.3 python27 /
see also http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/mozilla/python27-dmg.sh
(Assignee)

Comment 22

5 years ago
Created attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)
Attachment #660843 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 23

5 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #21)
> see also
> http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/
> mozilla/python27-dmg.sh

Hrmm, do given the contents of https://hg.mozilla.org/build/puppet/file/dee2f60946dd/modules/packages/manifests/mozilla, should I also be building new DMGs for py27_mercurial and py27_virtualenv?
I believe so, yes.
(Assignee)

Comment 25

5 years ago
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Canceling review until I get the other packages ready too.
Attachment #660843 - Attachment is obsolete: true
Attachment #660843 - Flags: review?(bugspam.Callek)
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Review of attachment 660843 [details] [diff] [review]:
-----------------------------------------------------------------

Note that the files dustin pointed at are "PuppetAgain" and for the 10.8 Mac machine, and Cent6 B2G builders, which are already using py2.7[.2]

I'm happy to update there as well if you like, since its simple, but I don't think we need to update the virtualenv/mercurial packages for 10.7/10.6/10.5 for this (I'm not sure if 10.7 has an hg/virtualenv tied to this version of python, doesn't look like it per: http://scl3-production-puppet.srv.releng.scl3.mozilla.com/production/darwin11-x86_64/build/DMGs/)
Attachment #660843 - Flags: review+
> I'm happy to update there as well if you like, since its simple

Yes, please synchronize to Python 2.7.3.
fwiw, bug 804865 is the bug that requires Python 2.7 to build the tree. We now require Python 2.6 per bug 800614.
Gregory mentions in bug 800614 comment 0 that releng builders all run Python 2.7. Coop, should this bug be updated?
Flags: needinfo?(coop)
(Assignee)

Comment 30

5 years ago
They're still not all running 2.7.3 - it's a mix of 2.7.2 and 2.7.3. I'd still like to standardize on a single version.
Flags: needinfo?(coop)
(Assignee)

Updated

5 years ago
Summary: Deploy python 2.7 to all build machines → Deploy python 2.7.3 to all build machines
(Assignee)

Updated

5 years ago
Attachment #660843 - Attachment is obsolete: false
(Assignee)

Comment 31

5 years ago
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

https://hg.mozilla.org/build/puppet-manifests/rev/b95fc6e3e5ef
Attachment #660843 - Flags: checked-in+
I'm confused at the disconnect between this and bug 820238..
(Assignee)

Comment 33

5 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> I'm confused at the disconnect between this and bug 820238..

Disconnect? Not sure I understand. 

Recall that lion builders aren't on PuppetAgain (yet), and we're deploying the same version in both places (2.7.3).
In this bug we're looking to have the same version on all slaves; in the other bug, that seems to be not a concern.
(Assignee)

Comment 35

5 years ago
Python 2.7.3 has replaced 2.7.2 on the Lion machines now.
Apparently, the windows builders on elm are not using python 2.7 at all, which a) prevents any merge from m-c b) broke my landing of the last version of bug 780561.
(In reply to Mike Hommey [:glandium] from comment #36)

The build logs have the path to the python27 install present (and early). The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the python 2.7 switch merged from m-c or a clobber might help.
(In reply to Nick Thomas [:nthomas] from comment #37)
> (In reply to Mike Hommey [:glandium] from comment #36)
> 
> The build logs have the path to the python27 install present (and early).
> The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the
> python 2.7 switch merged from m-c or a clobber might help.

I did clobber and retrigger, and that still used python2.6. AFAICT, the python 2.7 path is *not* in PATH, so it doesn't find python2.7, and finds python2.6. (the patch removing 2.6 support has not been merged yet)
(In reply to Mike Hommey [:glandium] from comment #38)
> (the patch removing 2.6 support has not been merged yet)

Which needs to be done.

gps discovered (after initial backout of py2.7 req) that we don't have python2.7 binary in the PATH, so it still finds python2.6 by default :(

So you do need to land the remove 2.6 patch and clobber.
Which builders is this about?  w64-ix-slave*?
(In reply to Justin Wood (:Callek) from comment #39)
> (In reply to Mike Hommey [:glandium] from comment #38)
> > (the patch removing 2.6 support has not been merged yet)
> 
> Which needs to be done.
> 
> gps discovered (after initial backout of py2.7 req) that we don't have
> python2.7 binary in the PATH, so it still finds python2.6 by default :(
> 
> So you do need to land the remove 2.6 patch and clobber.

It worked. Thanks. Any reason we're not changing the PATH?
(Assignee)

Updated

5 years ago
Depends on: 839560

Updated

5 years ago
No longer blocks: 784841
(Assignee)

Updated

4 years ago
Status: ASSIGNED → NEW
Priority: P2 → P3
I think this may have caused bug 858040.
I see from https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb we still have a few builders on Python 2.7.2. Now that MozillaBuild 1.7 is out and shipping with Python 2.7.4, I'd *really* like to bump our build requirements to 2.7.3 so we don't have to continue working around numerous bugs in older Python releases.

Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or 2.7.4)?

Updated

4 years ago
Blocks: 870420
(Assignee)

Comment 44

4 years ago
(In reply to Gregory Szorc [:gps] from comment #43) 
> Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or
> 2.7.4)?

Work is happening in bug 839560.
It looks like this was done in PuppetAgain, but for linux only.  So the Lion machines on puppetagain are using 2.7.2.  We should (a) pin the version of puppet on linux and (b) upgrade Darwin to 2.7.3.
(Assignee)

Comment 46

4 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #45)
> It looks like this was done in PuppetAgain, but for linux only.  So the Lion
> machines on puppetagain are using 2.7.2.  We should (a) pin the version of
> puppet on linux and (b) upgrade Darwin to 2.7.3.

Is there a new bug for the python 2.7.3 deploy on Mac, or are we going to track that here?
Between here and bug 882869 (two birds with one stone).
Component: Release Engineering → Release Engineering: Machine Management
QA Contact: armenzg
(Assignee)

Updated

4 years ago
Depends on: 882869
OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:

  10.7/python27-2.7.3-1.dmg
  10.8/python27-2.7.3-1.dmg

All that remains is changing the versions in the puppet manifests, and verifying on a test system.
(Assignee)

Comment 49

4 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #48)
> OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:
> 
>   10.7/python27-2.7.3-1.dmg
>   10.8/python27-2.7.3-1.dmg
> 
> All that remains is changing the versions in the puppet manifests, and
> verifying on a test system.

I've grabbed bld-lion-r5-043 if we want to try the 10.7 package there.

We can handle 10.8 (and other test platforms) in follow-up bugs.
How do you want to work that?  Probably the easiest way is to pin that host to your environment, and change the version number in that environment.

Updated

4 years ago
Blocks: 897067
We should get this going - we're about to introduce a bunch of other mac silos on puppetagain, and it'd be great to just bring them up with 2.7.3.
(Assignee)

Updated

4 years ago
No longer blocks: 897067
(Assignee)

Comment 52

4 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #50)
> How do you want to work that?  Probably the easiest way is to pin that host
> to your environment, and change the version number in that environment.

The slave is already attached to my buildbot dev master, or are you talking about puppet-specific changes here?
Puppet-specific changes.  You can "pin" a host to a particular puppet server and environment on that server (similar to allocating a slave to a master).  So the suggestion is to set up a puppet environment for yourself on a server of your choice, change the version number in that environment without checking in, and then pin a host or two to that environment.
(Assignee)

Comment 54

4 years ago
I have a build running in staging now to test this. The PATH already tells me that it's using 2.7.3, but I want to make sure we're not missing a module or something.
Status: NEW → ASSIGNED
Priority: P3 → P2
(Assignee)

Comment 55

4 years ago
(In reply to Chris Cooper [:coop] from comment #54)
> I have a build running in staging now to test this. The PATH already tells
> me that it's using 2.7.3, but I want to make sure we're not missing a module
> or something.

Build ran green. I'm happy here.
Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+ from me assuming the patch is just a version-number change.
(Assignee)

Comment 57

4 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #56)
> Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+
> from me assuming the patch is just a version-number change.

https://hg.mozilla.org/build/puppet/rev/401e2c4bfc39
(Assignee)

Comment 58

4 years ago
Deployed to the bld-lion-r5 builders.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Just be aware that ESR17 is still using Python 2.5 (though we'll be rid of it in a few months at least) - hence bugs like bug 914965.
And bug 915075.
Looks like some build machines are still running <2.7.

I hit failed builds while trying to do dict comprehensions on slaves running 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build slaves) 

maybe for for the logical reasons we had a while back: https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up upgrading these slaves.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 62

4 years ago
(In reply to Jordan Lund (:jlund) from comment #61)
> Looks like some build machines are still running <2.7.
> 
> I hit failed builds while trying to do dict comprehensions on slaves running
> 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build
> slaves) 
> 
> maybe for for the logical reasons we had a while back:
> https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up
> upgrading these slaves.

It's there, but it's not the default.

[cltbld@bld-linux64-ix-032.build.scl1.mozilla.com ~]$ which python2.7
/usr/local/bin/python2.7
[cltbld@bld-linux64-ix-032.build.scl1.mozilla.com ~]$ python2.7 --version
Python 2.7.3
(Assignee)

Comment 63

4 years ago
Jordan: are the dict comprehensions you're doing happening inside of mock, or outside as part of the factory? Can you target 2.7 directly? 

I believe the mock env itself relies on python 2.6 which is why we've opted to install 2.7 in parallel here rather than simply upgrading.
Flags: needinfo?(jlund)
this was patch for script.py (used everywhere in mozharness). I used the following syntax since I assume we want to stay compatible with <2.7:
    dict((key_expr, value_expr) for targets in iterable)

but to answer, this was outside of mock. Some slaves have 2.6 as default, as you mentioned above. This will be what mozharness uses when the script is called from bbot (I think buildbot uses `which python` and is not explicit)

I am not sure what dep mock_mozilla has but if it needs 2.6 then that makes sense. I do know that after mock is setup, if you supply 'python' in list of mock_packages, it will install 2.6.6. I think that is the default in whatever cache/dir the mock packages come from (maybe it's yum or some local file cache).

At any rate, it sounds like we need to have both. I could reach 2.7 if I need but to not mess with mozharness requirements, I won't use it.

closing since we do have 2.7 installed (but not as default)
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Flags: needinfo?(jlund)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.