Closed Bug 602908 Opened 14 years ago Closed 10 years ago

Deploy python 2.7.3 to all build machines

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
All
Type:
task

Tracking

(firefox8-)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox8 - ---

People

(Reporter: khuey, Assigned: coop)

References

Details

(Whiteboard: [buildslaves][puppet][buildfaster:p2])

Attachments

(1 file)

Install python 2.7.3 on Lion (puppet-manifests)
2.33 KB, patch
Callek
: review+
coop
: checked-in+
Details | Diff | Splinter Review 
If we decide to use the patch in Bug 593585 as is, we'll need python 2.6 to use client.mk.  We could scope that down to Windows only, in which case this reduces to upgrading the mozilla-build version, but I'd prefer to keep the configure logic the same across the board.
OS: Windows 7 → All
Whiteboard: [buildslaves][puppet]
This is low priority until I have a patch for 593585 that works.
Priority: -- → P5
We're going to need this (at least on Windows) for Bug 563318.
Blocks: msvc2010
Priority: P5 → --
We're also about to upgrade the Buildbot and Twisted version on slaves, and make it consistent across the pool, so this could fit in nicely.
I guess we're not jumping straight to Python 2.7 because of bug 659881 / http://bugs.python.org/issue9516.
tracking-firefox8: --- → ?
Priority: -- → P3
We don't need to track this for Firefox 8 as bug 563318 hasn't landed in that version
tracking-firefox8: ? → -
No longer blocks: msvc2010
(In reply to Jesse Ruderman from comment #4)
> I guess we're not jumping straight to Python 2.7 because of bug 659881 /
> http://bugs.python.org/issue9516.

Can we, now? MozillaBuild 1.6 (addressing some MSVC issues) has been released and has Python 2.7.2, Lion has 2.7 by default, as do some of the recent Linux distributions (e.g. Ubuntu 11.10).

And bug 659881 has been fixed.
Blocks: 724191
I did some probing into the build slaves and realized we're on a combination of 2.5, 2.5.1 and 2.5.2 for Windows (2003 Server?) / Mac Leopard / Linux.

Mac Snow Leopard is on 2.6.4.

It would be really nice to get us all on at least Python 2.6.x, or 2.7.x soon. Python 2.7.3 is coming out soon, too, 2.7.3 RC 1 was released on Feb 23, 2012.

This will also aid bug 636155 and bug 724191.
historically we've done this sort of thing by installing a Python version to /tools/python-x.y.z with puppet and then using that path explicitly either to run apps or to set up virtualenvs.
After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I think we're all agreed that 2.7 is the desired target here. That's what will really help us get where we want to go (bug 724191).
Summary: Deploy python 2.6 to all build machines → Deploy python 2.7 to all build machines
Blocks: 740147
No longer blocks: 740147
Depends on: 746723
This would be fixed for Linux with bug 670707.
According to #build, all our mozilla-central Windows builds are on Win64 machines, all of which have Python 2.6 by default. l10n repacks happen on Win32 machines, which have older python.

For pymake we need at least 2.6, which means that as long as we can use MSYS make to run l10n repacks we should be good on Windows.
No longer blocks: 593585
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: P3 → P2
Most recent version of python installed on each class of build machines:

bld-centos6-hp: 2.7.2 (/tools/python27/bin/python)
bld-lion-r5:    2.7.2 (/tools/python-2.7.2/bin/python)
w64-ix:         2.7.3 (/c/mozilla-build/python27/python.exe)

Notes:
* I've ignored the mw32-ix machines. We'll be off of them soon.
* I've also ignored the linux-ix/linux64-ix slaves because we'll be moving those builds to the hp builders/AWS (using mock) shortly.

Is that sufficiently consistent to start targeting python 2.7 on the builders? Should I work to get 2.7.3 (or greater) installed everywhere?
> Is that sufficiently consistent to start targeting python 2.7 on the
> builders? Should I work to get 2.7.3 (or greater) installed everywhere?

Synchronizing to 2.7.3 would be ideal, though I can't think of anything off the top of my head that would differ from 2.7.2 to 2.7.3 at the moment.

(There has been historical cases of minor point release divergence causing issues and wasted developer time, e.g. during 2.5.x.)
2.7.3 would be good, since it fixes at least one regression that might affect Pymake. http://bugs.python.org/issue11583
(In reply to Siddharth Agarwal [:sid0] from comment #14)
> 2.7.3 would be good, since it fixes at least one regression that might
> affect Pymake. http://bugs.python.org/issue11583

Sid, do you know of more Python bugs that might affect pymake that have landed in 2.7 dev branch but not made it into a release yet? (i.e. will be fixed in the future 2.7.4)

If not, 2.7.3 should be good to go as of this point in time.

coop, will the default python command be these 2.7.3 versions? Or does one have to specify the full directory to these binaries?

e.g.

Entering `python` gives default python which may not be 2.7

Entering `/tools/python-2.7.2/bin/python` gives 2.7
(In reply to Gary Kwong [:gkw, :nth10sd] from comment #15)
> coop, will the default python command be these 2.7.3 versions? Or does one
> have to specify the full directory to these binaries?
> 
> e.g.
> 
> Entering `python` gives default python which may not be 2.7
> 
> Entering `/tools/python-2.7.2/bin/python` gives 2.7

Not yet, no. For now if you require 2.7, you'll need to call it directly until I can run tests on each platform to make sure changing the default python version doesn't break the build process in some weird way.

I will try to get some linux/mac packages built for 2.7.3 this week. I think we (releng) would prefer to keep them identical anyway.
> Not yet, no. For now if you require 2.7, you'll need to call it directly
> until I can run tests on each platform to make sure changing the default
> python version doesn't break the build process in some weird way.

Do they have an environment variable set (e.g. PYTHON27=<path>) so a script can access them across multiple platforms?

Also, is there a bug for changing the default python version?
Depends on: 788908
I don't think there's a bug on increasing the version of Python required to build Mozilla yet, no. Seems preemptive until we get this rolled out.
Blocks: 784841
Whiteboard: [buildslaves][puppet] → [buildslaves][puppet][buildfaster:p2]
I have a new 2.7.3 DMG ready for testing in staging, but the instructions for building new python packages for mock are somewhat involved. It may take a while there.
Python DMG creation instructions:

curl -LO http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
tar jxf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --enable-framework=/tools/python-2.7.3 \
  --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk/ \
  --prefix=/tools/python-2.7.3 --with-universal-archs=intel
make -j8
make install DESTDIR=$PWD/installroot
sh create-dmg.sh installroot/. python-2.7.3 python27 /

        Attachment #660843 -
        Flags: review?(bugspam.Callek)

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #21)
> see also
> http://hg.mozilla.org/build/puppet/file/tip/modules/packages/manifests/
> mozilla/python27-dmg.sh

Hrmm, do given the contents of https://hg.mozilla.org/build/puppet/file/dee2f60946dd/modules/packages/manifests/mozilla, should I also be building new DMGs for py27_mercurial and py27_virtualenv?

    
    

    
  
I believe so, yes.

    
    

    
  
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Canceling review until I get the other packages ready too.

        Attachment #660843 -
        Attachment is obsolete: true

        Attachment #660843 -
        Flags: review?(bugspam.Callek)

    
    

    
  
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

Review of attachment 660843 [details] [diff] [review]:
-----------------------------------------------------------------

Note that the files dustin pointed at are "PuppetAgain" and for the 10.8 Mac machine, and Cent6 B2G builders, which are already using py2.7[.2]

I'm happy to update there as well if you like, since its simple, but I don't think we need to update the virtualenv/mercurial packages for 10.7/10.6/10.5 for this (I'm not sure if 10.7 has an hg/virtualenv tied to this version of python, doesn't look like it per: http://scl3-production-puppet.srv.releng.scl3.mozilla.com/production/darwin11-x86_64/build/DMGs/)

        Attachment #660843 -
        Flags: review+

    
    

    
  
> I'm happy to update there as well if you like, since its simple

Yes, please synchronize to Python 2.7.3.

    
    

    
  
fwiw, bug 804865 is the bug that requires Python 2.7 to build the tree. We now require Python 2.6 per bug 800614.

    
    

    
  
Gregory mentions in bug 800614 comment 0 that releng builders all run Python 2.7. Coop, should this bug be updated?
Flags: needinfo?(coop)

    
    

    
  
They're still not all running 2.7.3 - it's a mix of 2.7.2 and 2.7.3. I'd still like to standardize on a single version.
Flags: needinfo?(coop)

    
  
Summary: Deploy python 2.7 to all build machines → Deploy python 2.7.3 to all build machines

    
  

        Attachment #660843 -
        Attachment is obsolete: false

    
    

    
  
Comment on attachment 660843 [details] [diff] [review]
Install python 2.7.3 on Lion (puppet-manifests)

https://hg.mozilla.org/build/puppet-manifests/rev/b95fc6e3e5ef

        Attachment #660843 -
        Flags: checked-in+

    
    

    
  
I'm confused at the disconnect between this and bug 820238..

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> I'm confused at the disconnect between this and bug 820238..

Disconnect? Not sure I understand. 

Recall that lion builders aren't on PuppetAgain (yet), and we're deploying the same version in both places (2.7.3).

    
    

    
  
In this bug we're looking to have the same version on all slaves; in the other bug, that seems to be not a concern.

    
    

    
  
Python 2.7.3 has replaced 2.7.2 on the Lion machines now.

    
    

    
  
Apparently, the windows builders on elm are not using python 2.7 at all, which a) prevents any merge from m-c b) broke my landing of the last version of bug 780561.

    
    

    
  
(In reply to Mike Hommey [:glandium] from comment #36)

The build logs have the path to the python27 install present (and early). The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the python 2.7 switch merged from m-c or a clobber might help.

    
    

    
  
(In reply to Nick Thomas [:nthomas] from comment #37)
> (In reply to Mike Hommey [:glandium] from comment #36)
> 
> The build logs have the path to the python27 install present (and early).
> The virtualenv in the build dir is 2.6.5 though, so perhaps not all of the
> python 2.7 switch merged from m-c or a clobber might help.

I did clobber and retrigger, and that still used python2.6. AFAICT, the python 2.7 path is *not* in PATH, so it doesn't find python2.7, and finds python2.6. (the patch removing 2.6 support has not been merged yet)

    
    

    
  
(In reply to Mike Hommey [:glandium] from comment #38)
> (the patch removing 2.6 support has not been merged yet)

Which needs to be done.

gps discovered (after initial backout of py2.7 req) that we don't have python2.7 binary in the PATH, so it still finds python2.6 by default :(

So you do need to land the remove 2.6 patch and clobber.

    
    

    
  
Which builders is this about?  w64-ix-slave*?

    
    

    
  
(In reply to Justin Wood (:Callek) from comment #39)
> (In reply to Mike Hommey [:glandium] from comment #38)
> > (the patch removing 2.6 support has not been merged yet)
> 
> Which needs to be done.
> 
> gps discovered (after initial backout of py2.7 req) that we don't have
> python2.7 binary in the PATH, so it still finds python2.6 by default :(
> 
> So you do need to land the remove 2.6 patch and clobber.

It worked. Thanks. Any reason we're not changing the PATH?

    
  
Depends on: 839560

    
  
No longer blocks: 784841

    
  
Status: ASSIGNED → NEW
Priority: P2 → P3

    
    

    
  
I think this may have caused bug 858040.

    
    

    
  
I see from https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb we still have a few builders on Python 2.7.2. Now that MozillaBuild 1.7 is out and shipping with Python 2.7.4, I'd *really* like to bump our build requirements to 2.7.3 so we don't have to continue working around numerous bugs in older Python releases.

Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or 2.7.4)?

    
  
Blocks: 870420

    
    

    
  
(In reply to Gregory Szorc [:gps] from comment #43) 
> Can someone provide an ETA for upgrading the remaining builders to 2.7.3 (or
> 2.7.4)?

Work is happening in bug 839560.

    
    

    
  
It looks like this was done in PuppetAgain, but for linux only.  So the Lion machines on puppetagain are using 2.7.2.  We should (a) pin the version of puppet on linux and (b) upgrade Darwin to 2.7.3.

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #45)
> It looks like this was done in PuppetAgain, but for linux only.  So the Lion
> machines on puppetagain are using 2.7.2.  We should (a) pin the version of
> puppet on linux and (b) upgrade Darwin to 2.7.3.

Is there a new bug for the python 2.7.3 deploy on Mac, or are we going to track that here?

    
    

    
  
Between here and bug 882869 (two birds with one stone).
Component: Release Engineering → Release Engineering: Machine Management
QA Contact: armenzg

    
  
Depends on: 882869

    
    

    
  
OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:

  10.7/python27-2.7.3-1.dmg
  10.8/python27-2.7.3-1.dmg

All that remains is changing the versions in the puppet manifests, and verifying on a test system.

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #48)
> OK, bug 882869 is closed.  Python-2.7.3 is ready to deploy on Darwin, then:
> 
>   10.7/python27-2.7.3-1.dmg
>   10.8/python27-2.7.3-1.dmg
> 
> All that remains is changing the versions in the puppet manifests, and
> verifying on a test system.

I've grabbed bld-lion-r5-043 if we want to try the 10.7 package there.

We can handle 10.8 (and other test platforms) in follow-up bugs.

    
    

    
  
How do you want to work that?  Probably the easiest way is to pin that host to your environment, and change the version number in that environment.

    
  
Blocks: 897067

    
    

    
  
We should get this going - we're about to introduce a bunch of other mac silos on puppetagain, and it'd be great to just bring them up with 2.7.3.

    
  
No longer blocks: 897067

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #50)
> How do you want to work that?  Probably the easiest way is to pin that host
> to your environment, and change the version number in that environment.

The slave is already attached to my buildbot dev master, or are you talking about puppet-specific changes here?

    
    

    
  
Puppet-specific changes.  You can "pin" a host to a particular puppet server and environment on that server (similar to allocating a slave to a master).  So the suggestion is to set up a puppet environment for yourself on a server of your choice, change the version number in that environment without checking in, and then pin a host or two to that environment.

    
    

    
  
I have a build running in staging now to test this. The PATH already tells me that it's using 2.7.3, but I want to make sure we're not missing a module or something.
Status: NEW → ASSIGNED
Priority: P3 → P2

    
    

    
  
(In reply to Chris Cooper [:coop] from comment #54)
> I have a build running in staging now to test this. The PATH already tells
> me that it's using 2.7.3, but I want to make sure we're not missing a module
> or something.

Build ran green. I'm happy here.

    
    

    
  
Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+ from me assuming the patch is just a version-number change.

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #56)
> Cool!  Do you want to commit the change, then, or should I?  Pre-emptive r+
> from me assuming the patch is just a version-number change.

https://hg.mozilla.org/build/puppet/rev/401e2c4bfc39

    
    

    
  
Deployed to the bld-lion-r5 builders.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering

    
    

    
  
Just be aware that ESR17 is still using Python 2.5 (though we'll be rid of it in a few months at least) - hence bugs like bug 914965.

    
    

    
  
Looks like some build machines are still running <2.7.

I hit failed builds while trying to do dict comprehensions on slaves running 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build slaves) 

maybe for for the logical reasons we had a while back: https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up upgrading these slaves.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
(In reply to Jordan Lund (:jlund) from comment #61)
> Looks like some build machines are still running <2.7.
> 
> I hit failed builds while trying to do dict comprehensions on slaves running
> 2.6.6: bld-linux64-ix-032 and bld-centos6-hp-012 (both inhouse linux build
> slaves) 
> 
> maybe for for the logical reasons we had a while back:
> https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c12, we never ended up
> upgrading these slaves.

It's there, but it's not the default.

[cltbld@bld-linux64-ix-032.build.scl1.mozilla.com ~]$ which python2.7
/usr/local/bin/python2.7
[cltbld@bld-linux64-ix-032.build.scl1.mozilla.com ~]$ python2.7 --version
Python 2.7.3

    
    

    
  
Jordan: are the dict comprehensions you're doing happening inside of mock, or outside as part of the factory? Can you target 2.7 directly? 

I believe the mock env itself relies on python 2.6 which is why we've opted to install 2.7 in parallel here rather than simply upgrading.
Flags: needinfo?(jlund)

    
    

    
  
this was patch for script.py (used everywhere in mozharness). I used the following syntax since I assume we want to stay compatible with <2.7:
    dict((key_expr, value_expr) for targets in iterable)

but to answer, this was outside of mock. Some slaves have 2.6 as default, as you mentioned above. This will be what mozharness uses when the script is called from bbot (I think buildbot uses `which python` and is not explicit)

I am not sure what dep mock_mozilla has but if it needs 2.6 then that makes sense. I do know that after mock is setup, if you supply 'python' in list of mock_packages, it will install 2.6.6. I think that is the default in whatever cache/dir the mock packages come from (maybe it's yum or some local file cache).

At any rate, it sounds like we need to have both. I could reach 2.7 if I need but to not mess with mozharness requirements, I won't use it.

closing since we do have 2.7 installed (but not as default)
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Flags: needinfo?(jlund)
Resolution: --- → FIXED

    
  
Product: Release Engineering → Infrastructure & Operations

    
  
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: