839560 - Create and deploy python 2.7.3 packages for linux/linux64 builders

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

We're trying to standardize on python 2.7.3 across the buildfarm (bug 602908), so we need to create new python packages for mock for both 32- and 64-bit variants of Linux. 

I'm not sure what other packages rely on the current python 2.7.2 packages (mercurial?, venv?) but those will also need to be updated.

Comment 1

[Chris Cooper [:coop] (he/him)]

For reference, here are the instructions I received from jhford when I first asked about this back in the Fall:

You'll need to be on a machine that has Fedora or CentOS and the upstream mock.  Mock_mozilla would work but it doesn't have the configuration files you'll want to use ("yum install mock").  I have a rudimentary build system for these rpms which you can use if you like, but it might only make sense to me.  The command you'd use for that is "./actions.sh mock_build fedora-16-i386 i686 python27.spec", but I'll describe the generic build process below.

1) change into centos6/python27 and obtain your sources locally.  Small things like patches are in the repository, but large tarballs and new sources need to be downloaded.  A common and good practice is to use the url to a source tarball as the Source: tag in the spec file
2) modify your specfile as needed.  For the python 2.7 one [1], it should be a matter of changing the pyrel macro to 3
3) Pick your mock target and architecture.  RHEL derivatives (e.g. centos) use the name epel with the rhel matching version, centos 6 is epel-6 [2].  The list of configurations that you can use can be determined by running 

    ls /etc/mock/*.cfg | sed -e "s,/etc/mock/\(.*\)\.cfg"

4) create an SRPM file using mock.  This step is needed because if you build your SRPM using rpmbuild on your host, you might end up with an SRPM that the version of rpmbuild in your mock target cannot understand.  You can build the srpm with:

    mock -r $target --buildsrpm --spec python27.spec --sources . --resultdir .

5) build the binary version of the package using mock.  The log from the actual build will be in a file called "build.log".  This is almost the same as building the srpm:

    mock -r fedora-16-i386 mozilla-python27-2.7.2-3.fc16.src.rpm --resultdir .

6) test the rpms.  Testing host packages should just be rpm -Uvh <rpmfile> on that type of host.  To test in a mock chroot, you'll have to create a local repository and add that repository to the mock_mozilla.cfg file you're using on your local test machine.
7) file a relops bug to get your rpm included in the appropriate Yum mirror when you are satisfied that it works.

Once that's done, you have new rpms.  If you want to upgrade the packages that depend on Python 2.7, you can.  The packaging was done in such a way that the dependent packages only care that you are using python 2.7, not the release (.2 or .3).  If you want to rebuild for those, you need to have the updated python 2.7 package in the mock root you're using to build.  That requires creating a local yum repository and adding it to the mock configuration you use.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #1)
> For reference, here are the instructions I received from jhford when I first
> asked about this back in the Fall:

FWIW, there may be better docs/process out there for this now, I'm just relaying the information I have in case it's helpful.

Comment 3

[John Hopkins (:jhopkins)]

$ repoquery --whatrequires mozilla-python27-2.7.2-5.el6
mozilla-python27-0:2.7.2-5.el6.x86_64
mozilla-python27-MySQL-python-0:1.2.3-1.el6.x86_64
mozilla-python27-mercurial-0:2.1.1-3.el6.x86_64
mozilla-python27-mercurial-0:2.1.1-4.el6.x86_64
mozilla-python27-python-setuptools-0:0.6.10-1.el6.noarch
mozilla-python27-twisted-0:12.0.0-1.el6.x86_64
mozilla-python27-virtualenv-0:1.7.1.2-1.el6.x86_64
mozilla-python27-virtualenv-0:1.7.1.2-2.el6.x86_64
mozilla-python27-zope-interface-0:3.3.0-1.el6.x86_64

Comment 4

[John Hopkins (:jhopkins)]

Packages built using 'mach', uploaded to http://people.mozilla.org/~jhopkins/bug839560/

I've started testing.

Comment 5

[Justin Wood (:Callek)]

As far as I know, when you update the python/packages in puppet you'll need to regen the buildbot venv's.

We don't have login in puppetAgain for that yet, but it *shouldn't* be too hard. You'll also need to regen the puppetAgain based buildbotmasters venv's.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

If you're using a different tool, please make sure that the replication procedure is the same as for mock.  In particular, I should be able to pull the updated .specs out of http://hg.mozilla.org/build/puppet, run 'em through mock to build an SRPM and an RPM, and get equivalent output.

If mach helps with the chained dependencies, all the better -- please document on the PuppetAgain wiki pages.

Comment 7

[Justin Wood (:Callek)]

Discussed briefly in IRC and jhopkins agrees we should probably just update to 2.7.4 (just released) since this hasn't been rolled out yet.

Useful notes about that -- 

Full change history: http://hg.python.org/cpython/file/9290822f2280/Misc/NEWS
- Assorted useful changes arbitrarily called out: 
* Issue #7358: cStringIO.StringIO now supports writing to and reading from a stream larger than 2 GiB on 64-bit systems.
* Issue #17051: Fix a memory leak in os.path.isdir() on Windows.
* Issue #16176: Properly identify Windows 8 via platform.platform()
* Issue #14157: Fix time.strptime failing without a year on February 29th.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #7)
> Discussed briefly in IRC and jhopkins agrees we should probably just update
> to 2.7.4 (just released) since this hasn't been rolled out yet.
> 
> Useful notes about that -- 
> 
> Full change history: http://hg.python.org/cpython/file/9290822f2280/Misc/NEWS
> - Assorted useful changes arbitrarily called out: 
> * Issue #7358: cStringIO.StringIO now supports writing to and reading from a
> stream larger than 2 GiB on 64-bit systems.
> * Issue #17051: Fix a memory leak in os.path.isdir() on Windows.
> * Issue #16176: Properly identify Windows 8 via platform.platform()
> * Issue #14157: Fix time.strptime failing without a year on February 29th.

Are you signing up to update python on Mac and Windows too? 

I *really* don't want to end up in the situation we're in right now with hg. Having a single version allows developers to either target it for features or just ignore it completely because it's the same everywhere.

Comment 9

[John Hopkins (:jhopkins)]

Attachment: diff of original vs. rebuilt mozilla-python27 RPM

Any differences here to be concerned about?

Comment 10

[John Hopkins (:jhopkins)]

(In reply to Chris Cooper [:coop] from comment #8)
> Are you signing up to update python on Mac and Windows too?

I, personally, am not.

> I *really* don't want to end up in the situation we're in right now with hg.
> Having a single version allows developers to either target it for features
> or just ignore it completely because it's the same everywhere.

Why can't developers target the lowest common denominator?

Comment 11

[Justin Wood (:Callek)]

(In reply to John Hopkins (:jhopkins) from comment #10)
> (In reply to Chris Cooper [:coop] from comment #8)
> > Are you signing up to update python on Mac and Windows too?
> 
> I, personally, am not.

Ooo, in my brief IRC chat with jhopkins I thought mac and windows py update syncs were "still to do"

If those are all already py2.7.3, I'd rather NOT go through the effort of upgrading them, and would rather leave our sync for linux to be 2.7.3. Even though I want .4 its not worth an effort of redoing all our sync efforts-up-until-now.

(hope that clears this up)

Comment 12

[John Hopkins (:jhopkins)]

> If those are all already py2.7.3, I'd rather NOT go through the effort of upgrading them

Works for me.  Reverting the bug description.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 734607 [details] [diff] [review]
diff of original vs. rebuilt mozilla-python27 RPM

It's hard to tell from that output, although it looks pretty innocuous.

What is the difference in the .spec?

Comment 14

[John Hopkins (:jhopkins)]

> What is the difference in the .spec?

The spec I used is in this .src.rpm: 

 http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/mozilla-python27-2.7.2-5.el6.src.rpm

I used it as-is to rebuild the mozilla-python27 RPM for comparison purposes.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, I see.  The diff looks good then.

Comment 16

[John Hopkins (:jhopkins)]

Unfortunately, the additional "libz.so.1(ZLIB_1.2.0)(64bit)" dependency mentioned in the diff I attached prevents the RPM from installing (we use zlib 1.2.3).  The dependency is required by a rebuilt tools/python27/lib/python2.7/lib-dynload/zlib.so (as reported by |readelf -a|) but the original zlib.so simply depends on libz.so.1.

I rebuilt the src.rpm under mock on bld-centos6-hp-003 to confirm whether this was a mach versus mach issue, but the mock-built RPM still contained the ZLIB_1.2.0 dependency.

The original RPM was apparently built in an environment that did not add that dependency.  

After some more digging, I found that disabling RPM's internal dependency generator and using the (unfortunately, deprecated) find-requires script does not yield the ZLIB_1.2.0 dependency which masks the problem.  Testing zlib's compress and decompress functions in a python shell does work, so we could potentially use this as a solution, but I would feel more comfortable if we could reproduce the original RPM (and its package dependency list) and use the same environment for rebuilding 2.7.3.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, I thought environment isolation was what mock was all about.  Are you using mock against the PuppetAgain repos?

Comment 18

[John Hopkins (:jhopkins)]

I was not using mock against the PuppetAgain repos.  I've just now tried that and the package build fails on "C compiler cannot create executables".  Removing the gcc45_0moz3-4.5.2-0moz3_mock_centos6 package and retrying results in a successful build which still has the ZLIB_1.2.0 package requirement.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Are you sure that the zlib SO version and the zlib version are the same?  I assume if it's finding ZLIB_1.2.0, that it's getting it from a zlib package installed in the mock environment.  Does zlib-1.2.3 advertise an SO_VERSION of 1.2.0?

Comment 20

[John Hopkins (:jhopkins)]

I've discovered that there are two different repos at work here which are giving us two different builds of zlib, each with a different set of 'provides'.

1) The puppetagain zlib (http://repos/repos/yum/mirrors/centos/6/latest/os/x86_64//zlib-1.2.3-27.el6.x86_64.rpm) provides these dependencies:
libz.so.1()(64bit)  
zlib = 1.2.3-27.el6
zlib(x86-64) = 1.2.3-27.el6

2) The upstream zlib (in this case http://mirrors.rit.edu/centos/6.4/os/x86_64//Packages/zlib-1.2.3-29.el6.i686.rpm) provides these dependencies:
libz.so.1  
libz.so.1(ZLIB_1.2.0)  
libz.so.1(ZLIB_1.2.0.2)  
libz.so.1(ZLIB_1.2.0.8)  
libz.so.1(ZLIB_1.2.2)  
libz.so.1(ZLIB_1.2.2.3)  
libz.so.1(ZLIB_1.2.2.4)  
zlib = 1.2.3-29.el6
zlib(x86-32) = 1.2.3-29.el6

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

There shouldn't be any "upstream" repositories involved here..

Comment 22

[John Hopkins (:jhopkins)]

Ok, that's a good assertion to work from.  I created new mock configs for package building which only include Mozilla's RPM repos.  I ran into a couple of problems (like the package group buildsys-group not existing) but found solutions.

I've now produced a clean build of mozilla-python27-2.7.3-0.el6.x86_64.rpm built on bld-centos6-hp-003 using mock with the correct zlib package dependency.  Will build i386 and do some testing tomorrow.

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Can you update the wiki docs regarding how to do package builds?

If you think it's worth it, we could pretty easily build a standalone RPM-generation VM (just file another bug).

Comment 24

[John Hopkins (:jhopkins)]

Yes, I can document my steps and update those steps once we have a standard RPM build environment.

I've filed bug 860260 to track creating a standalone RPM generation VM.

Comment 25

[John Hopkins (:jhopkins)]

Documentation updated, testing looks good.  I'm aiming to roll out the update in the next hour.

Comment 26

[John Hopkins (:jhopkins)]

Attachment: [puppet] update python version

Comment 27

[John Hopkins (:jhopkins)]

Comment on attachment 736789 [details] [diff] [review]
[puppet] update python version

Landed in http://hg.mozilla.org/build/puppet/rev/6ecb7490965e

The mozilla-python 2.7.3 RPMs have been pushed to YUM and are visible here:

 http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/
 http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/i386/

Comment 28

[John Hopkins (:jhopkins)]

Builds with the updated Python are looking good.

Comment 29

[Chris Cooper [:coop] (he/him)]

Looks like the b2g gecko configs are still missing this new package: https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb

Comment 30

[John Hopkins (:jhopkins)]

We'll have to add "notify" events for the mozilla-f16*.cfg files so the yum cache is cleared when they change (same as for the mozilla-centos6-*.cfg files):

 http://hg.mozilla.org/build/puppet/file/f4ff10a3b2bd/modules/mockbuild/manifests/init.pp

Comment 31

[John Hopkins (:jhopkins)]

Attachment: [puppet] expire mozilla-f16-*.cfg mock caches

needs testing

Comment 32

[John Hopkins (:jhopkins)]

Another, more substantial problem is the build referenced here is not using the mozilla-python27 package (which I had rebuilt to 2.7.3 for CentOS) but is using the Fedora python 2.7.2 which is fetched from our static yum repo.

Some options:
A)
- build the mozilla-python27-2.7.3 package on Fedora
- modify the b2g build configs to use the mozilla-python27 package on Fedora
B)
- refresh our Fedora package mirror snapshot so it includes python-2.7.3.
C)
- copy the Fedora python-2.7.3 package (+ updated dependencies) to the releng Fedora repo (http://repos/repos/yum/releng/public/Fedora/16/) to avoid updating the mirror in B.

Comment 33

[Chris AtLee [:catlee]]

D) fix bug 816793 so these use the centos environment, and then we don't have this problem with fedora

Comment 34

[Chris Cooper [:coop] (he/him)]

D sounds like the correct solution, but also sounds like the most work. 

What is the quickest path forward here out of the other 3 options, or do we want to invest the time in D?

Comment 35

[John Hopkins (:jhopkins)]

Option C should be the quickest solution of the four.  Catlee, can we start with C and plan to do option D at a later time?

Comment 36

[Chris AtLee [:catlee]]

I think this now fixed by bug 816793
https://tbpl.mozilla.org/php/getParsedLog.php?id=23828459&tree=Mozilla-Inbound&full=1

checking for python2.7... /tools/python27/bin/python2.7

Comment 37

[Chris AtLee [:catlee]]

Comment on attachment 747589 [details] [diff] [review]
[puppet] expire mozilla-f16-*.cfg mock caches

Review of attachment 747589 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think we need this any more. In fact, do we need the mozilla-f16-* mock environments any more now?