Create and deploy python 2.7.3 packages for linux/linux64 builders

RESOLVED FIXED

Status

Infrastructure & Operations
CIDuty
RESOLVED FIXED
6 years ago
2 months ago

People

(Reporter: coop, Assigned: jhopkins)

Tracking

Details

(Whiteboard: [pymake][standardize])

Attachments

(3 attachments)

(Reporter)

Description

6 years ago
We're trying to standardize on python 2.7.3 across the buildfarm (bug 602908), so we need to create new python packages for mock for both 32- and 64-bit variants of Linux. 

I'm not sure what other packages rely on the current python 2.7.2 packages (mercurial?, venv?) but those will also need to be updated.
(Reporter)

Comment 1

6 years ago
For reference, here are the instructions I received from jhford when I first asked about this back in the Fall:

You'll need to be on a machine that has Fedora or CentOS and the upstream mock.  Mock_mozilla would work but it doesn't have the configuration files you'll want to use ("yum install mock").  I have a rudimentary build system for these rpms which you can use if you like, but it might only make sense to me.  The command you'd use for that is "./actions.sh mock_build fedora-16-i386 i686 python27.spec", but I'll describe the generic build process below.

1) change into centos6/python27 and obtain your sources locally.  Small things like patches are in the repository, but large tarballs and new sources need to be downloaded.  A common and good practice is to use the url to a source tarball as the Source: tag in the spec file
2) modify your specfile as needed.  For the python 2.7 one [1], it should be a matter of changing the pyrel macro to 3
3) Pick your mock target and architecture.  RHEL derivatives (e.g. centos) use the name epel with the rhel matching version, centos 6 is epel-6 [2].  The list of configurations that you can use can be determined by running 

    ls /etc/mock/*.cfg | sed -e "s,/etc/mock/\(.*\)\.cfg"

4) create an SRPM file using mock.  This step is needed because if you build your SRPM using rpmbuild on your host, you might end up with an SRPM that the version of rpmbuild in your mock target cannot understand.  You can build the srpm with:

    mock -r $target --buildsrpm --spec python27.spec --sources . --resultdir .

5) build the binary version of the package using mock.  The log from the actual build will be in a file called "build.log".  This is almost the same as building the srpm:

    mock -r fedora-16-i386 mozilla-python27-2.7.2-3.fc16.src.rpm --resultdir .

6) test the rpms.  Testing host packages should just be rpm -Uvh <rpmfile> on that type of host.  To test in a mock chroot, you'll have to create a local repository and add that repository to the mock_mozilla.cfg file you're using on your local test machine.
7) file a relops bug to get your rpm included in the appropriate Yum mirror when you are satisfied that it works.

Once that's done, you have new rpms.  If you want to upgrade the packages that depend on Python 2.7, you can.  The packaging was done in such a way that the dependent packages only care that you are using python 2.7, not the release (.2 or .3).  If you want to rebuild for those, you need to have the updated python 2.7 package in the mock root you're using to build.  That requires creating a local yum repository and adding it to the mock configuration you use.
(Reporter)

Comment 2

6 years ago
(In reply to Chris Cooper [:coop] from comment #1)
> For reference, here are the instructions I received from jhford when I first
> asked about this back in the Fall:

FWIW, there may be better docs/process out there for this now, I'm just relaying the information I have in case it's helpful.

Updated

6 years ago
Assignee: nobody → jhopkins
(Assignee)

Comment 3

6 years ago
$ repoquery --whatrequires mozilla-python27-2.7.2-5.el6
mozilla-python27-0:2.7.2-5.el6.x86_64
mozilla-python27-MySQL-python-0:1.2.3-1.el6.x86_64
mozilla-python27-mercurial-0:2.1.1-3.el6.x86_64
mozilla-python27-mercurial-0:2.1.1-4.el6.x86_64
mozilla-python27-python-setuptools-0:0.6.10-1.el6.noarch
mozilla-python27-twisted-0:12.0.0-1.el6.x86_64
mozilla-python27-virtualenv-0:1.7.1.2-1.el6.x86_64
mozilla-python27-virtualenv-0:1.7.1.2-2.el6.x86_64
mozilla-python27-zope-interface-0:3.3.0-1.el6.x86_64
(Assignee)

Comment 4

5 years ago
Packages built using 'mach', uploaded to http://people.mozilla.org/~jhopkins/bug839560/

I've started testing.
As far as I know, when you update the python/packages in puppet you'll need to regen the buildbot venv's.

We don't have login in puppetAgain for that yet, but it *shouldn't* be too hard. You'll also need to regen the puppetAgain based buildbotmasters venv's.
If you're using a different tool, please make sure that the replication procedure is the same as for mock.  In particular, I should be able to pull the updated .specs out of http://hg.mozilla.org/build/puppet, run 'em through mock to build an SRPM and an RPM, and get equivalent output.

If mach helps with the chained dependencies, all the better -- please document on the PuppetAgain wiki pages.
Discussed briefly in IRC and jhopkins agrees we should probably just update to 2.7.4 (just released) since this hasn't been rolled out yet.

Useful notes about that -- 

Full change history: http://hg.python.org/cpython/file/9290822f2280/Misc/NEWS
- Assorted useful changes arbitrarily called out: 
* Issue #7358: cStringIO.StringIO now supports writing to and reading from a stream larger than 2 GiB on 64-bit systems.
* Issue #17051: Fix a memory leak in os.path.isdir() on Windows.
* Issue #16176: Properly identify Windows 8 via platform.platform()
* Issue #14157: Fix time.strptime failing without a year on February 29th.
Summary: Create and deploy python 2.7.3 packages for linux/linux64 builders → Create and deploy python 2.7.4 packages for linux/linux64 builders
(Reporter)

Comment 8

5 years ago
(In reply to Justin Wood (:Callek) from comment #7)
> Discussed briefly in IRC and jhopkins agrees we should probably just update
> to 2.7.4 (just released) since this hasn't been rolled out yet.
> 
> Useful notes about that -- 
> 
> Full change history: http://hg.python.org/cpython/file/9290822f2280/Misc/NEWS
> - Assorted useful changes arbitrarily called out: 
> * Issue #7358: cStringIO.StringIO now supports writing to and reading from a
> stream larger than 2 GiB on 64-bit systems.
> * Issue #17051: Fix a memory leak in os.path.isdir() on Windows.
> * Issue #16176: Properly identify Windows 8 via platform.platform()
> * Issue #14157: Fix time.strptime failing without a year on February 29th.

Are you signing up to update python on Mac and Windows too? 

I *really* don't want to end up in the situation we're in right now with hg. Having a single version allows developers to either target it for features or just ignore it completely because it's the same everywhere.
(Assignee)

Comment 9

5 years ago
Created attachment 734607 [details] [diff] [review]
diff of original vs. rebuilt mozilla-python27 RPM

Any differences here to be concerned about?
Flags: needinfo?(dustin)
Attachment #734607 - Flags: feedback?(dustin)
(Assignee)

Comment 10

5 years ago
(In reply to Chris Cooper [:coop] from comment #8)
> Are you signing up to update python on Mac and Windows too?

I, personally, am not.

> I *really* don't want to end up in the situation we're in right now with hg.
> Having a single version allows developers to either target it for features
> or just ignore it completely because it's the same everywhere.

Why can't developers target the lowest common denominator?
(In reply to John Hopkins (:jhopkins) from comment #10)
> (In reply to Chris Cooper [:coop] from comment #8)
> > Are you signing up to update python on Mac and Windows too?
> 
> I, personally, am not.

Ooo, in my brief IRC chat with jhopkins I thought mac and windows py update syncs were "still to do"

If those are all already py2.7.3, I'd rather NOT go through the effort of upgrading them, and would rather leave our sync for linux to be 2.7.3. Even though I want .4 its not worth an effort of redoing all our sync efforts-up-until-now.

(hope that clears this up)
(Assignee)

Comment 12

5 years ago
> If those are all already py2.7.3, I'd rather NOT go through the effort of upgrading them

Works for me.  Reverting the bug description.
Summary: Create and deploy python 2.7.4 packages for linux/linux64 builders → Create and deploy python 2.7.3 packages for linux/linux64 builders
Comment on attachment 734607 [details] [diff] [review]
diff of original vs. rebuilt mozilla-python27 RPM

It's hard to tell from that output, although it looks pretty innocuous.

What is the difference in the .spec?
Flags: needinfo?(dustin)
Attachment #734607 - Flags: feedback?(dustin) → feedback+
(Assignee)

Comment 14

5 years ago
> What is the difference in the .spec?

The spec I used is in this .src.rpm: 

 http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/mozilla-python27-2.7.2-5.el6.src.rpm

I used it as-is to rebuild the mozilla-python27 RPM for comparison purposes.
Ah, I see.  The diff looks good then.
(Assignee)

Comment 16

5 years ago
Unfortunately, the additional "libz.so.1(ZLIB_1.2.0)(64bit)" dependency mentioned in the diff I attached prevents the RPM from installing (we use zlib 1.2.3).  The dependency is required by a rebuilt tools/python27/lib/python2.7/lib-dynload/zlib.so (as reported by |readelf -a|) but the original zlib.so simply depends on libz.so.1.

I rebuilt the src.rpm under mock on bld-centos6-hp-003 to confirm whether this was a mach versus mach issue, but the mock-built RPM still contained the ZLIB_1.2.0 dependency.

The original RPM was apparently built in an environment that did not add that dependency.  

After some more digging, I found that disabling RPM's internal dependency generator and using the (unfortunately, deprecated) find-requires script does not yield the ZLIB_1.2.0 dependency which masks the problem.  Testing zlib's compress and decompress functions in a python shell does work, so we could potentially use this as a solution, but I would feel more comfortable if we could reproduce the original RPM (and its package dependency list) and use the same environment for rebuilding 2.7.3.
Hm, I thought environment isolation was what mock was all about.  Are you using mock against the PuppetAgain repos?
(Assignee)

Comment 18

5 years ago
I was not using mock against the PuppetAgain repos.  I've just now tried that and the package build fails on "C compiler cannot create executables".  Removing the gcc45_0moz3-4.5.2-0moz3_mock_centos6 package and retrying results in a successful build which still has the ZLIB_1.2.0 package requirement.
Are you sure that the zlib SO version and the zlib version are the same?  I assume if it's finding ZLIB_1.2.0, that it's getting it from a zlib package installed in the mock environment.  Does zlib-1.2.3 advertise an SO_VERSION of 1.2.0?
(Assignee)

Comment 20

5 years ago
I've discovered that there are two different repos at work here which are giving us two different builds of zlib, each with a different set of 'provides'.

1) The puppetagain zlib (http://repos/repos/yum/mirrors/centos/6/latest/os/x86_64//zlib-1.2.3-27.el6.x86_64.rpm) provides these dependencies:
libz.so.1()(64bit)  
zlib = 1.2.3-27.el6
zlib(x86-64) = 1.2.3-27.el6

2) The upstream zlib (in this case http://mirrors.rit.edu/centos/6.4/os/x86_64//Packages/zlib-1.2.3-29.el6.i686.rpm) provides these dependencies:
libz.so.1  
libz.so.1(ZLIB_1.2.0)  
libz.so.1(ZLIB_1.2.0.2)  
libz.so.1(ZLIB_1.2.0.8)  
libz.so.1(ZLIB_1.2.2)  
libz.so.1(ZLIB_1.2.2.3)  
libz.so.1(ZLIB_1.2.2.4)  
zlib = 1.2.3-29.el6
zlib(x86-32) = 1.2.3-29.el6
There shouldn't be any "upstream" repositories involved here..
(Assignee)

Comment 22

5 years ago
Ok, that's a good assertion to work from.  I created new mock configs for package building which only include Mozilla's RPM repos.  I ran into a couple of problems (like the package group buildsys-group not existing) but found solutions.

I've now produced a clean build of mozilla-python27-2.7.3-0.el6.x86_64.rpm built on bld-centos6-hp-003 using mock with the correct zlib package dependency.  Will build i386 and do some testing tomorrow.
Can you update the wiki docs regarding how to do package builds?

If you think it's worth it, we could pretty easily build a standalone RPM-generation VM (just file another bug).
(Assignee)

Comment 24

5 years ago
Yes, I can document my steps and update those steps once we have a standard RPM build environment.

I've filed bug 860260 to track creating a standalone RPM generation VM.
(Assignee)

Comment 25

5 years ago
Documentation updated, testing looks good.  I'm aiming to roll out the update in the next hour.
(Assignee)

Comment 26

5 years ago
Created attachment 736789 [details] [diff] [review]
[puppet] update python version
Attachment #736789 - Flags: review?(catlee)

Updated

5 years ago
Attachment #736789 - Flags: review?(catlee) → review+
(Assignee)

Comment 28

5 years ago
Builds with the updated Python are looking good.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 29

5 years ago
Looks like the b2g gecko configs are still missing this new package: https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 30

5 years ago
We'll have to add "notify" events for the mozilla-f16*.cfg files so the yum cache is cleared when they change (same as for the mozilla-centos6-*.cfg files):

 http://hg.mozilla.org/build/puppet/file/f4ff10a3b2bd/modules/mockbuild/manifests/init.pp
(Assignee)

Comment 31

5 years ago
Created attachment 747589 [details] [diff] [review]
[puppet] expire mozilla-f16-*.cfg mock caches

needs testing
Attachment #747589 - Flags: review?(catlee)
(Assignee)

Comment 32

5 years ago
Another, more substantial problem is the build referenced here is not using the mozilla-python27 package (which I had rebuilt to 2.7.3 for CentOS) but is using the Fedora python 2.7.2 which is fetched from our static yum repo.

Some options:
A)
- build the mozilla-python27-2.7.3 package on Fedora
- modify the b2g build configs to use the mozilla-python27 package on Fedora
B)
- refresh our Fedora package mirror snapshot so it includes python-2.7.3.
C)
- copy the Fedora python-2.7.3 package (+ updated dependencies) to the releng Fedora repo (http://repos/repos/yum/releng/public/Fedora/16/) to avoid updating the mirror in B.
D) fix bug 816793 so these use the centos environment, and then we don't have this problem with fedora
(Reporter)

Comment 34

5 years ago
D sounds like the correct solution, but also sounds like the most work. 

What is the quickest path forward here out of the other 3 options, or do we want to invest the time in D?
(Assignee)

Comment 35

5 years ago
Option C should be the quickest solution of the four.  Catlee, can we start with C and plan to do option D at a later time?
Flags: needinfo?(catlee)
I think this now fixed by bug 816793
https://tbpl.mozilla.org/php/getParsedLog.php?id=23828459&tree=Mozilla-Inbound&full=1

checking for python2.7... /tools/python27/bin/python2.7
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Flags: needinfo?(catlee)
Resolution: --- → FIXED
Comment on attachment 747589 [details] [diff] [review]
[puppet] expire mozilla-f16-*.cfg mock caches

Review of attachment 747589 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think we need this any more. In fact, do we need the mozilla-f16-* mock environments any more now?
Attachment #747589 - Flags: review?(catlee)
Product: mozilla.org → Release Engineering
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.