747998 - RHEL6 build issue: "python26-simplejson" incompatible with "python-simplejson"

Status: VERIFIED INCOMPLETE

(Cloud Services :: Server: Other, defect)

Description

[:Atoll]

On all of our python products, building RPMs for RHEL6 produces, among others:

python26-simplejson-2.1.6-1.x86_64.rpm

This appears to be a result of the simplejson package trying to install to /usr/lib64/python2.6 (system stock python), rather than /usr/lib/python2.6 (mozilla python).

Other dependent packages do not seem to have this issue.

Comment 1

[:Atoll]

# rpm -ivh python26-simplejson-2.1.6-1.x86_64.rpm 
Preparing...                ########################################### [100%]
	file /usr/lib64/python2.6/site-packages/simplejson/__init__.py from install of python26-simplejson-2.1.6-1.x86_64 conflicts with file from package python-simplejson-2.0.9-3.1.el6.x86_64
[above error repeated for every file in the rpm]

Comment 2

[Ryan Kelly [:rfkelly]]

Unfortunately the 2.0.9 version of simplejson is missing some APIs that our apps depend on, so we can't just use the system version.  Is it possible to uninstall python-simplejson and use python26-simplejson instead?  Or will this break other system packages that depend on the system simplejson package?

Comment 3

[Ryan Kelly [:rfkelly]]

(In reply to Richard Soderberg [:atoll] from comment #0)
> On all of our python products, building RPMs for RHEL6 produces, among
> others:
> 
> python26-simplejson-2.1.6-1.x86_64.rpm
> 
> This appears to be a result of the simplejson package trying to install to
> /usr/lib64/python2.6 (system stock python), rather than /usr/lib/python2.6
> (mozilla python).

Is there such a thing as "system stock python" versus "mozilla python" on the deployment machines?  From what I can see on r6.build there is only one python installation.  Pure-python libraries get installed into /usr/lib/python2.6, while libraries with compiled extension modules get installed into /usr/lib64/python2.6, and the system python will look for libraries in both locations.

The following is what I hacked up for the Sync2.0/AITC build scripts:

    # The simplejson rpms conflict with a RHEL6 system package.
    # Do a custom build so that they can overwrite rather than conflict.
    rm -f $(CURDIR)/rpms/python26-simplejson-2.4.0-1.x86_64.rpm
    wget -O ${BUILD_TMP}/simplejson-2.4.0.tar.gz http://pypi.python.org/packages/source/s/simplejson/simplejson-2.4.0.tar.gz
    cd ${BUILD_TMP}; tar -xzvf simplejson-2.4.0.tar.gz
    cd ${BUILD_TMP}/simplejson-2.4.0; python setup.py  --command-packages=pypi2rpm.command bdist_rpm2 --binary-only --name=python-simplejson --dist-dir=$(CURDIR)/rpms
    rm -rf ${BUILD_TMP}/simplejson-2.4.0
    rm -f ${BUILD_TMP}/simplejson-2.4.0.tar.gz

Basically this rebuilds the python26-simplejson RPM so that it is named python-simplejson, allowing it to be used as an upgrade of the system package rather than conflicting with it.  Is this an acceptable solution for other products as well?

Comment 4

[:Atoll]

What about:

Provides: python-simplejson

?

Comment 5

[:Atoll]

Sorry, I didn't mean to submit that yet.

What about:

Provides: python-simplejson

?  So that yum is aware that python26-simplejson is able to upgrade the system one.

I'm not comfortable with renaming to "python-" because we're about to ship Python 2.7 support, which will require the "python27-" prefix (so we don't break system python).

Perhaps we shouldn't be using the default libdir in a non-default Python package?

Comment 6

[Ryan Kelly [:rfkelly]]

(In reply to Richard Soderberg [:atoll] from comment #5)
> Sorry, I didn't mean to submit that yet.
> 
> What about:
> 
> Provides: python-simplejson
> 
> ?  So that yum is aware that python26-simplejson is able to upgrade the
> system one.

I'll see if I can make that happen, I'm not sure just how much pypi2rpm will allow me to customize the details of the RPM.

> I'm not comfortable with renaming to "python-" because we're about to ship
> Python 2.7 support, which will require the "python27-" prefix (so we don't
> break system python).

Will this be as a second, non-system installation of python?  In that case we wouldn't run into this conflicting-versions problem at all.

> Perhaps we shouldn't be using the default libdir in a non-default Python
> package?

One alternative would be to ship a single RPM with a full virtualenv
inside it, so its packages are completely isolated from the system python.  But that's a pretty significant change from the current setup.

Comment 7

[:Atoll]

I looked and, when the system python matches our target version, we simply use system python. I don't see any value in changing that now, just for the sake of it.

So, presuming Python 2.6:

(1) When we say, via a make option of some sort, "build RPMS with prefix python-", it builds all RPMs as "python-" rather than "python26-". Then using that option on RHEL6 would fix the compatibility issues.

(2) When we say, on RHEL6, "build RPMS with prefix python26- and Provides: python-", it does so. There is technically support for a "provides" value in the bdist_rpm code, but it doesn't seem exposed to our control yet.

I suspect that (1) is the correct solution.

Comment 8

[Pete Fritchman [:petef]]

(In reply to Ryan Kelly [:rfkelly] from comment #6)
> (In reply to Richard Soderberg [:atoll] from comment #5)
> > Sorry, I didn't mean to submit that yet.
> > 
> > What about:
> > 
> > Provides: python-simplejson
> > 
> > ?  So that yum is aware that python26-simplejson is able to upgrade the
> > system one.
> 
> I'll see if I can make that happen, I'm not sure just how much pypi2rpm will
> allow me to customize the details of the RPM.

I think you can provide a template .spec file to override some stuff like this.
In addition to Provides, we may want "Obsoletes: python-simplejson" too.

FWIW, we have the same issue with "python26-m2crypto" (part of the tokenserver build) vs the system-level "m2crypto" RPM.

I'm hoping to avoid these issues in the future by changing our deploy method to a chroot, where everything we need is self-contained in the deploy package (so we don't interfere with or depend on the system Python).

Comment 9

[:Atoll]

As a chroot alternative, there's also adding --prefix=/opt/python to python.spec and recompiling. Any lib built with that binary would presumably then install to /opt/python/lib automatically, providing the chroot-like behavior desired.

Comment 10

[:Atoll]

We don't need any short-term solutions, and any long-term solution will be some other bug. Closing as INCOMPLETE.

Comment 11

[James Bonacci [:jbonacci]]

Verifying as such...