485956 - Set up python2.5 or 2.6 with MySQLdb extension on graphs-stage2

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Chris AtLee [:catlee]]

The regression detection code currently requires Python 2.5 or higher to run, as well at the MySQLdb extension.

Comment 1

[Dave Miller [:justdave]]

I found RHEL5 srpms for Python 2.5.1 and 2.6 which install in an alternate location so they won't overwrite the built-in python (since it's used for almost everything by the core OS utilities) and tried to build them back on Friday.  Unfortunately those are both outdated versions, and the newer upstream version was not a drop-in replacement in the srpm because of patch conflicts, so it got put on hold for the Firefox release.

Comment 2

[Ryan Doherty (:rdoherty)]

Any updates for this? This is one of 2 remaining issues for switching over to the new codebase. Thanks!

Comment 3

[matthew zeier [:mrz]]

Reed, can you pitch in here for justdave?  Can you provide an ETA too (you can shift this ahead of other things you have)?

Comment 4

[Dave Miller [:justdave]]

er, was going to say (before midairing with mrz)

---
I haven't had a chance to mess with 2.6.1 yet, but I just installed 2.6 for you
to get you started until I have time to mess with it.  If you're missing any
libraries or anything let me know.
---

But in this case, it'll tide you over until Reed can mess with updating it for the current version. :)

Comment 5

[matthew zeier [:mrz]]

Is this fixed then?  You wanted 2.6 installed, Dave's comment in comment #4 says that's installed.  Do you need 2.6.1?

Comment 6

[Chris AtLee [:catlee]]

Yes, this is fixed now.  Thanks!

Comment 7

[Dave Miller [:justdave]]

No, I want the security update applied.  This is open to remind me that I still need to build it.  It's no longer a high priority though because he has a working version to use for development in the mean time.

Comment 8

[Chris AtLee [:catlee]]

What I'm doing on this machine will most likely be moved over to the machine from bug 492135 once it's ready.

Comment 9

[Shyam Mani [:fox2mike]]

Alright, I've built the following python-2.6.2 RPMs :

python26-2.6.2-geekymedia1.rhel5.i386.rpm
python26-debuginfo-2.6.2-geekymedia1.rhel5.i386.rpm
python26-devel-2.6.2-geekymedia1.rhel5.i386.rpm
python26-libs-2.6.2-geekymedia1.rhel5.i386.rpm
python26-test-2.6.2-geekymedia1.rhel5.i386.rpm
python26-tools-2.6.2-geekymedia1.rhel5.i386.rpm

I'll hold on installing them to graphs-stage2 for now, till I'm sure what the standard procedure to push out RPMs is (mrepo or otherwise). Once it's verified and tested, I'll close the bug.

Comment 10

[Shyam Mani [:fox2mike]]

I've upgraded what Dave had originally installed (from 2.6 to 2.6.2). Please reopen in case you see any major issues after this.

Python 2.6.2 (r262:71600, Jun  1 2009, 20:55:03) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2

Comment 11

[Dave Miller [:justdave]]

Thanks Shyam!  There were two or three extensions I had to rebuild against python26 as well, the existing ones I had there still worked and didn't have to be rebuilt for 2.6.2 I take it?  (mysqldb comes to mind, seems like there was another, too)

The RPMs were just locally installed on that box, but we do need to get them into mrepo or similar before it goes on a production box.

Comment 12

[Shyam Mani [:fox2mike]]

(In reply to comment #11)
> Thanks Shyam!  There were two or three extensions I had to rebuild against
> python26 as well, the existing ones I had there still worked and didn't have to
> be rebuilt for 2.6.2 I take it?  (mysqldb comes to mind, seems like there was
> another, too)

I think the only issue was with tkinter26-2.6-geekymedia1.rhel5.i386.rpm which I didn't rebuild (since it said it was a graphical editor of sorts) and forced the others through. No other dependency errors were reported.

We might want to re-look at this and either 
a) remove the tkinter dep before we take it to prod
b) Simply look at tkinter's src rpm and make it dep on 2.6.2 instead of 2.6 (I didn't think of this before) and rebuild.

Thoughts?

Comment 13

[Dave Miller [:justdave]]

I prefer rebuilding to fix the dependencies than trying to do a force-install.  Any time we have to --no-deps on an install means we're going to break any automated installs later.

Comment 14

[Shyam Mani [:fox2mike]]

(In reply to comment #13)
> I prefer rebuilding to fix the dependencies than trying to do a force-install. 

Point. I'll look at tkinter and update the bug.

Comment 15

[Shyam Mani [:fox2mike]]

Nothing special was needed, tkinter-2.6.2 was already built using the 2.6.2 spec file, just that I missed out copying over the rpm when doing the localinstall. I've gone ahead and fixed that.

Comment 16

[Chris AtLee [:catlee]]

Having problems with at least the datetime module:
[catlee@bm-buildgraph01 analysis]$ python2.6
Python 2.6.2 (r262:71600, Jun  1 2009, 20:55:03) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/lib/python2.6/lib-dynload/datetimemodule.so: undefined symbol: _PyTime_DoubleToTimet

Comment 17

[Shyam Mani [:fox2mike]]

Okay, funny way to fix that from this Japanese page (translated via google) : http://is.gd/Pb1a 

[root@bm-buildgraph01 ~]# export  LD_PRELOAD=/usr/lib/python2.6/lib-dynload/timemodule.so
[root@bm-buildgraph01 ~]# python26 
Python 2.6.2 (r262:71600, Jun  1 2009, 20:55:03) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>>

I'll leave this open for now till we can figure out why this is happening. For now this solution should help you move along without being blocked.

Comment 18

[Mark Hahnenberg (:mhahnenberg)]

It appears that this workaround solves the datetime problem but creates a new problem in at least one other area when trying to spawn subprocesses:

[root@khan ~]$ LD_PRELOAD=/usr/lib/python2.6/lib-dynload/timemodule.so python -c "import subprocess; subprocess.Popen(['/bin/ls'])"
/bin/ls: symbol lookup error: /usr/lib/python2.6/lib-dynload/timemodule.so: undefined symbol: PyExc_ValueError

Comment 19

[Jeff Balogh (:jbalogh)]

Mark: The problem is actually when you use ls, not subprocess, right?  Exporting LD_PRELOAD and then hitting `ls` shows the "symbol lookup error".

There's better ways to get files than `ls`: use os.path or path.py: http://pypi.python.org/pypi/path.py

Comment 20

[Mark Hahnenberg (:mhahnenberg)]

No, it's not the fact that I used ls.  It originally failed when I used /usr/bin/firefox.  /bin/ls was just for sake of an example that didn't require any special programs to test it with.

Comment 21

[Shyam Mani [:fox2mike]]

I think I'm going to do what I did on khan here, bug 510051 for more details and just close this :) I'll do that tomorrow.

Comment 22

[Shyam Mani [:fox2mike]]

Trevor, this is fixed now?

Comment 23

[chizu]

python26-2.6.2-geekymedia1.1.rhel5.i386.rpm has the fix for the timemodule.so problem.

Comment 24

[Shyam Mani [:fox2mike]]

Trevor,

Does it also fix the issue mentioned in comment #18? If that's the case, I think we can close this bug.

Comment 25

[Jeff Balogh (:jbalogh)]

(In reply to comment #24)
> Does it also fix the issue mentioned in comment #18? If that's the case, I
> think we can close this bug.

If datetime is fixed the work-around is unnecessary.

Comment 26

[chizu]

(In reply to comment #24)
> Trevor,
> 
> Does it also fix the issue mentioned in comment #18? If that's the case, I
> think we can close this bug.
The problem was subprocess inheriting LD_PRELOAD. ls attempts to dynamically link the timemodule.so library and discovers it doesn't have the required Python libraries loaded to do so. 

Without LD_PRELOAD, subprocess.Popen works as expected.

Comment 27

[Shyam Mani [:fox2mike]]

I think this can RIP now :)

Thanks chizu!