Closed Bug 778977 Opened 12 years ago Closed 12 years ago

Python imaging problems on air-dev.allizom.org

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: peterbe, Assigned: jd)

Details

The PIL python package appears to be correctly installed (unsure about version, tested locally with PIL==1.1.7) but it's unable to use the compiled C binary. 

Traceback: http://peterbe.pastebin.mozilla.org/1725347
Not sure what the issue is, don't see version or arch mismatch, more digging required, relevant info follows:

[root@generic1.dev.webapp.phx1 air]# yum info python-imaging
Loaded plugins: security
Installed Packages
Name        : python-imaging
Arch        : x86_64
Version     : 1.1.7
Release     : 19.el6
Size        : 2.4 M
Repo        : installed
From repo   : mozilla

[root@generic1.dev.webapp.phx1 air]# yum info libjpeg
Loaded plugins: security
Installed Packages
Name        : libjpeg
Arch        : x86_64
Version     : 6b
Release     : 46.el6
Size        : 293 k
Repo        : installed
From repo   : anaconda-RedHatEnterpriseLinux-201009221801.x86_64

[root@generic1.dev.webapp.phx1 air]# ./manage.py shell
~snip~
>>> import Image
# /usr/lib64/python2.6/site-packages/PIL/Image.pyc matches /usr/lib64/python2.6/site-packages/PIL/Image.py
import Image # precompiled from /usr/lib64/python2.6/site-packages/PIL/Image.pyc
dlopen("/usr/lib64/python2.6/site-packages/PIL/_imaging.so", 2);
import _imaging # dynamically loaded from /usr/lib64/python2.6/site-packages/PIL/_imaging.so
# /usr/lib64/python2.6/site-packages/PIL/ImageMode.pyc matches /usr/lib64/python2.6/site-packages/PIL/ImageMode.py
import ImageMode # precompiled from /usr/lib64/python2.6/site-packages/PIL/ImageMode.pyc
# /usr/lib64/python2.6/site-packages/PIL/ImagePalette.pyc matches /usr/lib64/python2.6/site-packages/PIL/ImagePalette.py
import ImagePalette # precompiled from /usr/lib64/python2.6/site-packages/PIL/ImagePalette.pyc
dlopen("/usr/lib64/python2.6/lib-dynload/arraymodule.so", 2);
import array # dynamically loaded from /usr/lib64/python2.6/lib-dynload/arraymodule.so
# /usr/lib64/python2.6/site-packages/PIL/ImageColor.pyc matches /usr/lib64/python2.6/site-packages/PIL/ImageColor.py
import ImageColor # precompiled from /usr/lib64/python2.6/site-packages/PIL/ImageColor.pyc
# /usr/lib64/python2.6/string.pyc matches /usr/lib64/python2.6/string.py
import string # precompiled from /usr/lib64/python2.6/string.pyc
dlopen("/usr/lib64/python2.6/lib-dynload/stropmodule.so", 2);
import strop # dynamically loaded from /usr/lib64/python2.6/lib-dynload/stropmodule.so
dlopen("/usr/lib64/python2.6/lib-dynload/operator.so", 2);
import operator # dynamically loaded from /usr/lib64/python2.6/lib-dynload/operator.so
>>> 
>>> import PIL
import PIL # directory /usr/lib64/python2.6/site-packages/PIL
# /usr/lib64/python2.6/site-packages/PIL/__init__.pyc matches /usr/lib64/python2.6/site-packages/PIL/__init__.py
import PIL # precompiled from /usr/lib64/python2.6/site-packages/PIL/__init__.pyc
The way the library uses it, it's like this::

  >>> from PIL import Image

By the way, the PIL package sucks and has always sucked in python.
Any progress on this? It's not desperately urgent but we're meeting all AirMozilla people today for a project meeting and it would be nice to hand over the dev server to them for testing.
When I run manage.py shell and 

>>> import Image
>>> import PIL
>>> from PIL import Image

I get no errors.  I do not have time to dig into this right now (have blockers).  If you can provide me with some explicate tests to run on the server I can do so but unfortunately that is about all the time I can commit to this today.
One way to test it is to do something like this:


1. cd into the git repo (airmozilla/)

2. start python
>>> from PIL import Image
>>> im = Image.open('airmozilla/main/static/main/img/menu-current.png')
>>> im.size
(24, 10)
this appears to work as expected on both the clusters admin node and web heads...

  [root@generic1.dev.webapp.phx1 air]# python
  Python 2.6.6 (r266:84292, Sep 12 2011, 14:03:14) 
  [GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from PIL import Image
  >>> im = Image.open('airmozilla/main/static/main/img/menu-current.png')
  >>> im.size
  (24, 10)
Peter,

As with every test to date this one completes exactly as expected:

# ssh generic1.dev.webapp.phx1.mozilla.com <- This is the dev environment web server
# cd /data/www/air-dev.allizom.org/air/ <- This is the git repo location (doc root)
# ./manage.py shell <- Also works if I call 'python2.6' directly instead of 'manage.py shell'
>>> from PIL import Image
>>> im = Image.open('airmozilla/main/static/main/img/menu-current.png')
>>> im.size
(24, 10)
>>> quit()

Are you sure that it is PIL causing the issue? The pastebin is no longer up so I can not see it, perhaps you can attach a traceback to the bug as a text file?

Do you have any other ideas as to what it might be or have a test I can run on the python cli to reproduce the error you see in the traceback?
Assignee: server-ops-webops → jcrowe
Thanks for looking into that!
So, it's still happening. Are you sure python was reloaded after you fixed PIL?

http://www.hastebin.com/vineyuwivu.vhdl

The error is quite deep into PIL. The last thing it tries to do is to create a blank canvas for the crop. Something we don't need to know. All we're doing is using the API to resize the image. In fact, another package does that for us called sorl.thumbnail. 

Usually when you do something like `pip install PIL` it spits out a little summary of what will work. I installed PIL==1.1.7 recently and it looks like this:
http://www.hastebin.com/kohajehuti.vhdl

Can you attempt something like that (ie. not using the Red Hat package for python-imaging) and we can perhaps see if we're missing both PNG and JPG libs or something like that?
We've fixed the problem - PIL was coming from vendor-local/ (accidentally included and compiled for the wrong machine) instead of the system libraries (/usr/lib64...).

Everything looks to be working now, thanks!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Peter,

I am curious about one thing here. I can do basic image manipulation from the cli without error.  I notice in the traceback that you are loading PIL from:

/data/www/air-dev.allizom.org/air/vendor-local/lib/python/PIL/Image.py

But the one from the package installed by Debian is:

/usr/lib64/python2.6/site-packages/PIL/Image.py

Is it possible that the one you have shipped with the site is broken on the server, like a $PATH issue or the like?  Is there a way to test the site using the one in /usr instead of the project local?

I ask since no matter what I try I cannot seem to duplicate the error.  I have tried loading the lib from the project directory and I still cannot seem to duplicate the error.  Is it possible for you to give me a set of commands that I can use locally to generate the error?  From the source:

    # If the _imaging C module is not present, you can still use
    # the "open" function to identify files, but you cannot load
    # them.

Hence I assume if I call im.load() it should generate this error, however:

# python2.6
Python 2.6.6 (r266:84292, Sep 12 2011, 14:03:14) 
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> cwd = os.getcwd()
>>> os.chdir ('vendor-local/lib/python/PIL')
>>> import Image
>>> os.chdir (cwd)
>>> im = Image.open('airmozilla/main/static/main/img/menu-current.png')
>>> im.size
(24, 10)
>>> im.load()
<PixelAccess object at 0x7fdfb57b01b0>
>>>

Although I think that calling im.size should call load directly, which should cause the error.  FYI I get identical results loading the sys lib instead of the project local.  At the end of the day I just cannot seem to be able to generate the error. Also I am no python guru so feel free to call me a moron if I am missing some basic idea.
HAH, I guess I should reload before I submit an response.

Ignore moron post
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.