Closed Bug 1227652 Opened 4 years ago Closed 2 years ago

(harmless) ImportError: No module named pygtk (only mochitests)

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: armenzg, Unassigned)

References

Details

It seems that we're hitting a pygtk import issue during certain jobs.
I tried installing it but the build system would fail.

I have tried to install the wrong thing (a Python package) [1]

The only reference I see it is a commented line in a pylintrc file [2]

The latest release is in here [3]

I will need to look a bit more into this. Like determining what is installed on the EC2 instances.

[1] https://hg.mozilla.org/try/rev/10a7aef794ab#l1.17
[2] https://dxr.mozilla.org/mozilla-central/search?q=pygtk&redirect=false&case=false
[3] http://ftp.gnome.org/pub/GNOME/sources/pygtk/2.24/

#### BUILD ERROR ####
Step 21 : RUN pip install pygtk
 ---> Running in 8c7fa0fa02bc
Downloading/unpacking pygtk
  Running setup.py egg_info for package pygtk
    ********************************************************************
    * Building PyGTK using distutils is only supported on windows. *
    * To build PyGTK in a supported way, read the INSTALL file.    *
    ********************************************************************
    Complete output from command python setup.py egg_info:
    ********************************************************************

* Building PyGTK using distutils is only supported on windows. *

* To build PyGTK in a supported way, read the INSTALL file.    *

********************************************************************

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/worker/.pip/pip.log
The command '/bin/sh -c pip install pygtk' returned a non-zero code: 1
What's the context where you're seeing this?  It looks like it's in a docker build, but you're saying it's during a job?
I'm a bit confused why that's chunk 7 out of 15, when it appears from testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml that mochitest-plain has 5 chunks.  I was hoping to find this test in the buildbot runs.

I don't see anything about pygtk in puppet, and the tests run in a virtualenv created within the mozharness script, so pygtk shouldn't need to be installed systemwide.  Finally, it seems that the test run continues despite the error.

It's unhelpful that something is catching this Python exception and not printing the traceback, as we don't know what is importing pygtk.  As you've seen, it's nothing in gecko.

That said, I do see it on your tester:

[root@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]# find /usr -name 'pygtk*'
/usr/lib/python2.7/dist-packages/pygtk.pyc
/usr/lib/python2.7/dist-packages/pygtk.pth
/usr/lib/python2.7/dist-packages/gi/pygtkcompat.py
/usr/lib/python2.7/dist-packages/gi/pygtkcompat.pyc
/usr/lib/python2.7/dist-packages/pygtk.py

and that seems to be required by python-gobject-2:
[root@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]# dpkg -S /usr/lib/python2.7/dist-packages/pygtk.py
python-gobject-2: /usr/lib/python2.7/dist-packages/pygtk.py

Tracing that back:

[root@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]# apt-cache rdepends --installed python-gobject-2
python-gobject-2
Reverse Depends:
  python-gtk2
  python-gobject
  python-gconf
 |python-dbus
  hplip

looking in the apt history log, I see

Start-Date: 2013-02-16  17:01:16
Commandline: apt-get install -y ubuntu-desktop openssh-server makedev curl grub linux-image-generic
.. python-gobject-2:amd64 ..

suggesting that this was installed along with ubuntu-desktop long, long ago when the base image for testers was built.

So, if you can verify that the apt-get install of ubuntu-desktop in the docker build process doesn't install this package, then perhaps it makes sense to just list it explicitly as a package to install in that process.  It looks like installing it with pip is a bad idea :)
You're talking about python-gobject-2:amd64, right?
I'm very puzzled as to why the issue show under Mozharness [1].
It starts almost as soon as the tests start running.
I thought it might have had to do with how the venv is set up or the python versions [2].

In armenzg/0.4.5, the package is not installed meaning that ubuntu-desktop installs it:
 dpkg -l | grep python-gobject-2

Did I answer your question in your last comment? [3]

I think I have a theory. We have something else putting output to the root logger and we're getting confused with Mozharness' output.
The output just after is GStreamer output.

I also wonder if this has anything to do with the lack of PULSE_AUDIO:
+ : NEED_PULSEAUDIO false

[1] Beginnig of running tests
18:39:49     INFO - Calling ['/home/worker/workspace/build/venv/bin/python', '-u', '/home/worker/workspace/build/tests/mochitest/runtests.py', '--total-chunks', '15', '--this-chunk', '7', '--appname=/home/worker/workspace/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://queue.taskcluster.net/v1/task/KDzsbLz3T0qlmu7v5rZMYA/artifacts/public/build/target.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--setpref=webgl.force-enabled=true', '--quiet', '--log-raw=/home/worker/workspace/build/blobber_upload_dir/plain-chunked_raw.log', '--log-errorsummary=/home/worker/workspace/build/blobber_upload_dir/plain-chunked_errorsummary.log', '--use-test-media-devices', '--screenshot-on-fail', '--chunk-by-dir=4'] with output_timeout 1000
18:39:49     INFO -  Checking for orphan ssltunnel processes...
18:39:49     INFO -  Checking for orphan xpcshell processes...
18:39:50     INFO -  SUITE-START | Running 143 tests
...
18:39:50     INFO -  TEST-SKIP | dom/plugins/test/mochitest/test_windowed_invalidate.html | took 1ms
18:39:50     INFO -  dir: dom/media/webspeech/recognition/test
18:39:51     INFO -  ImportError: No module named pygtk
11:22:16     INFO -  Setting pipeline to PAUSED ...
11:22:16     INFO -  libv4l2: error getting pixformat: Invalid argument
11:22:16     INFO -  Pipeline is PREROLLING ...
11:22:17     INFO -  Pipeline is PREROLLED ...
11:22:17     INFO -  Setting pipeline to PLAYING ...
11:22:17     INFO -  New clock: GstSystemClock
11:22:17     INFO -  Got EOS from element "pipeline0".
11:22:17     INFO -  Execution ended after 32402009 ns.
11:22:17     INFO -  Setting pipeline to PAUSED ...
11:22:17     INFO -  Setting pipeline to READY ...
11:22:17     INFO -  Setting pipeline to NULL ...
11:22:17     INFO -  Freeing pipeline ...
11:22:17     INFO -  23
11:22:18     INFO -  pk12util: PKCS12 IMPORT SUCCESSFUL
11:22:18     INFO -  MochitestServer : launching [u'/builds/slave/test/build/tests/bin/xpcshell', '-g', '/builds/slave/test/build/application/firefox', '-v', '170', '-f', '/builds/slave/test/build/tests/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpu8BnZl.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/builds/slave/test/build/tests/mochitest/server.js']

[2]
### Docker image
root@taskcluster-worker:~# python --version
Python 2.7.3
root@taskcluster-worker:~# virtualenv --version
13.1.2
root@taskcluster-worker:~# python -c "import pygtk"
root@taskcluster-worker:~# virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip, wheel...done.
root@taskcluster-worker:~# source venv/bin/activate
(venv)root@taskcluster-worker:~# python -c "import pygtk"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pygtk

### Test host
[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ /tools/buildbot/bin/python --version
Python 2.7.3
[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --version
1.7.1.2
[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ /tools/buildbot/bin/python /tools/misc-python/virtualenv.py venv
Using real prefix '/usr'
New python executable in venv/bin/python
Installing setuptools.............................done.
Installing pip.................done.
[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ source venv/bin/activate
(venv)[cltbld@tst-linux64-ec2-armenzg.test.releng.use1.mozilla.com ~]$ python -c "import pygtk"Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pygtk


[3]
root@taskcluster-worker:~# find /usr -name 'pygtk*'
/usr/share/pyshared/gi/pygtkcompat.py
/usr/share/pyshared/pygtk.py
/usr/share/pyshared/pygtk.pth
/usr/share/pygtk
/usr/lib/python2.7/dist-packages/gi/pygtkcompat.py
/usr/lib/python2.7/dist-packages/gi/pygtkcompat.pyc
/usr/lib/python2.7/dist-packages/pygtk.py
/usr/lib/python2.7/dist-packages/pygtk.pyc
/usr/lib/python2.7/dist-packages/pygtk.pth
root@taskcluster-worker:~# dpkg -S /usr/lib/python2.7/dist-packages/pygtk.py
python-gobject-2: /usr/lib/python2.7/dist-packages/pygtk.py
root@taskcluster-worker:~# apt-cache rdepends --installed python-gobject-2
python-gobject-2
Reverse Depends:
  python-gobject
  hplip
  python-gtk2
  python-gobject
  python-gconf
 |python-dbus
  hplip
root@taskcluster-worker:~# apt-cache showpkg python-gobject-2
Package: python-gobject-2
Versions: 
2.28.6-10ubuntu1 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise_main_binary-amd64_Packages.gz) (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise_main_binary-amd64_Packages.gz
                  MD5: a8aab9350d9d42a2df907a51577f79b7


Reverse Depends: 
  python-gobject,python-gobject-2
  hplip,python-gobject-2
  python-gobject-2:i386,python-gobject-2
  python-indicate,python-gobject-2 2.28.1
  meld,python-gobject-2 2.16
  bzr-dbus,python-gobject-2
  python-gtk2-doc,python-gobject-2
  python-gtk2,python-gobject-2 2.21.3
  python-gobject-2-dev,python-gobject-2 2.28.6-10ubuntu1
  python-gobject-2-dbg,python-gobject-2 2.28.6-10ubuntu1
  python-gobject,python-gobject-2
  python-gnome2,python-gobject-2 2.17.0
  python-gconf,python-gobject-2 2.17.0
  python-dbus,python-gobject-2
  hplip,python-gobject-2
Dependencies: 
2.28.6-10ubuntu1 - python2.7 (0 (null)) python (2 2.7.1-0ubuntu2) python (3 2.8) libc6 (2 2.14) libffi6 (2 3.0.4) libglib2.0-0 (2 2.24.0) python-gobject-2-dbg (0 (null)) python-gobject (3 2.90) python-gobject:i386 (3 2.90) python-gobject (3 2.90) python-gobject:i386 (3 2.90) python-gobject-2:i386 (0 (null)) 
Provides: 
2.28.6-10ubuntu1 - python2.7-gobject-2 python2.7-gobject 
Reverse Provides: 
root@taskcluster-worker:~# ls -l /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise_main_binary-amd64_Packages.gz
-rw-r--r-- 1 root root 1640344 Apr 25  2012 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise_main_binary-amd64_Packages.gz

root@taskcluster-worker:~# dpkg -l | grep python-gobject-2
ii  python-gobject-2                       2.28.6-10ubuntu1                                    deprecated static Python bindings for the GObject library
> In armenzg/0.4.5, the package is not installed meaning that ubuntu-desktop installs it:
> dpkg -l | grep python-gobject-2

Yep, I see that it is not installed on that image, and that `apt-get install ubuntu-desktop` will pull it in.  I have no idea how that image was created, though -- the last patch I saw had ubuntu-desktop installed, yet it's not installed on that image?

So I think the problem actually isn't very complex: we will get pygtk for free by installing a window manager (bug 1223123).

The rest of your message confuses me.  In [2], it looks like pygtk is already installed, but in the first part of your comment you said it isn't, and my verification of armenzg/desktop-test:0.4.5 indicates the same thing.  And I don't see how not starting pulseaudio could cause an unrelated Python package to disappear from the filesystem.
Yes, it is installed.
The problem is that somewhere, that import error is being raised.
I don't know why we don't raise the same import error on the ec2 releng machines since both there and on the docker image we have pygtk installed at the host level.

In both cases we don't have it installed under the venv.
No longer blocks: tc-linux64-debug
Summary: Fix pygtk import issue → (harmless) ImportError: No module named pygtk (only mochitests)
I will repeat the relevant information. This is a harmless issue that only shows up on certain desktop tests under TaskCluster/docker.

> We have something else putting output to [Python's] root logger and we're getting confused with Mozharness' output.
> The output just after is GStreamer output.
---
> 18:39:49     INFO -  Checking for orphan ssltunnel processes...
> 18:39:49     INFO -  Checking for orphan xpcshell processes...
> 18:39:50     INFO -  SUITE-START | Running 143 tests
> ...
> 18:39:50     INFO -  TEST-SKIP | dom/plugins/test/mochitest/test_windowed_invalidate.html | took 1ms
> 18:39:50     INFO -  dir: dom/media/webspeech/recognition/test
> 18:39:51     INFO -  ImportError: No module named pygtk
> 11:22:16     INFO -  Setting pipeline to PAUSED ...
> 11:22:16     INFO -  libv4l2: error getting pixformat: Invalid argument
Assignee: armenzg → nobody
"Harmless" exceptions being printed to the logs are still pretty problematic - they cause false positives when log parsing, and so confusion with both devs and also things like auto-classify (eg bug 1268333).

As such, it would be great to try and suppress this exception :-)
Blocks: 1268333
Based on comment 3, we really have no idea what's causing this.  I can think of two things to try:

 1. Glandium did some work related to GTK recently, so perhaps has some context?
 2. In my experience it's usually mozharness scripts that consume and log (incompletely) errors, so perhaps addressing some of those bugs will help provide context for this one.
Component: Docker Images → Mozharness
Flags: needinfo?(mh+mozilla)
Product: Taskcluster → Release Engineering
QA Contact: jlund
What is using pygtk? Certainly not firefox itself.
Flags: needinfo?(mh+mozilla)
I've pushed a change that will *hopefully* (I doubt it though) show us which module is logging that message.

We might want to make Mozharness hide such message.
I assume some code we call for these mochitest is a python module which also logs to the root logger. Thus showing up in Mozharness' logging.
Duplicate of this bug: 1282597
Given the location of the error in the log, I'm suspicious that it's something in `findTestMediaDevices` triggering it:
https://dxr.mozilla.org/mozilla-central/rev/b69a5bbb5e40bd426e35222baa600b481e50d265/testing/mochitest/runtests.py#1448

...which matches up with Armen's earlier investigation of it being related to `NEED_PULSEAUDIO`--we need PulseAudio for media jobs where we need `--use-test-media-devices`.

I thought maybe it might be `gst-launch-0.10` or `pactl` trying to do the import, but those both appear to be native binaries on my Ubuntu 14.04 system.
See Also: → 1272849
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.