1303843 - Rebuilding desktop-test image introduces new test failures seemingly due to python-imaging 1.1.7-4ubuntu0.12.04.2 update

Status: RESOLVED INVALID

(Testing :: General, defect)

Description

[Gregory Szorc [:gps]]

Broken off from bug 1289812 comment 17:

My push at https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=21fdf73bbb17e34cfe00e372695c4f21e4ba3e6a (and the subsequent partial backout of that push) resulted in spurious test failures on Linux64 (like https://treeherder.mozilla.org/logviewer.html#?job_id=3573922&repo=autoland#L10745). My changes should not have introduced regressions this way. The only thing I can come up with is that by triggering Docker images regeneration, we picked up a new version of an upstream package and that somehow broke tests.

So now we have a timebomb on both the Linux build and test images: the next person to touch them breaks things. This basically blocks my work on running tests from a source checkout. I don't like this rabbit hole.

Comment 1

[Gregory Szorc [:gps]]

For posterity:

good: https://queue.taskcluster.net/v1/task/DDyV00WJTM66izR_GlrRGw/runs/0/artifacts/public%2Fimage.tar
bad: https://queue.taskcluster.net/v1/task/MNJ0VnRNSsGyvOv5fSLrtQ/runs/0/artifacts/public%2Fimage.tar

You can load those into Docker via `curl -L <url> | docker load`.

Comment 2

[Gregory Szorc [:gps]]

The regression is either due to GCC 4.6.3 -> 4.6.4 or python-imaging.

python-imaging is a security update and I don't see how it could cause multiple test failures.

The GCC package update also includes updates to libstdc++6, libg++, etc, which are all derived from the "gcc-4.6" source package. So I'm going to assume the regression is related to the GCC version bump. Why exactly, I'm not sure.

Upon further investigation, 5290d874c8c5 (bug 1280178) introduced the ubuntu-toolchain-r PPA into the Docker image. This PPA provides a GCC 4.6 package which overrides the default shipped by Ubuntu. This is where the new 4.6.4 GCC packages are coming from.

I reckon if we prevent gcc-4.6 (and derived) packages to come from this PPA, this regression will sort itself out.

Comment 3

[Gregory Szorc [:gps]]

Gah - wrong theory about GCC 4.6.4. We do retain the Ubuntu GCC 4.6.3 in the Docker images because we don't do an `apt-get upgrade` after the PPA is configured.

The only apt package varying between the good and bad image is python-imaging. WTF.

Comment 4

[Gregory Szorc [:gps]]

I diffed the filesystems of the good and bad Docker images and can confirm that python-imaging is the only changed package. 1.1.7-4ubuntu0.12.04.1 to 1.1.7-4ubuntu0.12.04.2. Not sure how that could possibly result in Firefox tests failing from within Firefox (we fail tests like test_popupZoom.xul).

There are a number of changed files. Many of these appear to be due to timestamps in files. Few of them seem like they could cause a change in Firefox behavior. Those that could include:

* /usr/share/icons/*/*.cache (e.g. /usr/share/icons/unity-icon-theme/icon-theme.cache)
* /var/cache/fontconfig/*.cache*

I guess I'll try pinning python-imaging in the rare case it is the culprit.

Comment 5

[Gregory Szorc [:gps]]

For my sanity:

Previous push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d41128490bb70045b36caab39239011621b94728
Bad push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=21fdf73bbb17e34cfe00e372695c4f21e4ba3e6a

Comment 6

[Gregory Szorc [:gps]]

I think the security/package update for python-imaging (1.1.7-4ubuntu0.12.04.2 - http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/python-imaging/precise-security/revision/29?start_revid=29) somehow broke Ubuntu 12.04.

When I spawn an interactive TaskCluster worker and connect to the VNC display, the bad Docker image (post python-imaging update) doesn't render correctly. The window title isn't visible and the close, minimize, and maximize icons don't render. It appears the window is shifted up several pixels. This happens with *every* window inside the window manager, not just Firefox.

The failing tests all seem to measure visual alignment of various elements. I reckon the different behavior of the window decorations is causing the tests to fail, possibly due to pixels in the window bar not being visible due to being shifted off the painted area of the desktop.

It looks like python-imaging is used by various system processes, including fontconfig. It is possible the security patch introduced bad behavior or something was relying on the previous bad behavior and now broke.

I'll upload screenshots in a few minutes...

Comment 7

[Gregory Szorc [:gps]]

Attachment: good.png

This shows the rendering of Firefox on a desktop on a good Docker image.

Comment 8

[Gregory Szorc [:gps]]

Attachment: bad.png

And here's how Firefox renders after python-imaging is upgraded. Note the top of the window appears cropped off.

Comment 9

[Gregory Szorc [:gps]]

I filed https://bugs.launchpad.net/ubuntu/+source/python-imaging/+bug/1625415.

Not the greatest upstream bug report. Maybe, just maybe, someone will recognize the symptoms and know what tires to kick.

It's near EOD for me. So don't expect much movement on this from my end until tomorrow.

Comment 10

[(not currently active) Ted Mielczarek]

It looks like maybe the window manager is deciding to place the windows at a strange location?

Comment 11

[Gregory Szorc [:gps]]

I manually built the old python-imaging package. However, https://treeherder.mozilla.org/#/jobs?repo=try&revision=f1011338f42c575dc439109f7e2d91d6b72b77c9 is showing the same failures.

So maybe this has nothing to do with python-imaging after all. On one hand I'm relieved because the python-imaging change should have not impacted things. On the other, now I have to go back to square 1 to see what else is different in the images to figure out what caused this regression.

I hate rabbit holes.

Comment 12

[Gregory Szorc [:gps]]

This was due to a bug in the patch in bug 1302907. Specifically, we got bit by a bash-ism of sorts. I think this was triggered from accidentally running compiz when we shouldn't have been. Oy.