405384 - pngsuite reftest fails on Windows with slight color differences

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect, P5)

Description

[Justin Dolske [:Dolske]]

Spinoff from bug 386651 comment 14. Also see attachment 288612 [details].

So, after checking in Part 1, all the pngsuite reftests failed on the Windows
boxes (qm-win2k3-01, qm-winxp01). I checked MD5s of the logged images, and both
boxes failed in an identical manner on the first test, which I'll examine
here... The reftests pass on my OS X and Solaris boxes. Manually the test
html/png under Vista seems ok to me.

Attached is a composite of the results for z00n2c08.png. The test image is just
a boring RGB gradient, no transparency, alpha, gamma, ICC, etc (verified with
pngcheck). The docs say these particular test images were made with different
zlib compression levels specified, which I wouldn't expect to be relevant to
the displayed result.

The image on the left is from the reftest log for the "reference" image (the
HTML table). The middle image is from the "test" image (the PNG loaded in the
reftest browser window). The rightmost image is the PNG as loaded in GIMP when
I created this composite.

Note that the left and right image appear identical (although I didn't test
this exhaustively). The middle image has a subtle but noticeable horizontal
interruption in the center of the image. You'll need to zoom in. :-) Flipping
between them, it looks like most of the image differs subtly but more
noticeably towards the middle. Some quick GIMP filtering seems to show that
most of the image has a slightly-off green channel, with a handful of pixels
having red/blue differences.

I measured the blueish pixel marked with a black pixel between images... The
pixels in the left and right images are rgb(0,123,132). The value specified in
the HTML is rgb(0,123,132). The value in the middle image is rgb(0,121,132) --
WRONG! [All pixels were fully opaque]

This implies that PNG is being painted incorrectly in the test environment [or
perhaps that drawWindow() is mangling it?].

Comment 1

[Justin Dolske [:Dolske]]

So, I added some more tests (modules/libpr0n/test/reftest/pngsuite-basic-n/), and again there are subtle Windows-only failures...

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1196052420.1196055118.25279.gz

The good news is that the 1-bit greyscale test (basn0g01.png) passes, so there's some progress! :-) The bad news is that the others all fail. :-(

To start off with, I looked at basn0g02.png. Attached is a composite of the original PNG and 2 tests.

The PNG is a 2-bit / 4-color grayscale PNG -- black, white, 85/85/85 and 170/170/170 (triple checked with Gimp, Photoshop, and ImageMagick ("convert foo.png foo.txt")). Completely opaque, no gamma or color profile chunks. 

The "reference" result (HTML table) pixels are ok for black and white. The gray shades are 155/155/155 and 212/212/212.

The "test" result (rendered PNG) pixels are ok for black and white, but the gray shades are 156/154/156 and 214/215/214.

Note that:

* The "test" result has an abnormal green channel, it's grayscale so it should be identical to the red/blue channels
* The red/blue channels are close, but still off by 1 or 2 from the reference value.
* The reference test is actually wrong (compared to original PNG). This looks like a cross-platform issue, because I generated the test on OS X. I'll file a separate bug on that issue.

Comment 2

[Justin Dolske [:Dolske]]

Attachment: Results with basn0g02.png

Comment 3

[Justin Dolske [:Dolske]]

I noticed that reftests support conditional-fails, so I've reenabled these reftests on non-Windows platforms via entries like:

fails-if(MOZ_WIDGET_TOOLKIT=="windows") == z00n2c08.png z00n2c08.html

Comment 4

[Stuart Parmenter]

tor, glenn: any idea what is going on here?

Comment 5

[Glenn Randers-Pehrson]

I can't see the problem described in the "description" comment
when I look at the original file,
http://www.schaik.com/pngsuite/pngsuite_zlb_png.html
with FF3.0beta1, color_management disabled, WinXP.

Note, the problem described in comment #1 was spun off to another bug and found to be invalid.

Comment 6

[Justin Dolske [:Dolske]]

I've got almost all of the pngsuite images checked in as reftests now [http://mxr.mozilla.org/mozilla/source/modules/libpr0n/test/reftest/], and almost all of them are marked with "fails-if(...)" as in comment 3. Since the reftests still check to see if they actually fail, that implies this problem is happening with all the tests marked as such (otherwise reftest would complain about an unexpected pass).

The only images which pass are:

pngsuite-basic-i/basi0g01.png
pngsuite-basic-i/basi3p02.png
pngsuite-basic-n/basn0g01.png
pngsuite-basic-n/basn3p02.png
pngsuite-oddsizes/s01i3p01.png
pngsuite-oddsizes/s01n3p01.png
pngsuite-oddsizes/s02i3p01.png
pngsuite-oddsizes/s02n3p01.png

These all seem to have R/G/B values of 0 or 255.


(In reply to comment #5)

> Note, the problem described in comment #1 was spun off to another bug and
> found to be invalid.

Just to be pedantic, only the last bullet-item in comment 1 was the what the invalid bug 405398 was about. The R/G/B channels being off by 1 or 2 is still a legit issue for this bug.

Comment 7

[Glenn Randers-Pehrson]

Attachment: simple test image, monocolor rgb-000-123-132

The attached is an 8x8 png, all pixels colored rgb-000-123-132.  How does it fare as a reftest?

Comment 8

[Glenn Randers-Pehrson]

I got correct results when I right-clicked the image, selected "copy image", and pasted it into Paint.  That is, the result was an 8x8 square of rgb(0,123,132) on a larger white background.  When I enabled gfx.color_management, the square was rgb(0,114,122).  When I used "save image" the results were correct.  It appears that the "copy image" operation is applying an unwanted gamma correction.

Comment 9

[Justin Dolske [:Dolske]]

I finally get around to looking at this again, with a local build running on XP in VMWare... I ran the reftests for pngsuite-basic-n and pngsuite-zlib, and didn't see the color differences [which is to say, all the tests marked fails-if(MOZ_WIDGET_TOOLKIT=="windows") reported "unexpected pass" errors because the test and reference were actually identical.].

I also loaded up basn0g02.png in a nightly and looked at the pixels values from a screenshot, and didn't see any anomalies. This, along with Glenn's check in comment #5, would suggest that whatever is causing this is unique to the Windows unit test box.

While it's good to know this probably isn't a code problem, having the problem only exist on the test box makes this rather hard to debug. :-(

Comment 10

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Something that's worth trying once you can reproduce it is to run with MOZ_DISABLE_IMAGE_OPTIMIZE=1 in the environment; that way we'll always just use raw image surfaces and never try to create DDBs which might potentially get munged by the display driver.

Comment 11

[Justin Dolske [:Dolske]]

Ok, I can reproduce this with a build I did on a clone of qm-winxp01, so I'll try to narrow down the cause now...

Comment 12

[Justin Dolske [:Dolske]]

The tests work as expected with MOZ_DISABLE_IMAGE_OPTIMIZE=1, so that confirms that the color wonkyness isn't in Mozilla code.

I see that the VM itself is running in 16-bit color mode, which I'd strongly suspect is what's causing this problem. The Windows control panel doesn't seem to let me use anything other than  1024x768/16-bit. Is this some limitation of the VM or VMWare host? Maybe it just needs a driver updated/installed? Right now it's a "VMWare SVGA II" driver, version 11.2.0.0.

Comment 13

[Dave Miller [:justdave]]

I just logged into the console and looked, and it's set to 800x600/32-bit already.  I just changed it to 1024x768/32-bit and seems to be working fine.  Sure it's not the VNC server limiting it to 16-bit?

Comment 14

[Justin Dolske [:Dolske]]

Ah-ha, got it working in 24-bit mode! Google turned up the following:

1. Run GPEDIT.MSC;
2. Go to: Local Computer Policy/Computer Configuration/Administrative Templates/
            Windows Components/Terminal Services;
3. Open the policy 'Limit maximum colour depth';
4. Set this to 'Disabled' or 'Enabled and 24 bit' or 'Enabled and Client 
   Compatible'.

Step 4 with "Disabled" didn't work, but Enabled/24bit did.

Logged back in, confirmed 24-bit mode from the Windows display properties control panel, and now the reftests work (REFTEST UNEXPECTED PASS). Yay!

I'll run through a full reftest/mochitest run, and see if anything is expecting the broken 16-bit behavour.

Comment 15

[Justin Dolske [:Dolske]]

Attachment: Patch to reenable reftests v.1

This reenables the pngsuite reftests on Windows, by removing all the |fails-if(MOZ_WIDGET_TOOLKIT=="windows")| conditionals. After setting the color depth to 24-bit on the VM clone and applying this patch, the reftests successfully run.

Comment 16

[Chris Cooper [:coop] (he/him)]

Color depth has been changed on qm-win2k3-01. Land at will.

Comment 17

[Justin Dolske [:Dolske]]

Landed  Patch to reenable reftests v.1  (attachment 300195 [details] [diff] [review]).

Comment 18

[Stuart Parmenter]

had to disable again.  doesn't look like the color depth change survived a reboot.

Comment 19

[Justin Dolske [:Dolske]]

Aaaaaaaand relanded. Again. IT did a unrelated reboot, and the colordepth was 24 again. So, the last problem was probably due to a 16-bit RDP client.

Lest this problem happen AGAIN (and, *sigh*, it probably will), let's track the problem over in bug 414720 and leave this one closed.

Comment 20

[Timothy Nikkel (:tnikkel)]

The reftest tests in modules/libpr0n/test/reftest/pngsuite-background/ are only enabled on Mac, was this bug supposed to enable them on Windows too?

Comment 21

[Justin Dolske [:Dolske]]

I think those were problematic because the blending worked differently on different platforms, so OS X was the only one expected to pass. You could enable them and see.

Comment 22

[Timothy Nikkel (:tnikkel)]

Actually, looking at the build logs from tinderbox the tests are run and still fail on Linux and Windows.