Last Comment Bug 411626 - Need regression tests for JPEG image decoding
: Need regression tests for JPEG image decoding
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: ImageLib (show other bugs)
: Trunk
: All All
: -- enhancement with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Milan Sreckovic [:milan]
Mentors:
Depends on: 797042
Blocks: 44781
  Show dependency treegraph
 
Reported: 2008-01-09 21:36 PST by Justin Dolske [:Dolske]
Modified: 2012-10-02 17:15 PDT (History)
13 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Basic test set (30.00 KB, application/x-tar)
2008-02-14 18:46 PST, Justin Dolske [:Dolske]
no flags Details
[checked in] Basic test set v.2 (110.00 KB, application/x-tar)
2008-02-14 21:53 PST, Justin Dolske [:Dolske]
no flags Details

Description Justin Dolske [:Dolske] 2008-01-09 21:36:06 PST
Spun off from bug 386651... We should have some tests to ensure that various flavors/options of JPEG are correctly decoded.

I looked around, and can't find any kind of existing JPEG test suite. So, we should probably just one from scratch. Things to test might include:
* Simple, baseline JPEG
* Progressive JPEG
* Different compression levels (this is mostly an encoder thing, not sure if
  it really ends up exercising the decoder differently)
* Inclusion of metadata (EXIF, thumbnail)
* Inclusion of a color profile
* 12-bit JPEG (unsupported? but don't barf on it.)
* CMYK JPEG (bug 44781)

Previously fixed JPEG bugs may have interesting images for a test suite as well.
Comment 1 Justin Dolske [:Dolske] 2008-02-14 18:46:15 PST
Created attachment 303442 [details]
Basic test set

Here's a set of basic test images...
* Sizes (1x1 to 33x33)
* Progressive
* grayscale
* cmyk

Just need to reftest-ize them with PNG references.
Comment 2 Justin Dolske [:Dolske] 2008-02-14 21:53:57 PST
Created attachment 303464 [details]
[checked in] Basic test set v.2

Corrected some test images, and added PNG references.
Comment 3 Justin Dolske [:Dolske] 2008-02-14 23:10:44 PST
Bah...

These reftests all pass on OS X and Solaris, but on Windows most of them fail with slight color differences. This isn't the same problem as bug 405384, so I'm guessing it could be due to the SSE2 stuff (ala bug 411379)? I don't think that's wrong (iirc the JPEG spec even explicitly allows some variation in decoding), but it makes life more difficult for reftests.

Example: With the jpeg-size-2x2.jpg, the values are...

          Reference PNG                           Test JPG
  rgb(179,179,93)  rgb(143,143,57)     rgb(178,178,92) rgb(143,143,57)
  rgb(106,106,20)  rgb( 80, 80, 0)     rgb(106,106,20) rgb( 80, 80, 0)

Note the slight difference in the upper-left pixel. The 1x1, 3x3, 4x4, 6x6, and 8x8 images are the only ones which pass on Windows.

For now, I'll probably land these as-is, but marked as fails-if() on Windows [as was done in bug 405384 while those issues were worked out]. Assuming the color differences are expected, we can then either (1) land reference images specific to Windows [and mark them as failing on other platforms], or (2) fix the general problem by adding some capability to reftest to allow for fuzzy comparisons. This would probably be useful in other places as well. For example, the canvas tests added in bug 407049 have a number of tests that pass if the actual value matches the expected value +/- a certain range.
Comment 4 Michael Moy 2008-02-15 03:46:00 PST
Regarding bug 411379, the SSE2 color processing hasn't been added yet. The only color-related code in 411379 is in using a static table for the color tables instead of computing them for every image.

Do you have a date when the tests started failing?
Comment 5 Justin Dolske [:Dolske] 2008-02-15 16:23:12 PST
Oh. Hmm. I haven't been following that closely, and thought it was already in.

I put up a testcase at http://people.mozilla.com/~dolske/tmp/testjpeg.html, which reports any differences between the 16x16 JPEG and PNG used in the reftests. With a current trunk nightly on Windows, I get:

Pixel (14,9) differs.
 jpg: rgb(21,102,132)
 png: rgb(21,102,131)
Pixel (14,10) differs.
 jpg: rgb(20,85,149)
 png: rgb(20,86,147)
Pixel (15,10) differs.
 jpg: rgb(10,84,157)
 png: rgb(10,84,155)
Pixel (13,11) differs.
 jpg: rgb(37,70,149)
 png: rgb(37,70,147)
Pixel (14,11) differs.
 jpg: rgb(21,69,167)
 png: rgb(21,70,165)
Test complete. Images are NOT identical.

Could you see if you get different results with your SSE2 patch? We could also check to see if this used to work in the past, but if the SSE2 patch is going to change things anyway, that might not really matter.
Comment 6 Steve England [:stevee] 2008-02-28 16:22:00 PST
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4pre) Gecko/2008022809 Minefield/3.0b4pre

With this build and a recent pull of the tree, I ran the reftests and got these results:

REFTEST PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-1x1.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-2x2.jpg
REFTEST PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-3x3.jpg
REFTEST PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-4x4.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-5x5.jpg
REFTEST PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-6x6.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-7x7.jpg
REFTEST PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-8x8.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-9x9.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-15x15.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-16x16.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-17x17.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-31x31.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-32x32.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-size-33x33.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-progressive.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-gray.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-cmyk-1.jpg
REFTEST UNEXPECTED PASS: file:///D:/moz/mozilla/modules/libpr0n/test/reftest/jpeg/jpg-cmyk-2.jpg

WinXP SP2, 1280x1024x32 @ 85Hz, GeForce 6800GT, Forceware 169.21 drivers, AGPx4, 256MB Video memory, video BIOS 5.40.02.15.01. AthlonXP2400, 1GB DDR RAM.
Comment 7 Matt Woodrow (:mattwoodrow) 2011-12-08 18:48:48 PST
https://hg.mozilla.org/integration/mozilla-inbound/rev/ed4957b534ea
Comment 8 Bobby Holley (:bholley) (busy with Stylo) 2011-12-08 18:53:05 PST
Woo close-to-4-year-old patches!

But we shouldn't resolved fixed until this lands on central, no?
Comment 9 Matt Woodrow (:mattwoodrow) 2011-12-08 19:03:08 PST
Oops, used the wrong tree for my link.

https://hg.mozilla.org/mozilla-central/rev/ed4957b534ea

This landed in 2008 :)

Note You need to log in before you can comment on or make changes to this bug.