466875 - CMS Testing Framework Needs updating

Status: NEW

(Core :: Graphics: Color Management, defect)

Description

[Bobby Holley (:bholley)]

The standalone CMS testing framework needs some updating to deal with new requirements imposed on it.

At minimum, we need the following:

1-Support for tests of different types, rather than just CRC verification tests (We need these to test things like bug 460629).
2-Support for shorter test runs so that we can plug the framework into the tinderboxen without wasting 7 minutes of everyone's time on each checkin.
3-Support for less ad-hoc statistics and for tracking errors all the way through the execution of the tests.
4-The tests need to be plugged into the build.

We then want to add tests for:
1-bug 460629
2-bug 458164

I've got a baseline patch that covers most of the framework changes except for plugging things in to be run automatically. I'll then put together patches for the new test cases we want to cover.

Comment 1

[Bobby Holley (:bholley)]

Attachment: Baseline overhaul patch to cover most of the infrastructural changes

Added the baseline overhaul patch

Comment 2

[Bobby Holley (:bholley)]

Also note that I forgot to mention that one of the specific test configurations was causing 4 off-by-2 issues between the fixed path and the floating point path. Per se there's nothing incorrect about this, and it's always been like this, but it required the person running the tests to know what should happen and look for it. Since we want the tests to be more automatic, and we can count on having no off-by-two issues otherwise (which gives us a great deal of test coverage), we just remove that configuration.

Comment 3

[Bobby Holley (:bholley)]

Ideally we'd also like to add tests for bug 460520.

Comment 4

[Joe Drew (not getting mail)]

Comment on attachment 350192 [details] [diff] [review]
Baseline overhaul patch to cover most of the infrastructural changes

> # Color Management Test Directive File
> #
> # Format:
> # InputProfileFilename OutputProfileFilename <CRC32 For Each Intent>
> #

Looks like you've changed the format - might want to change the documentation too :)

r+ with that change.

Comment 5

[Joe Drew (not getting mail)]

So, we should maybe update this for QCMS?

Comment 6

[Bobby Holley (:bholley)]

yikes, I'd more or less forgotten about this.

Jeff - this is more or less black-box testing, so it shouldn't be too hard to update for qcms. Let me know what you think.

Comment 7

[Joe Drew (not getting mail)]

Comment on attachment 350192 [details] [diff] [review]
Baseline overhaul patch to cover most of the infrastructural changes

I guess this is way out of date now. Luckily we don't need sr on it anymore!

Comment 8

[Bobby Holley (:bholley)]

reassigning to jeff.