Closed Bug 373610 (refdyn) Opened 16 years ago Closed 10 months ago
[meta] Bugs found by comparing renderings with and without dynamic changes
This reftest-based script exhaustively tests how rendering responds to simple DOM changes. For example, it tries removing a node, forcing a complete relayout of the page, and putting the node back. It compares the two screenshots with the node missing to ensure that the removeChild operation is handled properly, and compares the two screenshots with the node present to ensure that the insertBefore operation is handled properly. It tests removeChild/insertBefore for every node in the document. It also tests attribute removal/setting and text node changes. Running it on all the reftest files (excluding the pixel-rounding directory) takes about 10 hours on my MacBook Pro, so it's too slow to be run automatically after every checkin. About 80% of the time is spent in PNG compression. Is there a way to avoid PNG compression in this kind of test? It has found 15 bugs so far, mostly incorrect-rendering bugs and a few bugs in reading the style attribute. But I'm filing this as security-sensitive for now because I'm worried it might find more severe bugs on branches and because my experience with bug 349611 makes me feel I should be cautious.
This is a great idea. It seems to me, a lot of bugs could also be found by comparing the offsetWidth/offsetHeight/computed width/computed height, not? In that case, you could avoid the slow image comparison.
Jesse, even if it takes 10 hours, we could (and probably should) still run the test nightly if we had appropriate automation. cc'ing pav/vlad about avoiding PNG compression. It probably wouldn't be that hard to add a parameter to avoid compression.
cool idea, Jesse. We should figure out how to get this running regularly on the test farm. I'll talk to you later about some ideas on irc.
The best way to avoid the PNG compression would be to write a small image comparison function in C++ rather than doing the image comparison via PNG data: URLs. It's something that's on my mental todo list for speeding up reftest, though there should probably be a bug filed on it. (We probably would still want to convert to data: URLs when reporting failures, but that's rare.)
Can the image comparison be done with getImageData and in JS? Not sure what the perf would be like, but something to consider would be to use a much smaller basea size for the tests, or at least have a test flag that indicates that a test needs a large canvas vs. the default (which should be small, like 100x100). A C++ thing could work fine as well.
I noticed in reftest.js that setAttribute is explicitly called on the Canvas to set its height and width. Does this necessarily need to be set to the size of the window, or can it be set to capture only a small square of the window? Or, can the width of the window be set here If this size could be made a parameter of the test, it would be most convenient.
Now much faster thanks to bug 387132. Also contains updated exclusions.
Attachment #270363 - Attachment is obsolete: true
Updated to understand the current reftest.list format. Many new exclusions.
Attachment #283658 - Attachment is obsolete: true
The bugs that force me to exclude the most tests are: * Bug 162063 - Tables * Bug 409089 - -moz-box * Bug 409125 - MathML * Bug 229915 - CSS + combinator * Bug 373298 - :-moz-first-node * Bug 145419 - dynamically added ::first-letter and ::first-line rules
After finding dozens of harmless layout inconsistencies, refdyn finally found a potential security hole! Bug 421234 causes a random character to appear, and I'm guessing it comes from uninitialized memory.
Refdyn has only found two security holes among about a hundred total bugs. I'd like to make it public eventually, but one of those bugs (bug 467323) isn't fixed on 3.0.x, so it might be a while.
Attachment #307930 - Attachment is obsolete: true
I've revived this tool by making it part of the DOM fuzzer. Now it makes a number of random dynamic changes before checking that the resulting dynamic rendering matches the static rendering of the same DOM tree.
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [sg:nse meta]
Assignee: jruderman → nobody
Summary: Bugs found by comparing renderings with and without dynamic changes → [meta] Bugs found by comparing renderings with and without dynamic changes
Status: NEW → RESOLVED
Closed: 10 months ago
Flags: needinfo?(choller) → needinfo?(twsmith)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.