Open Bug 1415558 (VisFuzz) Opened 4 years ago Updated 3 years ago
Develop a fuzzer to test for visual invalidation issues (Vis
No description provided.
Roughly speaking, this version of VisFuzz waits for reftest to get to the stage where it would normally be ready to render a test's output (after reftest-wait is removed, for example), applies a stack of randomized DOM/style changes, reverts those changes, and then checks that the rendering still matches the reference.
The current (still more clunky than I'd like) steps to use VisFuzz are as follows: * Apply the patch to m-c, rebuild layout/tools/reftest and then use `mach` to run some tests (possibly by pushing to Try). * Open the reftest analyzer and compare failing tests to their references. Currently there are still a lot of failures due to very small anti aliasing issues, so ignore that type of failure and look for failures with more significant rendering issues. * For any test of interest, find the corresponding "Fuzzing M0: ..." to "Fuzzing R0: ..." lines in the log output and paste those lines into the processing tool attached to this bug. That tool will output the lines of script that were used to apply and then revert the changes that the fuzzer made. The first half of the lines consist of one line per fuzzer change, and the second half are the lines to revert each change in reverse order (since the changes are applied as a stack they need to be applied in reverse order or else the referenced nodes may be wrong due to DOM tree changes made by other mutations). * Open the failing test in Firefox locally and open the devtools console. * Paste the first half of the output from the processing tool (everything down to and including the last `let n = ...` line) into the devtools console and hit Enter. (This is all the mutations the fuzzer came up with.) * Do the same with the rest of the output from the processing tool. (This reverts all the mutations that the fuzzer came up with.) * If the visual failure reproduces, then try reducing the number of mutate/restore lines (remembering that they are matching pairs). Initially it's best to work from the outside in or vice versa. * If the failure looks like a real failure (i.e. not a deficiency of the fuzzer) file a bug and link it to bug 1415131.
Most of the code is now separated out into a module to keep the general reftest code clean.
Attachment #8926390 - Attachment is obsolete: true
Comment on attachment 8932967 [details] [diff] [review] patch - First pass at implementing a visual fuzzer Daniel, what do you think about landing this sort of approach in the tree to make it easier for others to use?
Attachment #8932967 - Flags: review?(dholbert)
(In reply to Daniel Holbert [:dholbert] (recovering from vacation bugmail backlog) from comment #7) > Out of curiosity, is there a reason we're still using the (Sorry, disregard this ^^ unfinished review-comment. I started out asking a question here and then ended up asking the question elsewhere and forgot to clean this one up. [question was RE why we're comparing against reference case, as opposed to the unmutated-testcase.)
(In reply to Jonathan Watt [:jwatt] (needinfo? me) from comment #6) > Daniel, what do you think about landing this sort of approach in the tree to > make it easier for others to use? Answering this^ more meta-question: I think the general approach seems reasonable to land -- seems like a sort of DOM-mutation-analogue to "chaos mode" (which also lives in-tree). dbaron owns the reftest harness, though, so it'd probably be worth looping him in too, since this is a nontrivial extension to the harness's capabilities. (Nice work on modularizing it, though - that makes it easier to reason about independently.)
Comment on attachment 8932967 [details] [diff] [review] patch - First pass at implementing a visual fuzzer (No rush, but I'm marking "r-" for now to take this out of my review queue, since I've reviewed it & left some feedback.)
Attachment #8932967 - Flags: review?(dholbert) → review-
You need to log in before you can comment on or make changes to this bug.