Bug 373610 (refdyn)

Bugs found by comparing renderings with and without dynamic changes

NEW
Assigned to

Status

()

defect
12 years ago
2 years ago

People

(Reporter: jruderman, Assigned: jruderman)

Tracking

(Depends on 61 bugs, {meta})

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 obsolete attachments)

(Assignee)

Description

12 years ago
This reftest-based script exhaustively tests how rendering responds to simple DOM changes.  For example, it tries removing a node, forcing a complete relayout of the page, and putting the node back.    It compares the two screenshots with the node missing to ensure that the removeChild operation is handled properly, and compares the two screenshots with the node present to ensure that the insertBefore operation is handled properly.

It tests removeChild/insertBefore for every node in the document.  It also tests attribute removal/setting and text node changes.

Running it on all the reftest files (excluding the pixel-rounding directory) takes about 10 hours on my MacBook Pro, so it's too slow to be run automatically after every checkin.  About 80% of the time is spent in PNG compression.  Is there a way to avoid PNG compression in this kind of test?

It has found 15 bugs so far, mostly incorrect-rendering bugs and a few bugs in reading the style attribute.  But I'm filing this as security-sensitive for now because I'm worried it might find more severe bugs on branches and because my experience with bug 349611 makes me feel I should be cautious.
(Assignee)

Comment 1

12 years ago
Posted file refdyn.js (obsolete) —
(Assignee)

Updated

12 years ago
Whiteboard: [sg:nse meta]
This is a great idea.
It seems to me, a lot of bugs could also be found by comparing the offsetWidth/offsetHeight/computed width/computed height, not?
In that case, you could avoid the slow image comparison.

Comment 3

12 years ago
Jesse, even if it takes 10 hours, we could (and probably should) still run the test nightly if we had appropriate automation. cc'ing pav/vlad about avoiding PNG compression. It probably wouldn't be that hard to add a parameter to avoid compression.
cool idea, Jesse. We should figure out how to get this running regularly on the test farm. I'll talk to you later about some ideas on irc.
The best way to avoid the PNG compression would be to write a small image comparison function in C++ rather than doing the image comparison via PNG data: URLs.  It's something that's on my mental todo list for speeding up reftest, though there should probably be a bug filed on it.  (We probably would still want to convert to data: URLs when reporting failures, but that's rare.)
Can the image comparison be done with getImageData and in JS?  Not sure what the perf would be like, but something to consider would be to use a much smaller basea size for the tests, or at least have a test flag that indicates that a test needs a large canvas vs. the default (which should be small, like 100x100).  A C++ thing could work fine as well.

Comment 7

12 years ago
I noticed in reftest.js that setAttribute is explicitly called on the Canvas to set its height and width. Does this necessarily need to be set to the size of the window, or can it be set to capture only a small square of the window? Or, can the width of the window be set here

If this size could be made a parameter of the test, it would be most convenient.
(Assignee)

Updated

12 years ago
Alias: refdyn
(Assignee)

Updated

12 years ago
Depends on: 375493
(Assignee)

Updated

12 years ago
Depends on: 375497
(Assignee)

Updated

12 years ago
Depends on: 381277
(Assignee)

Updated

12 years ago
Depends on: 381279
(Assignee)

Updated

12 years ago
Depends on: 382146
(Assignee)

Updated

12 years ago
Depends on: 385607
(Assignee)

Comment 8

12 years ago
Posted file refdyn.js (obsolete) —
Attachment #258275 - Attachment is obsolete: true
(Assignee)

Updated

12 years ago
Depends on: 387510
(Assignee)

Updated

12 years ago
Depends on: 395125
(Assignee)

Updated

12 years ago
Depends on: 395130
(Assignee)

Updated

12 years ago
Depends on: 377519
(Assignee)

Updated

12 years ago
Depends on: 395155
(Assignee)

Updated

12 years ago
Depends on: 398092
(Assignee)

Updated

12 years ago
Depends on: 398095
(Assignee)

Updated

12 years ago
Depends on: 398101
(Assignee)

Updated

12 years ago
Depends on: 398105
(Assignee)

Comment 9

12 years ago
Posted file refdyn.zip (obsolete) —
Now much faster thanks to bug 387132.  Also contains updated exclusions.
Attachment #270363 - Attachment is obsolete: true
(Assignee)

Updated

12 years ago
Depends on: 398681
(Assignee)

Updated

12 years ago
Depends on: 398682
(Assignee)

Updated

12 years ago
Depends on: 398685
(Assignee)

Updated

12 years ago
Depends on: 398686
(Assignee)

Updated

12 years ago
Depends on: 398809
(Assignee)

Updated

12 years ago
Depends on: 398820
(Assignee)

Updated

12 years ago
Depends on: 398830
(Assignee)

Updated

12 years ago
Depends on: 398884
(Assignee)

Updated

12 years ago
Depends on: 401580
(Assignee)

Updated

12 years ago
Depends on: 403733
(Assignee)

Updated

12 years ago
Depends on: 405517
(Assignee)

Updated

12 years ago
Depends on: 407095
(Assignee)

Updated

12 years ago
Depends on: 407106
(Assignee)

Updated

12 years ago
Depends on: 407115
(Assignee)

Updated

12 years ago
Depends on: 407397
(Assignee)

Updated

12 years ago
Depends on: 407419
(Assignee)

Updated

12 years ago
Depends on: 408782
(Assignee)

Updated

12 years ago
Depends on: 409051
(Assignee)

Updated

12 years ago
Depends on: 409056
(Assignee)

Updated

12 years ago
Depends on: 409065
(Assignee)

Updated

12 years ago
Depends on: 409089
(Assignee)

Updated

12 years ago
Depends on: 409125
(Assignee)

Updated

12 years ago
Depends on: 409494
(Assignee)

Updated

12 years ago
Depends on: 409577
(Assignee)

Comment 10

12 years ago
Posted file refdyn.zip (obsolete) —
Updated to understand the current reftest.list format.  Many new exclusions.
Attachment #283658 - Attachment is obsolete: true
(Assignee)

Comment 11

12 years ago
The bugs that force me to exclude the most tests are:

* Bug 162063 - Tables
* Bug 409089 - -moz-box
* Bug 409125 - MathML
* Bug 229915 - CSS + combinator
* Bug 373298 - :-moz-first-node 
* Bug 145419 - dynamically added ::first-letter and ::first-line rules
(Assignee)

Updated

11 years ago
Depends on: 411367
(Assignee)

Updated

11 years ago
Depends on: 411374
(Assignee)

Updated

11 years ago
Depends on: 412352
(Assignee)

Updated

11 years ago
Depends on: 412365
(Assignee)

Updated

11 years ago
Depends on: 418574
(Assignee)

Updated

11 years ago
Depends on: 418756
(Assignee)

Updated

11 years ago
Depends on: 418766
(Assignee)

Updated

11 years ago
Depends on: 421234
(Assignee)

Updated

11 years ago
Depends on: 421239
(Assignee)

Updated

11 years ago
Depends on: 421419
(Assignee)

Updated

11 years ago
Depends on: 421425
(Assignee)

Comment 12

11 years ago
After finding dozens of harmless layout inconsistencies, refdyn finally found a potential security hole!  Bug 421234 causes a random character to appear, and I'm guessing it comes from uninitialized memory.
(Assignee)

Comment 13

11 years ago
Posted file refdyn.zip (obsolete) —
Attachment #294382 - Attachment is obsolete: true
(Assignee)

Updated

11 years ago
Depends on: 423130
(Assignee)

Updated

11 years ago
Depends on: 429974
(Assignee)

Updated

11 years ago
Depends on: 429976
(Assignee)

Updated

11 years ago
Depends on: 442630
(Assignee)

Updated

11 years ago
Depends on: 442633
(Assignee)

Updated

11 years ago
Depends on: 450693
(Assignee)

Updated

11 years ago
Depends on: 467312
(Assignee)

Updated

11 years ago
Depends on: 467321
(Assignee)

Updated

11 years ago
Depends on: 467323
(Assignee)

Updated

11 years ago
Depends on: 467460
(Assignee)

Updated

11 years ago
Depends on: 467472
(Assignee)

Updated

11 years ago
Depends on: 467498
(Assignee)

Updated

11 years ago
Depends on: 467719
(Assignee)

Updated

11 years ago
Depends on: 467722
(Assignee)

Updated

11 years ago
Depends on: 467723
(Assignee)

Updated

10 years ago
Depends on: 475642
(Assignee)

Updated

10 years ago
Depends on: 475644
(Assignee)

Updated

10 years ago
Depends on: 475647
(Assignee)

Updated

10 years ago
Depends on: 478490
(Assignee)

Updated

10 years ago
Depends on: 478511
(Assignee)

Updated

10 years ago
Depends on: 478594
(Assignee)

Updated

10 years ago
Depends on: 489868
(Assignee)

Updated

10 years ago
Depends on: 489877
(Assignee)

Updated

10 years ago
Depends on: 489887
(Assignee)

Updated

10 years ago
Depends on: 489890
(Assignee)

Updated

10 years ago
Depends on: 490173
(Assignee)

Updated

10 years ago
Depends on: 490174
(Assignee)

Updated

10 years ago
Depends on: 490176
(Assignee)

Updated

10 years ago
Depends on: 490177
(Assignee)

Updated

10 years ago
Depends on: 490182
(Assignee)

Updated

10 years ago
Depends on: 490183
(Assignee)

Updated

10 years ago
Depends on: 490185
(Assignee)

Updated

10 years ago
Depends on: 490216
(Assignee)

Updated

10 years ago
Depends on: 490218
(Assignee)

Updated

10 years ago
Depends on: 490220
(Assignee)

Updated

10 years ago
Depends on: 492231
(Assignee)

Updated

10 years ago
Depends on: 492239
(Assignee)

Updated

10 years ago
Depends on: 492240
(Assignee)

Updated

10 years ago
Depends on: 492661
(Assignee)

Comment 14

10 years ago
Refdyn has only found two security holes among about a hundred total bugs.  I'd like to make it public eventually, but one of those bugs (bug 467323) isn't fixed on 3.0.x, so it might be a while.
Attachment #307930 - Attachment is obsolete: true
(Assignee)

Updated

10 years ago
Depends on: 493564
(Assignee)

Updated

10 years ago
Depends on: 501035
(Assignee)

Updated

10 years ago
Depends on: 501037
(Assignee)

Updated

10 years ago
Depends on: 501048
(Assignee)

Updated

10 years ago
Depends on: 501049
(Assignee)

Updated

10 years ago
Depends on: 521525
(Assignee)

Updated

10 years ago
Depends on: 521527
(Assignee)

Updated

10 years ago
Depends on: 521539
(Assignee)

Updated

10 years ago
Depends on: 521542
(Assignee)

Updated

10 years ago
Depends on: 521594
(Assignee)

Updated

10 years ago
Depends on: 521600
(Assignee)

Updated

10 years ago
Depends on: 521602
(Assignee)

Updated

10 years ago
Depends on: 521607
(Assignee)

Updated

10 years ago
Depends on: 521609
(Assignee)

Updated

10 years ago
Depends on: 521682
(Assignee)

Updated

10 years ago
Depends on: 521685
(Assignee)

Updated

10 years ago
Depends on: 521689
(Assignee)

Updated

10 years ago
Depends on: 521720
(Assignee)

Updated

10 years ago
Depends on: 521875
(Assignee)

Updated

10 years ago
Depends on: 522390
(Assignee)

Updated

10 years ago
Depends on: 522393
(Assignee)

Updated

10 years ago
Depends on: 526536
(Assignee)

Updated

10 years ago
Depends on: 526596
(Assignee)

Updated

10 years ago
Depends on: 526602
(Assignee)

Updated

10 years ago
Depends on: 526634
(Assignee)

Updated

10 years ago
Depends on: 534526
(Assignee)

Updated

10 years ago
Depends on: 534793
(Assignee)

Updated

10 years ago
Depends on: 534802
(Assignee)

Updated

10 years ago
Depends on: 534806
(Assignee)

Updated

10 years ago
Depends on: 534808
(Assignee)

Updated

10 years ago
Depends on: 534811
(Assignee)

Updated

9 years ago
Depends on: 537875
(Assignee)

Updated

9 years ago
Depends on: 543791
(Assignee)

Updated

9 years ago
Depends on: 549797
(Assignee)

Updated

9 years ago
Depends on: 550047
(Assignee)

Updated

9 years ago
Depends on: 550065
(Assignee)

Updated

9 years ago
Depends on: 550661
(Assignee)

Updated

9 years ago
Depends on: 551239
(Assignee)

Updated

9 years ago
Depends on: 551838
(Assignee)

Comment 15

7 years ago
I've revived this tool by making it part of the DOM fuzzer. Now it makes a number of random dynamic changes before checking that the resulting dynamic rendering matches the static rendering of the same DOM tree.
(Assignee)

Updated

7 years ago
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [sg:nse meta]
(Assignee)

Updated

7 years ago
Depends on: 726548
(Assignee)

Updated

7 years ago
Depends on: 728100
(Assignee)

Updated

7 years ago
Depends on: 731521
(Assignee)

Updated

7 years ago
Depends on: 732740
(Assignee)

Updated

7 years ago
Depends on: 752779
(Assignee)

Updated

7 years ago
Depends on: 763560
(Assignee)

Updated

7 years ago
Depends on: 764256
(Assignee)

Updated

7 years ago
Depends on: 767233
(Assignee)

Updated

7 years ago
Depends on: 767279
(Assignee)

Updated

5 years ago
Depends on: 1036750
(Assignee)

Updated

4 years ago
Depends on: 1156244
(Assignee)

Updated

3 years ago
Depends on: 1258076

Updated

3 years ago
Component: Tracking → Platform Fuzzing Team
You need to log in before you can comment on or make changes to this bug.