Last Comment Bug 465479 - (js-differential-test) Bugs found by JS differential testing (comparing output with different JIT options)
(js-differential-test)
: Bugs found by JS differential testing (comparing output with different JIT op...
Status: NEW
: meta
Product: Core
Classification: Components
Component: Platform Fuzzing Team (show other bugs)
: Trunk
: x86 Mac OS X
-- normal with 5 votes (vote)
: ---
Assigned To: Jesse Ruderman
: chris hofmann
: Al Billings [:abillings]
Mentors:
Depends on: 948321 1027846 1085299 1271850 1293575 465424 465443 465453 465454 465460 465472 465473 465483 465484 465567 465605 465686 465688 465901 465902 465915 466076 466128 466262 466781 466787 466905 468711 469237 469239 469927 469938 469942 469943 470128 470129 470133 470139 470143 470144 470173 470176 470187 470300 470735 470736 470737 470738 470739 470779 470959 470964 471701 472941 473014 474639 474769 474835 474935 474951 475658 476118 489682 491965 491973 491989 493662 495563 495566 495773 495897 495907 495958 495962 496185 496245 496251 496270 496531 496532 496541 496544 496687 496784 496813 496816 496867 496922 496987 497015 498137 498236 498237 498549 498555 501275 502768 502777 503488 503648 503818 504516 504520 505003 507556 507678 508187 566637 566639 584587 584594 584603 584605 584647 585260 586547 593554 593556 601454 617288 620761 620902 621376 621377 621418 621464 622271 624377 627685 646255 647524 647695 648708 649339 651827 652414 655699 656228 656229 657193 657986 658539 660437 660438 673954 690292 699201 706710 712379 713957 715387 715400 718076 735161 735316 740595 743423 743425 755813 887521 887542 887544 887549 887556 891775 892787 906284 906285 906286 908608 908813 909601 909602 910012 911369 912303 912304 912316 912328 913749 913885 914341 916039 923765 937550 937922 939868 940635 940642 940864 941381 942390 942549 942550 944153 944266 944975 945512 946679 946969 958381 969203 969705 973118 975138 975636 981325 983840 995673 995675 995679 995816 995817 995826 996881 996895 997546 998059 998262 998580 998709 999790 999849 999857 1000605 1000606 1006910 1007213 1008818 1015656 1022948 1024444 1025587 1027359 1033873 1033946 1034280 1037665 1053074 1054531 1054541 1054545 1054568 1066496 1073910 1073928 1076091 1076283 1077074 1079062 1081850 1085298 1090424 1103032 1103048 1105574 1122338 1122344 1122401 1122402 1122403 1122839 1123011 1124421 1124448 1124485 1126066 1129088 1130679 1132290 1132396 1133389 1135047 1136542 1137610 1137616 1137624 1137998 1138740 1140890 1143878 1147662 1148973 1149739 1153153 1154971 1159899 1181354 1181828 1183423 1186226 1186271 1203862 1204675 1206265 1207449 1210596 1211100 1213552 1220275 1227287 1228397 1234736 1236114 1237403 1237464 1237564 1238461 1239075 1244502 1245187 1245627 1246200 1246552 1247701 1247862 1247863 1247871 1247877 1247880 1248153 1250863 1253898 1263525 1263811 1264561 1265159 1266242 1268224 1268955 1273267 1274429 1277118 1280252 1286407 1293542 1296243 1301208 1304638 1304640 1304641 1304643 1304649 1308743 1308802 1312620 1314438 1314545 1316830 1317943 1319242 1321437 1330234
Blocks: fuzz
  Show dependency treegraph
 
Reported: 2008-11-17 23:10 PST by Jesse Ruderman
Modified: 2017-01-10 23:58 PST (History)
14 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description User image Jesse Ruderman 2008-11-17 23:10:11 PST
Like arithfuzz (tracked in bug 465274), this fuzzer compares JIT results to interpreter results.  But this one compares the output of entire scripts, not just expressions inside loops.  It expects stdout and stderr to be exactly the same, except for the jitstats at the end of stdout.

This fuzzer is a lot slower than arithfuzz and jsfunfuzz, mostly because it keeps restarting ./js.  It tests more JS constructs than arithfuzz, but fewer than jsfunfuzz (which does no correctness testing except for decompiler and uneval).

In theory, this fuzzer could compare the behavior of two completely different JS engines if it were lenient about differences in error messages.  But I'm currently only testing "js" against "js -j".
Comment 1 User image Jesse Ruderman 2015-07-27 02:22:05 PDT
Differential testing is now integrated with jsfunfuzz.

A random set of flags is chosen when running jsfunfuzz:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/shellFlags.py

The output is compared against running the same shell with no special flags:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/compareJIT.py

We also check that poking the garbage collector never affects output:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/shared/testing-functions.js

Differential testing excludes Date, Math.random, and a few other things:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/avoid-known-bugs.js#L18

Note You need to log in before you can comment on or make changes to this bug.