Note: There are a few cases of duplicates in user autocompletion which are being worked on.
Bug 465479 (js-differential-test)

Bugs found by JS differential testing (comparing output with different JIT options)

NEW
Assigned to

Status

()

Core
Platform Fuzzing Team
9 years ago
11 days ago

People

(Reporter: Jesse Ruderman, Assigned: Jesse Ruderman)

Tracking

(Depends on: 4 bugs, Blocks: 1 bug, {meta})

Trunk
x86
Mac OS X
Points:
---
948321, 1027846, 1085299, 1293575, 465424, 465443, 465453, 465454, 465460, 465472, 465473, 465483, 465484, 465567, 465605, 465686, 465688, 465901, 465902, 465915, 466076, 466128, 466262, 466781, 466787, 466905, 468711, 469237, 469239, 469927, 469938, 469942, 469943, 470128, 470129, 470133, 470139, 470143, 470144, 470173, 470176, 470187, 470300, 470735, 470736, 470737, 470738, 470739, 470779, 470959, 470964, 471701, 472941, 473014, 474639, 474769, 474835, 474935, 474951, 475658, 476118, 489682, 491965, 491973, 491989, 493662, 495563, 495566, 495773, 495897, 495907, 495958, 495962, 496185, 496245, 496251, 496270, 496531, 496532, 496541, 496544, 496687, 496784, 496813, 496816, 496867, 496922, 496987, 497015, 498137, 498236, 498237, 498549, 498555, 501275, 502768, 502777, 503488, 503648, 503818, 504516, 504520, 505003, 507556, 507678, 508187, 566637, 566639, 584587, 584594, 584603, 584605, 584647, 585260, 586547, 593554, 593556, 601454, 617288, 620761, 620902, 621376, 621377, 621418, 621464, 622271, 624377, 627685, 646255, 647524, 647695, 648708, 649339, 651827, 652414, 655699, 656228, 656229, 657193, 657986, 658539, 660437, 660438, 673954, 690292, 699201, 706710, 712379, 713957, 715387, 715400, 718076, 735161, 735316, 740595, 743423, 743425, 755813, 887521, 887542, 887544, 887549, 887556, 891775, 892787, 906284, 906285, 906286, 908608, 908813, 909601, 909602, 910012, 911369, 912303, 912304, 912316, 912328, 913749, 913885, 914341, 916039, 923765, 937550, 937922, 939868, 940635, 940642, 940864, 941381, 942390, 942549, 942550, 944153, 944266, 944975, 945512, 946679, 946969, 958381, 969203, 969705, 973118, 975138, 975636, 981325, 983840, 995673, 995675, 995679, 995816, 995817, 995826, 996881, 996895, 997546, 998059, 998262, 998580, 998709, 999790, 999849, 999857, 1000605, 1000606, 1006910, 1007213, 1008818, 1015656, 1022948, 1024444, 1025587, 1027359, 1033873, 1033946, 1034280, 1037665, 1053074, 1054531, 1054541, 1054545, 1054568, 1066496, 1073910, 1073928, 1076091, 1076283, 1077074, 1079062, 1081850, 1085298, 1090424, 1103032, 1103048, 1105574, 1122338, 1122344, 1122401, 1122402, 1122403, 1122839, 1123011, 1124421, 1124448, 1124485, 1126066, 1129088, 1130679, 1132290, 1132396, 1133389, 1135047, 1136542, 1137610, 1137616, 1137624, 1137998, 1138740, 1140890, 1143878, 1147662, 1148973, 1149739, 1153153, 1154971, 1159899, 1181354, 1181828, 1183423, 1186226, 1186271, 1203862, 1204675, 1206265, 1207449, 1210596, 1211100, 1213552, 1220275, 1227287, 1228397, 1234736, 1236114, 1237403, 1237464, 1237564, 1238461, 1239075, 1244502, 1245187, 1245627, 1246200, 1246552, 1247701, 1247862, 1247863, 1247871, 1247877, 1247880, 1248153, 1250863, 1253898, 1263525, 1263811, 1264561, 1265159, 1266242, 1268224, 1268955, 1271850, 1273267, 1274429, 1277118, 1280252, 1286407, 1293542, 1296243, 1301208, 1304638, 1304640, 1304641, 1304643, 1304649, 1308743, 1308802, 1312620, 1314438, 1314545, 1316830, 1317943, 1319242, 1321437, 1330234, 1345707, 1368573, 1368574, 1368575, 1368576, 1368584, 1372956, 1377141, 1379936
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

9 years ago
Like arithfuzz (tracked in bug 465274), this fuzzer compares JIT results to interpreter results.  But this one compares the output of entire scripts, not just expressions inside loops.  It expects stdout and stderr to be exactly the same, except for the jitstats at the end of stdout.

This fuzzer is a lot slower than arithfuzz and jsfunfuzz, mostly because it keeps restarting ./js.  It tests more JS constructs than arithfuzz, but fewer than jsfunfuzz (which does no correctness testing except for decompiler and uneval).

In theory, this fuzzer could compare the behavior of two completely different JS engines if it were lenient about differences in error messages.  But I'm currently only testing "js" against "js -j".
(Assignee)

Updated

9 years ago
Depends on: 465483
(Assignee)

Updated

9 years ago
Depends on: 465484
(Assignee)

Updated

9 years ago
Depends on: 465567
(Assignee)

Updated

9 years ago
Depends on: 465605
(Assignee)

Updated

9 years ago
Depends on: 465686
(Assignee)

Updated

9 years ago
Depends on: 465688
(Assignee)

Updated

9 years ago
Depends on: 465901
(Assignee)

Updated

9 years ago
Depends on: 465902
(Assignee)

Updated

9 years ago
Depends on: 465915
(Assignee)

Updated

9 years ago
Depends on: 466076
(Assignee)

Updated

9 years ago
Depends on: 466128
(Assignee)

Updated

9 years ago
Depends on: 466262
(Assignee)

Updated

9 years ago
Depends on: 466781
(Assignee)

Updated

9 years ago
Depends on: 466787
(Assignee)

Updated

9 years ago
Depends on: 466905
(Assignee)

Updated

9 years ago
Depends on: 468711
(Assignee)

Updated

9 years ago
Depends on: 469237
(Assignee)

Updated

9 years ago
Depends on: 469239
(Assignee)

Updated

9 years ago
Depends on: 469927
(Assignee)

Updated

9 years ago
Depends on: 469938
(Assignee)

Updated

9 years ago
Depends on: 469942
(Assignee)

Updated

9 years ago
Depends on: 469943
(Assignee)

Updated

9 years ago
Depends on: 470128
(Assignee)

Updated

9 years ago
Depends on: 470129
(Assignee)

Updated

9 years ago
Depends on: 470133
(Assignee)

Updated

9 years ago
Depends on: 470139
(Assignee)

Updated

9 years ago
Depends on: 470143
(Assignee)

Updated

9 years ago
Depends on: 470144
(Assignee)

Updated

9 years ago
Depends on: 470173
(Assignee)

Updated

9 years ago
Depends on: 470176
(Assignee)

Updated

9 years ago
Depends on: 470187
(Assignee)

Updated

9 years ago
Depends on: 470300
(Assignee)

Updated

9 years ago
Depends on: 470735
(Assignee)

Updated

9 years ago
Depends on: 470736
(Assignee)

Updated

9 years ago
Depends on: 470737
(Assignee)

Updated

9 years ago
Depends on: 470738
(Assignee)

Updated

9 years ago
Depends on: 470739
(Assignee)

Updated

9 years ago
Depends on: 470779
(Assignee)

Updated

9 years ago
Depends on: 470959
(Assignee)

Updated

9 years ago
Depends on: 470964
(Assignee)

Updated

9 years ago
Depends on: 471701
(Assignee)

Updated

9 years ago
Depends on: 472941
(Assignee)

Updated

9 years ago
Depends on: 473014
(Assignee)

Updated

9 years ago
Depends on: 474639
(Assignee)

Updated

9 years ago
Depends on: 474769
(Assignee)

Updated

9 years ago
Depends on: 474835
(Assignee)

Updated

9 years ago
Depends on: 474935
(Assignee)

Updated

9 years ago
Depends on: 474951
(Assignee)

Updated

9 years ago
Depends on: 475658
(Assignee)

Updated

9 years ago
Depends on: 476118
(Assignee)

Updated

8 years ago
Depends on: 489682
(Assignee)

Updated

8 years ago
Depends on: 491965
(Assignee)

Updated

8 years ago
Depends on: 491973
(Assignee)

Updated

8 years ago
Depends on: 491989
(Assignee)

Updated

8 years ago
Depends on: 493662
(Assignee)

Updated

8 years ago
Depends on: 495563
(Assignee)

Updated

8 years ago
Depends on: 495566
(Assignee)

Updated

8 years ago
Depends on: 495773
(Assignee)

Updated

8 years ago
Depends on: 495897
(Assignee)

Updated

8 years ago
Depends on: 495907
(Assignee)

Updated

8 years ago
Depends on: 495958
(Assignee)

Updated

8 years ago
Depends on: 495962
(Assignee)

Updated

8 years ago
Depends on: 496185
(Assignee)

Updated

8 years ago
Depends on: 496245
(Assignee)

Updated

8 years ago
Depends on: 496251
(Assignee)

Updated

8 years ago
Depends on: 496270
Depends on: 496530
No longer depends on: 496530
Depends on: 496531
Depends on: 496532
Depends on: 496541
Depends on: 496544
Depends on: 496687
Depends on: 496784
(Assignee)

Updated

8 years ago
Depends on: 496813
Depends on: 496816
Depends on: 496867
Depends on: 496922
Depends on: 496987
Depends on: 497015
Depends on: 498137
(Assignee)

Updated

8 years ago
Depends on: 498236
(Assignee)

Updated

8 years ago
Depends on: 498237
Depends on: 498549
Depends on: 498555
(Assignee)

Updated

8 years ago
Depends on: 501275
(Assignee)

Updated

8 years ago
Depends on: 502768
(Assignee)

Updated

8 years ago
Depends on: 502777
Depends on: 503488
Depends on: 503648
Depends on: 503818
Depends on: 504516
Depends on: 504520
Depends on: 505003
Depends on: 507556
Depends on: 507678
Depends on: 508187
(Assignee)

Updated

7 years ago
Depends on: 566637
(Assignee)

Updated

7 years ago
Depends on: 566639
(Assignee)

Updated

7 years ago
Depends on: 584587
(Assignee)

Updated

7 years ago
Depends on: 584594
(Assignee)

Updated

7 years ago
Depends on: 584603
(Assignee)

Updated

7 years ago
Depends on: 584605
(Assignee)

Updated

7 years ago
Depends on: 584647
(Assignee)

Updated

7 years ago
Depends on: 585260
Depends on: 586547
(Assignee)

Updated

7 years ago
Depends on: 593554
(Assignee)

Updated

7 years ago
Depends on: 593556
(Assignee)

Updated

7 years ago
Depends on: 601454
(Assignee)

Updated

7 years ago
Depends on: 617288
(Assignee)

Updated

7 years ago
Depends on: 620761
(Assignee)

Updated

7 years ago
Depends on: 620902
(Assignee)

Updated

7 years ago
Depends on: 621418
Depends on: 621376
Depends on: 621377
(Assignee)

Updated

7 years ago
Depends on: 621464
(Assignee)

Updated

7 years ago
Depends on: 622271
Depends on: 624377
Depends on: 627685
(Assignee)

Updated

6 years ago
Depends on: 646255
Depends on: 647524
Depends on: 647695
Depends on: 648708
Depends on: 649339
Depends on: 651827
Depends on: 652414
Depends on: 655699
Depends on: 656228
Depends on: 656229
Depends on: 657193
Depends on: 657986
Depends on: 658539
Depends on: 660438
Depends on: 660437
Depends on: 673954
Depends on: 690292
Depends on: 699201
Depends on: 706710
Depends on: 712379
Depends on: 713957
Depends on: 715387
Depends on: 715400
Depends on: 718076
(Assignee)

Updated

5 years ago
Depends on: 735161
(Assignee)

Updated

5 years ago
Depends on: 735316
Depends on: 740595
(Assignee)

Updated

5 years ago
Depends on: 743423
(Assignee)

Updated

5 years ago
Depends on: 743425
(Assignee)

Updated

5 years ago
Depends on: 755813
Depends on: 887521
Depends on: 887544
Depends on: 887542
Depends on: 887549
Depends on: 887556
Depends on: 891775
Depends on: 892787
Depends on: 906284
Depends on: 906285
Depends on: 906286
Depends on: 908608
Depends on: 908813
Depends on: 909601
Depends on: 909602
Depends on: 910012
Depends on: 911369
Depends on: 912303
Depends on: 912304
Depends on: 912316
Depends on: 912328
Depends on: 913749
Depends on: 913885
Depends on: 914341
Depends on: 916039
Depends on: 923765
Depends on: 937550
Depends on: 937922
(Assignee)

Updated

4 years ago
Depends on: 939868
(Assignee)

Updated

4 years ago
Alias: js-differential-test
(Assignee)

Updated

4 years ago
Depends on: 940635
(Assignee)

Updated

4 years ago
Depends on: 940642
(Assignee)

Updated

4 years ago
Depends on: 940864
(Assignee)

Updated

4 years ago
Depends on: 941381
Depends on: 942390
Depends on: 942549
Depends on: 942550
(Assignee)

Updated

4 years ago
Depends on: 944153
Depends on: 944266
Depends on: 944975
Depends on: 945512
(Assignee)

Updated

4 years ago
Depends on: 946679
Depends on: 946969
Depends on: 948321
Depends on: 958381
Depends on: 969203
Depends on: 969705
Depends on: 973118
Depends on: 975636
Depends on: 995673
Depends on: 975138
Depends on: 983840
Depends on: 981325
Depends on: 995675
Depends on: 995679
Depends on: 995817
Depends on: 995816
Depends on: 995826
Depends on: 996881
Depends on: 996895
Depends on: 997546
Depends on: 998059
Depends on: 998262
Depends on: 998580
Depends on: 998709
Depends on: 999790
Depends on: 999849
Depends on: 999857
Depends on: 1000605
Depends on: 1000606
Depends on: 1006910
Depends on: 1008818
Depends on: 1007213
Depends on: 1015656
Depends on: 1022948
Depends on: 1024444
Depends on: 1025587
Depends on: 1027359
Depends on: 1027846
Depends on: 1033873
Depends on: 1033946
Depends on: 1034280
Depends on: 1037665
Depends on: 1053074
Depends on: 1054531
Depends on: 1054541
Depends on: 1054545
Depends on: 1054568
Depends on: 1066496
Depends on: 1073910
Depends on: 1073928
Depends on: 1076091
Depends on: 1076283
Depends on: 1077074
Depends on: 1079062
Depends on: 1081850
Depends on: 1085298
Depends on: 1085299
Depends on: 1090424
Depends on: 1103032
Depends on: 1103048
Depends on: 1105574
Depends on: 1122338
Depends on: 1122344
Depends on: 1122401
Depends on: 1122402
Depends on: 1122403
Depends on: 1122839
Depends on: 1123011
Depends on: 1124421
Depends on: 1124448
Depends on: 1124485
Depends on: 1126066
Depends on: 1129088
Depends on: 1130679
Depends on: 1132290
Depends on: 1132396
Depends on: 1133389
Depends on: 1135047
Depends on: 1136542
Depends on: 1137610
Depends on: 1137616
Depends on: 1137624
Depends on: 1137998
Depends on: 1138740
Depends on: 1140890
Depends on: 1143878
Depends on: 1147662
Depends on: 1148973
(Assignee)

Updated

2 years ago
Depends on: 1149739
Depends on: 1153153
Depends on: 1154971
Depends on: 1159899
Depends on: 1181354
Depends on: 1181828
Depends on: 1183423
Depends on: 1186226
Depends on: 1186271
(Assignee)

Comment 1

2 years ago
Differential testing is now integrated with jsfunfuzz.

A random set of flags is chosen when running jsfunfuzz:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/shellFlags.py

The output is compared against running the same shell with no special flags:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/compareJIT.py

We also check that poking the garbage collector never affects output:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/shared/testing-functions.js

Differential testing excludes Date, Math.random, and a few other things:
https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/avoid-known-bugs.js#L18
Summary: Fuzzer that compares interpreter output with JIT output for entire scripts → Bugs found by JS differential testing (comparing output with different JIT options)
(Assignee)

Comment 2

2 years ago
These parts of jsfunfuzz are especially designed for differential testing:

https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/gen-asm.js
https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/gen-math.js
https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/test-asm.js
https://github.com/MozillaSecurity/funfuzz/blob/master/js/jsfunfuzz/test-math.js
Depends on: 1203862
Depends on: 1204675
Depends on: 1206265
Depends on: 1207449
Depends on: 1210596
Depends on: 1210607
No longer depends on: 1210607
Depends on: 1211100
Depends on: 1213552
Depends on: 1220275
Depends on: 1227287
Depends on: 1228397
Depends on: 1234736
Depends on: 1236114
Depends on: 1237403
Depends on: 1237464
Depends on: 1237564
Depends on: 1238461
Depends on: 1239075
Depends on: 1244502
Depends on: 1245187
Depends on: 1245627
Depends on: 1246200
Depends on: 1246552
Depends on: 1247701
Depends on: 1247862
Depends on: 1247863
Depends on: 1247871
Depends on: 1247877
Depends on: 1247880
Depends on: 1248153
Depends on: 1250863
Depends on: 1253898
Depends on: 1263525
Depends on: 1263811
Depends on: 1264561
Depends on: 1265159
Depends on: 1266242
Depends on: 1268224
Depends on: 1268955
Depends on: 1271850
Depends on: 1273267
Depends on: 1274429
Depends on: 1277118
Depends on: 1280252
Component: Tracking → Platform Fuzzing Team
Depends on: 1286407
Depends on: 1293542
Depends on: 1293575
Depends on: 1296243
Depends on: 1301208
Depends on: 1304638
Depends on: 1304640
Depends on: 1304641
Depends on: 1304643
Depends on: 1304649
Depends on: 1308743
Depends on: 1308802
Depends on: 1312620
Depends on: 1314438
Depends on: 1314545
Depends on: 1316830
Depends on: 1317943
Depends on: 1319242
Depends on: 1321437
Depends on: 1330234
Depends on: 1345707
Depends on: 1368573
Depends on: 1368574
Depends on: 1368575
Depends on: 1368576
Depends on: 1368584
Depends on: 1372956
Depends on: 1377141
Depends on: 1379936
You need to log in before you can comment on or make changes to this bug.