The number of crash and assertion OOM bugs is too damn high [meta]

NEW
Unassigned

Status

()

Core
JavaScript Engine
--
major
4 years ago
3 days ago

People

(Reporter: decoder, Unassigned)

Tracking

(Depends on: 9 bugs, {meta, sec-want})

Trunk
All
Linux
meta, sec-want
Points:
---
988953, 1269705, 1269714, 1269718, 1299115, 1305739, 1317329, 1344673, 1367526, 871862, 877437, 914598, 914601, 914614, 915336, 915497, 917759, 925146, 929065, 929221, 930526, 932530, 937083, 940025, 945568, 945754, 947233, 947958, 947963, 948023, 948187, 948188, 948233, 948647, 950474, 950658, 958598, 959167, 959208, 964803, 978802, 987910, 987933, 987935, 987947, 988097, 990071, 990096, 990787, 990806, 991027, 991036, 991249, 992274, 992968, 994159, 1000145, 1000182, 1026465, 1026476, 1111201, 1130672, 1133630, 1155618, 1164532, 1171909, 1175755, 1177122, 1180064, 1186982, 1188296, 1188301, 1188347, 1188390, 1188878, 1189343, 1191756, 1191758, 1193039, 1193043, 1193102, 1195452, 1196027, 1199175, 1204721, 1204725, 1204847, 1204849, 1204866, 1205603, 1205639, 1205708, 1206539, 1206677, 1207413, 1207569, 1207574, 1207863, 1208994, 1209001, 1209026, 1209497, 1209585, 1209943, 1209945, 1211009, 1211913, 1211939, 1211949, 1211956, 1211962, 1211964, 1211977, 1212094, 1212258, 1212278, 1212279, 1212296, 1212298, 1212343, 1212389, 1212390, 1212469, 1212927, 1214175, 1215058, 1215363, 1215600, 1215678, 1216157, 1216261, 1216599, 1216607, 1223021, 1223023, 1225078, 1232676, 1233115, 1234280, 1234387, 1234402, 1234410, 1234411, 1234414, 1236473, 1236476, 1236525, 1238555, 1238575, 1238577, 1238582, 1238610, 1240502, 1240503, 1240521, 1240527, 1240546, 1240736, 1240803, 1241731, 1242279, 1242812, 1242835, 1242840, 1243374, 1243397, 1243410, 1243787, 1245520, 1245862, 1246607, 1248101, 1252329, 1252707, 1252903, 1253124, 1254122, 1254123, 1254172, 1254190, 1254203, 1254578, 1255954, 1255956, 1257194, 1258999, 1260259, 1260725, 1261308, 1261329, 1261342, 1262936, 1263862, 1263865, 1263868, 1263870, 1263871, 1263874, 1263879, 1263884, 1263886, 1263895, 1263902, 1264612, 1264823, 1264948, 1264954, 1264961, 1264998, 1265690, 1265693, 1268309, 1269710, 1269722, 1269755, 1269756, 1269759, 1278193, 1278839, 1282743, 1282986, 1284485, 1284491, 1285217, 1285927, 1285934, 1287411, 1287412, 1292564, 1293311, 1296661, 1296667, 1296669, 1297142, 1298355, 1298776, 1298804, 1299103, 1299106, 1302411, 1302417, 1303015, 1305791, 1315946, 1328151, 1344265, 1344334, 1345453, 1346810, 1347120, 1348903, 1348904
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
The number of crashes and assertions related to OOM conditions keeps growing (every larger new piece of code introduces additional OOM failures). Today, I count at least a dozen signatures related to OOM (probably more), that I see daily in the fuzzer. While I understand that fixing these bugs is not priority, the reality is that they don't get fixed at all. This is dangerous because some of these bugs have somewhat generic signatures and other bugs with the same signatures might go unseen because of that.

With the testing function added in bug 872823, the fuzzer can easily create short test cases for some of these OOM bugs. However, these tests are usually short-lived and a developer needs to start working on it right after it was filed, not a month or two later (when it's guaranteed to not reproduce and also impossible to tell if it's fixed or not).

I'd like to discuss ways to achieve this. Even a commitment to fixing only *one* OOM bug per week would quickly drive down the number of bugs. Even one per month would improve the situation.
Like you said, it's not a high priority for us and most of these bugs are not dangerous, but I agree there are probably some serious issues.

It's a bit like the differential testing bugs Gary is filing. We have to fix many old bugs first; that will take a while and many of them are edge cases, but after that we will hopefully be able to catch new and more interesting bugs much sooner.

It would be nice if somebody could step up and triage/fix a few of these OOM bugs every week. I'd be happy to do it, but getting the differential testing bugs fixed already takes a lot of time. Anybody else maybe?
(Reporter)

Comment 2

4 years ago
Naveed, Hannes suggests that I file 1-2 OOM bugs (starting with the most common), and get you to assign an owner for them. Once they are solved, I can file a new one.

Due to the nature of OOM bugs, it is often not possible to bisect them at all. The initial owner could be any JS dev who then either fixes the problem directly, or assigns it to the person working in the area where the OOM is likely happening and not being handled properly.
Flags: needinfo?(nihsanullah)
I'm opposed to asking decoder to throttle these bugs on principle. We're talking about a quality problem, possibly a security problem.

Decoder offered this search as a starting point for OOM bugs:
  http://tinyurl.com/k8uscwm

I see 15 open bugs, including some old ones that might be gone.
(In reply to Jason Orendorff [:jorendorff] from comment #3)
> I'm opposed to asking decoder to throttle these bugs on principle. We're
> talking about a quality problem, possibly a security problem.

I used to throttle my Differential Testing bugs too until jandem picked them up recently and consistently kept fixing them (or assigned people to fix them), since he could do the initial triage and/or analysis and assign the right person.

Otherwise it's just too much to handle for a non-JS tech person, we need a point person for OOM bugs. Not really to fix everything, but to handle the initial triage/analysis in a role similar to jandem.
If Christian and Gary can order these by importance (e.g. likelyhood of masking more important signatures), then I think this should actually be quite a useful activity. I would be happy do triage, although I may have to rate-limit it to a handful a week.
Depends on: 871862
Depends on: 877437
Judging by the ones that *have* been fixed, about half seem to be due to failure to check for NULL return values.

You may be able to identify these easily:

  * the symptom is either "Assertion failure: foo != NULL" or "Assertion failure: foo"
    or some similar assertion, or a crash near NULL

  * they reproduce reliably -- might even bisect, if you can do that for OOMAfter bugs.

And they're easy to fix. So these should be filed as regressions, and if you find one that's a recent regression, I think we should back out the original patch if the committer isn't around or doesn't have time to fix it at the moment.

The other half don't seem to have a single common source.
(Reporter)

Updated

4 years ago
Depends on: 914598
(Reporter)

Updated

4 years ago
Depends on: 914601
(Reporter)

Updated

4 years ago
Depends on: 914614
Depends on: 915497
(Reporter)

Updated

4 years ago
Depends on: 917759
(Reporter)

Updated

4 years ago
Depends on: 925146
Depends on: 929065
(Reporter)

Updated

4 years ago
Depends on: 915336
Depends on: 932530
Depends on: 930526
Depends on: 940025
Jason and Terrence thanks for taking on this triage
Flags: needinfo?(nihsanullah)
(Reporter)

Updated

4 years ago
Depends on: 945568
(Reporter)

Updated

4 years ago
Depends on: 945754
(Reporter)

Updated

4 years ago
Depends on: 947233
(Reporter)

Updated

4 years ago
Depends on: 947958
Depends on: 947963
(Reporter)

Updated

4 years ago
Depends on: 948023
(Reporter)

Updated

4 years ago
Depends on: 948187
(Reporter)

Updated

4 years ago
Depends on: 948188
(Reporter)

Updated

4 years ago
Depends on: 948233
(Reporter)

Updated

4 years ago
Depends on: 937083
(Reporter)

Updated

4 years ago
Depends on: 948647
(Reporter)

Updated

4 years ago
Depends on: 950474
(Reporter)

Updated

4 years ago
Depends on: 950658
(Reporter)

Updated

4 years ago
Depends on: 929221
Keywords: meta
Summary: The number of OOM bugs is too damn high → The number of crash and assertion OOM bugs is too damn high [meta]
(Reporter)

Updated

3 years ago
Depends on: 958598
(Reporter)

Updated

3 years ago
Depends on: 959167
(Reporter)

Updated

3 years ago
Depends on: 959208
(Reporter)

Updated

3 years ago
Depends on: 964803
(Reporter)

Updated

3 years ago
Depends on: 978802

Updated

3 years ago
Depends on: 987910

Updated

3 years ago
Depends on: 987933

Updated

3 years ago
Depends on: 987935

Updated

3 years ago
Depends on: 987947
(Reporter)

Updated

3 years ago
Depends on: 988097
(Reporter)

Updated

3 years ago
Depends on: 988953
(Reporter)

Updated

3 years ago
Depends on: 990071
(Reporter)

Updated

3 years ago
Depends on: 990096
Depends on: 990787
Depends on: 990806
Depends on: 991027
Depends on: 991036
Depends on: 991249
(Reporter)

Updated

3 years ago
Depends on: 992968
Depends on: 992274
Depends on: 994159
(Reporter)

Updated

3 years ago
Depends on: 1000182
(Reporter)

Updated

3 years ago
Depends on: 1000145
(Reporter)

Updated

3 years ago
Depends on: 1026465
(Reporter)

Updated

3 years ago
Depends on: 1026476
(Assignee)

Updated

3 years ago
Assignee: general → nobody
(Reporter)

Updated

2 years ago
Depends on: 1164532
(Reporter)

Updated

2 years ago
Depends on: 1171909
(Reporter)

Updated

2 years ago
Depends on: 1175755
(Reporter)

Updated

2 years ago
Depends on: 1177122
(Reporter)

Updated

2 years ago
Depends on: 1188296
(Reporter)

Updated

2 years ago
Depends on: 1111201
(Reporter)

Updated

2 years ago
Depends on: 1188301
(Reporter)

Updated

2 years ago
Depends on: 1188347
(Reporter)

Updated

2 years ago
Depends on: 1133630
(Reporter)

Updated

2 years ago
Depends on: 1130672
(Reporter)

Updated

2 years ago
Depends on: 1180064
(Reporter)

Updated

2 years ago
Depends on: 1186982
(Reporter)

Updated

2 years ago
Depends on: 1188390
(Reporter)

Updated

2 years ago
Depends on: 1188878
(Reporter)

Updated

2 years ago
Depends on: 1191756
(Reporter)

Updated

2 years ago
Depends on: 1191758
(Reporter)

Updated

2 years ago
Depends on: 1193039
(Reporter)

Updated

2 years ago
Depends on: 1193043
(Reporter)

Updated

2 years ago
Depends on: 1193102
(Reporter)

Updated

2 years ago
Depends on: 1195452
(Reporter)

Updated

2 years ago
Depends on: 1196027
(Reporter)

Updated

2 years ago
Depends on: 1199175
(Reporter)

Updated

2 years ago
Depends on: 1204721
(Reporter)

Updated

2 years ago
Depends on: 1204725
(Reporter)

Updated

2 years ago
Depends on: 1204847
(Reporter)

Updated

2 years ago
Depends on: 1204849
(Reporter)

Updated

2 years ago
Depends on: 1204866
Depends on: 1205603
(Reporter)

Updated

2 years ago
Depends on: 1205639
(Reporter)

Updated

2 years ago
Depends on: 1205708
(Reporter)

Updated

2 years ago
Depends on: 1206539
(Reporter)

Updated

2 years ago
Depends on: 1206677
(Reporter)

Updated

2 years ago
Depends on: 1207413
(Reporter)

Updated

2 years ago
Depends on: 1207569
(Reporter)

Updated

2 years ago
Depends on: 1207574
(Reporter)

Updated

2 years ago
Depends on: 1207863
(Reporter)

Updated

2 years ago
Depends on: 1208994
(Reporter)

Updated

2 years ago
Depends on: 1209001
(Reporter)

Updated

2 years ago
Depends on: 1209026
(Reporter)

Updated

2 years ago
Depends on: 1209497
(Reporter)

Updated

2 years ago
Depends on: 1209585
(Reporter)

Updated

2 years ago
Depends on: 1209943
(Reporter)

Updated

2 years ago
Depends on: 1209945
(Reporter)

Updated

2 years ago
Depends on: 1211009
(Reporter)

Updated

2 years ago
Depends on: 1211913
(Reporter)

Updated

2 years ago
Depends on: 1211939
(Reporter)

Updated

2 years ago
Depends on: 1211949
(Reporter)

Updated

2 years ago
Depends on: 1211956
(Reporter)

Updated

2 years ago
Depends on: 1211962

Updated

2 years ago
Depends on: 1211964
(Reporter)

Updated

2 years ago
Depends on: 1211977
(Reporter)

Updated

2 years ago
Depends on: 1212094
(Reporter)

Updated

2 years ago
Depends on: 1212258
(Reporter)

Updated

2 years ago
Depends on: 1212278
(Reporter)

Updated

2 years ago
Depends on: 1212279
(Reporter)

Updated

2 years ago
Depends on: 1212296
(Reporter)

Updated

2 years ago
Depends on: 1212298

Updated

2 years ago
Depends on: 1212343
(Reporter)

Updated

2 years ago
Depends on: 1212389
(Reporter)

Updated

2 years ago
Depends on: 1212390
(Reporter)

Updated

2 years ago
Depends on: 1189343

Updated

2 years ago
Depends on: 1212927

Updated

2 years ago
Depends on: 1155618

Updated

2 years ago
Depends on: 1212469
(Reporter)

Updated

2 years ago
Depends on: 1214175

Updated

2 years ago
Depends on: 1215058
(Reporter)

Updated

2 years ago
Depends on: 1215363
(Reporter)

Updated

2 years ago
Depends on: 1215600
(Reporter)

Updated

2 years ago
Depends on: 1215678
(Reporter)

Updated

2 years ago
Depends on: 1216157
(Reporter)

Updated

2 years ago
Depends on: 1216261
(Reporter)

Updated

2 years ago
Depends on: 1216599
(Reporter)

Updated

2 years ago
Depends on: 1216607
(Reporter)

Updated

2 years ago
Depends on: 1223021
(Reporter)

Updated

2 years ago
Depends on: 1223023
(Reporter)

Updated

2 years ago
Depends on: 1225078
(Reporter)

Updated

2 years ago
Depends on: 1232676
(Reporter)

Updated

2 years ago
Depends on: 1233115
(Reporter)

Updated

a year ago
Depends on: 1234280
(Reporter)

Updated

a year ago
Depends on: 1234387
(Reporter)

Updated

a year ago
Depends on: 1234402
(Reporter)

Updated

a year ago
Depends on: 1234410
(Reporter)

Updated

a year ago
Depends on: 1234411
(Reporter)

Updated

a year ago
Depends on: 1234414
(Reporter)

Updated

a year ago
Depends on: 1236473
(Reporter)

Updated

a year ago
Depends on: 1236476
(Reporter)

Updated

a year ago
Depends on: 1236525
(Reporter)

Updated

a year ago
Depends on: 1238555
(Reporter)

Updated

a year ago
Depends on: 1238575
(Reporter)

Updated

a year ago
Depends on: 1238577
(Reporter)

Updated

a year ago
Depends on: 1238582
(Reporter)

Updated

a year ago
Depends on: 1238610
(Reporter)

Updated

a year ago
Depends on: 1240502
(Reporter)

Updated

a year ago
Depends on: 1240503
(Reporter)

Updated

a year ago
Depends on: 1240521
(Reporter)

Updated

a year ago
Depends on: 1240527
(Reporter)

Updated

a year ago
Depends on: 1240546
(Reporter)

Updated

a year ago
Depends on: 1240736
(Reporter)

Updated

a year ago
Depends on: 1240803
(Reporter)

Updated

a year ago
Depends on: 1241731
(Reporter)

Updated

a year ago
Depends on: 1242279
(Reporter)

Updated

a year ago
Depends on: 1242812
(Reporter)

Updated

a year ago
Depends on: 1242835
(Reporter)

Updated

a year ago
Depends on: 1242840
(Reporter)

Updated

a year ago
Depends on: 1243374
(Reporter)

Updated

a year ago
Depends on: 1243397
(Reporter)

Updated

a year ago
Depends on: 1243410
(Reporter)

Updated

a year ago
Depends on: 1243787
(Reporter)

Updated

a year ago
Depends on: 1245520
(Reporter)

Updated

a year ago
Depends on: 1245862
(Reporter)

Updated

a year ago
Depends on: 1246607
(Reporter)

Updated

a year ago
Depends on: 1248101
(Reporter)

Updated

a year ago
Depends on: 1252329
(Reporter)

Updated

a year ago
Depends on: 1252707
(Reporter)

Updated

a year ago
Depends on: 1252903
(Reporter)

Updated

a year ago
Depends on: 1253124
(Reporter)

Updated

a year ago
Depends on: 1254122
(Reporter)

Updated

a year ago
Depends on: 1254123
(Reporter)

Updated

a year ago
Depends on: 1254172
(Reporter)

Updated

a year ago
Depends on: 1254190
(Reporter)

Updated

a year ago
Depends on: 1254203
(Reporter)

Updated

a year ago
Depends on: 1254578
(Reporter)

Updated

a year ago
Depends on: 1255954
(Reporter)

Updated

a year ago
Depends on: 1255956
(Reporter)

Updated

a year ago
Depends on: 1257194
(Reporter)

Updated

a year ago
Depends on: 1258999
(Reporter)

Updated

a year ago
Depends on: 1260259
(Reporter)

Updated

a year ago
Depends on: 1260725
(Reporter)

Updated

a year ago
Depends on: 1261308
(Reporter)

Updated

a year ago
Depends on: 1261329
(Reporter)

Updated

a year ago
Depends on: 1261342
(Reporter)

Updated

a year ago
Depends on: 1262936
(Reporter)

Updated

a year ago
Depends on: 1263862
(Reporter)

Updated

a year ago
Depends on: 1263865
(Reporter)

Updated

a year ago
Depends on: 1263868
(Reporter)

Updated

a year ago
Depends on: 1263870
(Reporter)

Updated

a year ago
Depends on: 1263871
(Reporter)

Updated

a year ago
Depends on: 1263874
(Reporter)

Updated

a year ago
Depends on: 1263879
(Reporter)

Updated

a year ago
Depends on: 1263884
(Reporter)

Updated

a year ago
Depends on: 1263886
(Reporter)

Updated

a year ago
Depends on: 1263895
(Reporter)

Updated

a year ago
Depends on: 1263902
(Reporter)

Updated

a year ago
Depends on: 1264612
(Reporter)

Updated

a year ago
Depends on: 1264823
(Reporter)

Updated

a year ago
Depends on: 1264948
(Reporter)

Updated

a year ago
Depends on: 1264954
(Reporter)

Updated

a year ago
Depends on: 1264961
(Reporter)

Updated

a year ago
Depends on: 1264998
(Reporter)

Updated

a year ago
Depends on: 1265690
(Reporter)

Updated

a year ago
Depends on: 1265693
(Reporter)

Updated

a year ago
Depends on: 1268309
(Reporter)

Updated

a year ago
Depends on: 1269705
(Reporter)

Updated

a year ago
Depends on: 1269710
(Reporter)

Updated

a year ago
Depends on: 1269714
(Reporter)

Updated

a year ago
Depends on: 1269718
(Reporter)

Updated

a year ago
Depends on: 1269722
(Reporter)

Updated

a year ago
Depends on: 1269755
(Reporter)

Updated

a year ago
Depends on: 1269756
(Reporter)

Updated

a year ago
Depends on: 1269759
(Reporter)

Updated

a year ago
Depends on: 1278193
(Reporter)

Updated

a year ago
Depends on: 1278839
(Reporter)

Updated

11 months ago
Depends on: 1282743
(Reporter)

Updated

11 months ago
Depends on: 1282986
(Reporter)

Updated

11 months ago
Depends on: 1284485
(Reporter)

Updated

11 months ago
Depends on: 1284491
(Reporter)

Updated

11 months ago
Depends on: 1285217
(Reporter)

Updated

11 months ago
Depends on: 1285927
(Reporter)

Updated

11 months ago
Depends on: 1285934
(Reporter)

Updated

10 months ago
Depends on: 1287411
(Reporter)

Updated

10 months ago
Depends on: 1287412
(Reporter)

Updated

10 months ago
Depends on: 1292564
(Reporter)

Updated

10 months ago
Depends on: 1293311
(Reporter)

Updated

9 months ago
Depends on: 1296661
(Reporter)

Updated

9 months ago
Depends on: 1296667
(Reporter)

Updated

9 months ago
Depends on: 1296669
(Reporter)

Updated

9 months ago
Depends on: 1297142
(Reporter)

Updated

9 months ago
Depends on: 1298355
(Reporter)

Updated

9 months ago
Depends on: 1298776
(Reporter)

Updated

9 months ago
Depends on: 1298804
(Reporter)

Updated

9 months ago
Depends on: 1299103
(Reporter)

Updated

9 months ago
Depends on: 1299106
(Reporter)

Updated

9 months ago
Depends on: 1299115
(Reporter)

Updated

9 months ago
Depends on: 1302411
(Reporter)

Updated

9 months ago
Depends on: 1302417
(Reporter)

Updated

9 months ago
Depends on: 1303015

Updated

8 months ago
Depends on: 1305739
(Reporter)

Updated

8 months ago
Depends on: 1305791
(Reporter)

Updated

7 months ago
Depends on: 1315946
(Reporter)

Updated

6 months ago
Depends on: 1317329
(Reporter)

Updated

5 months ago
Depends on: 1328151
(Reporter)

Updated

3 months ago
Depends on: 1344265
(Reporter)

Updated

3 months ago
Depends on: 1344334
(Reporter)

Updated

3 months ago
Depends on: 1344673
(Reporter)

Updated

3 months ago
Depends on: 1345453
(Reporter)

Updated

3 months ago
Depends on: 1346810
(Reporter)

Updated

3 months ago
Depends on: 1347120
(Reporter)

Updated

2 months ago
Depends on: 1348903
(Reporter)

Updated

2 months ago
Depends on: 1348904
(Reporter)

Updated

3 days ago
Depends on: 1367526
You need to log in before you can comment on or make changes to this bug.