Open Bug 1861789 (bugbot-auto-crash) Opened 2 years ago Updated 1 day ago

[meta](BugBot) Automatic filing of new crashes

Categories

(Developer Infrastructure :: Source Code Analysis, enhancement)

Product:
Developer Infrastructure
Component:
Source Code Analysis
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: suhaib, Unassigned)

References

(Depends on 307 open bugs,
URL
)

Details

(Keywords: meta)

This meta bug is used to track the bugs that are automatically filed by BugBot.

Depends on: 1862275
Depends on: 1862276
Depends on: 1862277
Depends on: 1862278
Depends on: 1862279
Depends on: 1862280
Depends on: 1862281
Depends on: 1862282
Depends on: 1862283
Depends on: 1862284
Depends on: 1862345
Depends on: 1862346
Depends on: 1862347
Depends on: 1862348
Depends on: 1862349

How is it determined that these crashes are "actionable"?

(In reply to Timothy Nikkel (:tnikkel) from comment #1)

How is it determined that these crashes are "actionable"?

There is a set of rules the bot follows to filter out crashes, most of them listed in the "Identify new actionable crashes" in https://docs.google.com/document/d/1iZiM0eSSEXg0q_pLaUiHilzJaSc6KnG-CcM9-BC2g_Y/edit. Crashes matching the following are not automatically filed:

  • Hardware failure crashes
  • Out-of-memory crashes
  • Crashes reported from the same person
  • Shutdown hangs

We are planning to adjust the rules after the initial rollout, depending on what we see. We have already tuned them on Bugzilla Dev with the help of some engineers, but with a wider rollout I'm sure we'll find more things.
If you already have some suggestions based on the bugs you have seen so far, we are happy to add more rules or tune the existing ones! Also let us know if you see any bug that shouldn't have been filed, and we can try to tune the rules ourselves to make sure similar bugs don't get autofiled in the future.

Most of the bugs I'm seeing filed a very low volume. Engineers aren't going to spend time on them, it just wastes resources having to triage them.

There's only about a dozen bugs depending on this, pretty easy to skim through them to see.

(In reply to Timothy Nikkel (:tnikkel) from comment #4)

There's only about a dozen bugs depending on this, pretty easy to skim through them to see.

Suhaib does not regularly triage platform bugs, so more specific feedback from somebody like you who does would probably be more actionable. I don't think there's been any feedback from anybody who does a lot of graphics triage, so I'm sure that would be a welcome perspective.

I looked over all of the bugs reported so far, and it does look like the reporting threshold for bugs without anything to make them particularly actionable seems rather low. Specifically, there are a bunch of crashes in graphics drivers (bug 1862276, bug 1862279, bug 1862280, bug 1862283) that have only happened a couple of times in the last week on Nightly, so I don't think they should be filed. A few of them only happened once. Bug 1862347 is another crash in graphics drivers, though that one is a bit odd because it is a null pointer crash, but it is still happening in graphics code with very low volume, so the odds of us being able to take a action to fix it are very low.

Bug 1862348 has only happened twice in the last week on Nightly, and is not a null crash or an assert so it also seems like it would have been better to not file it.

Bug 1862284 and bug 1862349 also have only happened a few times each, but they are release asserts in our code, so there might be something actionable, so it looks reasonable to me (as a non-graphics person) that bugs were filed for them.

Depends on: 1862806
Depends on: 1862807
Depends on: 1862809
Depends on: 1863100
Depends on: 1863177
Depends on: 1863178
Depends on: 1863179
Depends on: 1863180
See Also: → 1863189

(In reply to Marco Castelluccio [:marco] from comment #2)

There is a set of rules the bot follows to filter out crashes, most of them listed in the "Identify new actionable crashes" [...] Crashes matching the following are not automatically filed:

  • Hardware failure crashes
  • Crashes reported from the same person

Counterexamples where the bot might not be applying these properly:

  • In bug 1863178, two out of the four crashes look to be from the same person, and potentially due to hardware issues (per the main paragraph in bug 1863178 comment 3)
  • In bug 1863177, eight out of the ten crashes look to be from the same person who was apparently testing a local web service on 127.0.0.1. (see bug 1863177 comment 2.)

(I'm not sure if BugBot mistakenly interpreted this^ crash volume as contributing to the "signal" here; it looks superficially like it did, but I suppose it might have been aware of these same-user/possible-hardware-issue factors and still filed because it decided that the remaining (small) crash volume was still enough to nudge it to file a bug.)

(In reply to Daniel Holbert [:dholbert] from comment #6)

Thank you for the feedback, this will help us improve the bot.

Counterexamples where the bot might not be applying these properly:

  • In bug 1863178, two out of the four crashes look to be from the same person, and potentially due to hardware issues (per the main paragraph in bug 1863178 comment 3)

The bot ignores crashes that happen on only one installation. In cases other cases where the crash does not show other interesting singles (e.g., security-related, crash on null, has potential regressor), the threshold increased to be crashes > 25 and installations > 5 (working on toning these thresholds, they will probably be increased). In the case of bug 1863178, it is null crash and has a potential regressor (the regressor is a false positive in this case). By checking the number of installations, it seems that the crash happened on more than one installation.

To ignore bad hardware related crashes, there are multiple criteria; the bit flip is one of them. These criteria focus on reports seen in the last two weeks. In the case of bug 1863178, none of the crash reports in the last two weeks have "Possible bit flips max confidence" (the mentioned reports are a bit older).

  • In bug 1863177, eight out of the ten crashes look to be from the same person who was apparently testing a local web service on 127.0.0.1. (see bug 1863177 comment 2.)

(I'm not sure if BugBot mistakenly interpreted this^ crash volume as contributing to the "signal" here; it looks superficially like it did, but I suppose it might have been aware of these same-user/possible-hardware-issue factors and still filed because it decided that the remaining (small) crash volume was still enough to nudge it to file a bug.)

The fact that 8 out of 10 crashes happened on null or near null, makes the crash qualifies to skip the volume threshold. However, in any case, there is no bug will be filed for a crash that happens on only one installation. In case of bug 1863177, the crash happened on 8 different installations on the last two weeks.

Thanks for the clarifications.

(In reply to Suhaib Mujahid [:suhaib] from comment #7)

The fact that 8 out of 10 crashes happened on null or near null, makes the crash qualifies to skip the volume threshold.

FWIW the "crash near null" here is in fact just a MOZ_RELEASE_ASSERT failing, in that particular case. (Maybe it makes sense to consider those as the same thing.)

In case of bug 1863177, the crash happened on 8 different installations on the last two weeks.

I'm not sure how we identify installations, but maybe it's too strict? It's pretty clear to me that those 8 different crashes were all on the same installation, for some definition of installation -- they all have an identical "install time" field (2023-10-31 19:20:56), identical hardware and OS metadata fields, and near-identical URLs (all 127.0.0.1; maybe a fuzzer or something else with local testing).

(Edit: I'm aware that the URL is a protected/hidden field in the crash report, and the bot might not have access to it; I just mention it since [for those with access] it's just a point of additional confirmation, on top of what is already pretty-clear-evidence IMO that those 8 crashes look like the same installation.)

Depends on: 1863238
Depends on: 1864087
Depends on: 1864241
Depends on: 1864305
Depends on: 1864306
Depends on: 1864574
Depends on: 1864771
Depends on: 1864988
Depends on: 1864989
Depends on: 1864990
Depends on: 1865236
Depends on: 1865443
Depends on: 1865444
Depends on: 1865498
Depends on: 1865499
Depends on: 1865500
Depends on: 1865569
Depends on: 1865753
Depends on: 1866385
Depends on: 1866595
Depends on: 1866657
Depends on: 1866932
Depends on: 1867165
Depends on: 1867166
Depends on: 1867167
Depends on: 1867414
Depends on: 1867695
Depends on: 1867696
Depends on: 1867982
Depends on: 1867983
Depends on: 1868038
Depends on: 1868058
Depends on: 1868252
Depends on: 1869139
Depends on: 1869140
Depends on: 1869141
Depends on: 1869203
Depends on: 1869676
Depends on: 1869932
Depends on: 1869933
Depends on: 1870181
Depends on: 1870182
Depends on: 1870183
Depends on: 1870545
Depends on: 1870546
Depends on: 1871173
Depends on: 1871174
Depends on: 1871175
Depends on: 1871505
Depends on: 1871506
Depends on: 1871828
Depends on: 1871890
Depends on: 1871891
Depends on: 1871998
Depends on: 1871999
Depends on: 1872000
Depends on: 1872210
Depends on: 1872211
Depends on: 1872212
Depends on: 1872317
Depends on: 1872416
No longer depends on: 1869140
Depends on: 1869140
Depends on: 1873388
Depends on: 1873389
Depends on: 1873572
Depends on: 1873801
Depends on: 1874518
Depends on: 1874574
Depends on: 1874575
Depends on: 1874576
Depends on: 1874577
Depends on: 1874578
Depends on: 1874794
Depends on: 1875192
Depends on: 1875193
Depends on: 1875407
Depends on: 1875956
Depends on: 1876425
Depends on: 1876939
Depends on: 1877044
Depends on: 1877045
Depends on: 1877046
Depends on: 1877331
Depends on: 1877590
Depends on: 1877876
Depends on: 1878195
Depends on: 1878437
Depends on: 1878438
Depends on: 1878585
Depends on: 1878586
Depends on: 1878984
Depends on: 1878985
Depends on: 1878986
Depends on: 1879229
Depends on: 1879451
Depends on: 1879982
Depends on: 1880197
Depends on: 1880389
Depends on: 1880390
Depends on: 1880580
Depends on: 1880581
Depends on: 1880699
Depends on: 1880737
Depends on: 1880768
Depends on: 1881372
Depends on: 1881870
Depends on: 1881935
Depends on: 1881936
Depends on: 1881990
Depends on: 1882423
Depends on: 1882424
Depends on: 1882795
Depends on: 1882796
Depends on: 1882933
Depends on: 1883167
Depends on: 1883580
Depends on: 1883794
Depends on: 1884047
No longer depends on: 1882933
Depends on: 1882933
Depends on: 1884275
Depends on: 1884469
Depends on: 1884470
Depends on: 1884546
Depends on: 1884624
Depends on: 1884823
Depends on: 1885702
Depends on: 1885758
Depends on: 1886062
Depends on: 1887702
Depends on: 1887703
Depends on: 1887704
Depends on: 1887793
Depends on: 1887794
Depends on: 1887795
Depends on: 1888010
Depends on: 1888305
Depends on: 1888306
Depends on: 1888592
Depends on: 1888734
Depends on: 1888797
Depends on: 1888798
Depends on: 1888800
Depends on: 1889014
Depends on: 1889824
Depends on: 1890154
Depends on: 1890155
Depends on: 1890156
Depends on: 1890219
Depends on: 1890452
Depends on: 1890891
Depends on: 1890892
Depends on: 1891395
Depends on: 1891630
Depends on: 1892131
Depends on: 1892132
Depends on: 1892361
Depends on: 1892368
Depends on: 1892600
Depends on: 1892644
Depends on: 1893353
Depends on: 1893354
Depends on: 1893563
Depends on: 1893775
Depends on: 1893836
Depends on: 1893837
Depends on: 1893909
Depends on: 1894405
Depends on: 1894835
Depends on: 1895035
Depends on: 1895413
Depends on: 1895636
Depends on: 1895694
Depends on: 1895814
Depends on: 1896038
Depends on: 1896039
Depends on: 1896040
Depends on: 1896041
Depends on: 1896241
Depends on: 1896279
Depends on: 1896537
Depends on: 1896647
Depends on: 1897033
Depends on: 1897034
Depends on: 1897035
Depends on: 1897036
Depends on: 1897335
Depends on: 1897336
Depends on: 1897337
Depends on: 1897572
Depends on: 1897573
Depends on: 1897625
Depends on: 1897626
Depends on: 1897674
Depends on: 1898122
Depends on: 1898397
Depends on: 1898399
Depends on: 1898999
Depends on: 1899733
Depends on: 1900284
Depends on: 1900506
Depends on: 1900507
Depends on: 1900724
Depends on: 1901342
Depends on: 1901343
Depends on: 1901440
Depends on: 1901447
Depends on: 1901750
Depends on: 1901751
Depends on: 1901752
Depends on: 1901907
Depends on: 1902550
Depends on: 1902801
Depends on: 1902880
Depends on: 1902944
Depends on: 1903169
Depends on: 1903170
Depends on: 1903446
Depends on: 1903678
Depends on: 1903932
Depends on: 1904172
Depends on: 1904173
Depends on: 1904222
Depends on: 1904737
Depends on: 1904991
Depends on: 1905253
Depends on: 1905457
Depends on: 1905517
Depends on: 1905557
Depends on: 1905558
Depends on: 1906019
Depends on: 1906020
Depends on: 1906216
Depends on: 1906391
Depends on: 1906490
Depends on: 1906519
Depends on: 1907226
Depends on: 1908574
Depends on: 1908810
Depends on: 1908811
Depends on: 1908812
Depends on: 1908813
Depends on: 1909077
Depends on: 1909129
Depends on: 1909130
Depends on: 1909131
Depends on: 1909132
Depends on: 1909377
Depends on: 1909775
Depends on: 1909776
Depends on: 1910013
Depends on: 1910014
Depends on: 1910207
Depends on: 1910208
Depends on: 1910265
Depends on: 1910320
Depends on: 1910516
Depends on: 1910517
Depends on: 1911016
Depends on: 1911488
Depends on: 1911932
Depends on: 1911933
Depends on: 1912160
Depends on: 1912549
Depends on: 1912550
Depends on: 1912551
Depends on: 1912626
Depends on: 1913652
Depends on: 1913653
Depends on: 1913654
Depends on: 1913655
Depends on: 1913849
Depends on: 1914092
Depends on: 1914522
Depends on: 1914523
Depends on: 1914755
Depends on: 1914823
Depends on: 1914824
Depends on: 1915055
Depends on: 1915056
Depends on: 1915302
No longer depends on: 1865236
Depends on: 1915760
Depends on: 1916079
Depends on: 1916128
Depends on: 1916308
Depends on: 1916309
Depends on: 1916569
Depends on: 1916570
Depends on: 1917147
Depends on: 1917148
Depends on: 1917384
Depends on: 1917444
Depends on: 1917738
Depends on: 1917997
Depends on: 1917998
Depends on: 1917999
Depends on: 1918280
Depends on: 1918556
Depends on: 1918823
Depends on: 1918897
Depends on: 1918898
Depends on: 1918899
Depends on: 1918959
Depends on: 1919177
Depends on: 1919178
Depends on: 1919422
Depends on: 1919423
Depends on: 1919424
Depends on: 1919678
Depends on: 1919679
Depends on: 1919680
Depends on: 1919929
Depends on: 1920265
Depends on: 1931459
Depends on: 1931650
Depends on: 1931733
Depends on: 1931783
Depends on: 1932522
Depends on: 1934149
Depends on: 1934345
Depends on: 1935284
Depends on: 1936851
Depends on: 1937946
Depends on: 1939071
Depends on: 1939072
Depends on: 1939209
Depends on: 1939368
Depends on: 1940010
Depends on: 1940011
Depends on: 1941085
Depends on: 1941454
Depends on: 1941955
Depends on: 1942545
Depends on: 1942729
Depends on: 1943494
Depends on: 1944438
Depends on: 1946261
Depends on: 1951432
Depends on: 1951433
Depends on: 1951434
Depends on: 1951563
Depends on: 1952408
Depends on: 1953317
Depends on: 1954243
Depends on: 1954392
Depends on: 1957527
Depends on: 1965418
Depends on: 1965607
Depends on: 1965713
Depends on: 1966549
Depends on: 1966803
Depends on: 1966804
Depends on: 1966805
Depends on: 1967108
Depends on: 1967175
Depends on: 1967176
Depends on: 1968425
Depends on: 1968426
Depends on: 1968427
Depends on: 1968428
Depends on: 1968429
Depends on: 1968430
Depends on: 1968608
Depends on: 1968609
Depends on: 1968826
Depends on: 1969152
Depends on: 1969604
Depends on: 1969772
Depends on: 1969773
Depends on: 1969808
Depends on: 1969893
Depends on: 1970273
Depends on: 1970432
Depends on: 1970531
Depends on: 1970532
Depends on: 1970533
Depends on: 1970534
Depends on: 1970535
Depends on: 1970768
Depends on: 1971020
Depends on: 1971221
Depends on: 1971222
Depends on: 1971223
Depends on: 1971224
Depends on: 1971660
Depends on: 1971661
Depends on: 1971909
Depends on: 1972145
Depends on: 1972146
Depends on: 1972147
Depends on: 1972185
Depends on: 1972186
Depends on: 1972187
Depends on: 1972188
Depends on: 1972231
Depends on: 1972250
Depends on: 1972730
Depends on: 1972731
Depends on: 1973137
Depends on: 1973138
Depends on: 1973139
Depends on: 1973140
Depends on: 1973141
Depends on: 1973369

Suhaib, I think the bot needs to be tweaked a bit. A lot of the recent crashes seem to be happening with CPU info family 6 model 183 stepping 1 so it feels like it is mostly reporting crashes for broken CPUs. Thanks.

Flags: needinfo?(smujahid)
Depends on: 1973698
Depends on: 1973699
Depends on: 1973700
Depends on: 1973701
Depends on: 1973874
Depends on: 1974133

(In reply to Andrew McCreight [:mccr8] from comment #9)

Suhaib, I think the bot needs to be tweaked a bit. A lot of the recent crashes seem to be happening with CPU info family 6 model 183 stepping 1 so it feels like it is mostly reporting crashes for broken CPUs. Thanks.

I will reach out to you to add a heuristic to exclude such crashes.

Flags: needinfo?(smujahid)

To quantify that a bit, starting from bug 1971223, it looks like 10 out of the last 28 bugs filed were mostly or entirely with cpu info family 6 model 183 stepping 1.

Depends on: 1974548
Depends on: 1974549
Depends on: 1974550
Depends on: 1974594
Depends on: 1974595
Depends on: 1974596
Depends on: 1974669
Depends on: 1974900
Depends on: 1975098
Depends on: 1975343
Depends on: 1975344
Depends on: 1975613
Depends on: 1975614
Depends on: 1975846
Depends on: 1975847
Depends on: 1976134
Depends on: 1976375
Depends on: 1976376
Depends on: 1977012
Depends on: 1977518
Depends on: 1978184
Depends on: 1978235
Depends on: 1978257
Depends on: 1978490
Depends on: 1978491
Depends on: 1978722
Depends on: 1978723
Depends on: 1978724
Depends on: 1978976
Depends on: 1978977
Depends on: 1979215
Depends on: 1979476
Depends on: 1979545
Depends on: 1979588
Depends on: 1979589
Depends on: 1980035
Depends on: 1980540
Depends on: 1980754
Depends on: 1981276
Depends on: 1981882
Depends on: 1981883
Depends on: 1981884
Depends on: 1981885
Depends on: 1981886
Depends on: 1982199
Depends on: 1982260
Depends on: 1982261
Depends on: 1982481
No longer depends on: 1981886
Depends on: 1982752
Depends on: 1982753
Depends on: 1983194
Depends on: 1983397
Depends on: 1984218
Depends on: 1984876
Depends on: 1985176
Depends on: 1985422
Depends on: 1985911
Depends on: 1985912
Depends on: 1986223
Depends on: 1986815
Depends on: 1986817
Depends on: 1987047
Depends on: 1987237
Depends on: 1987541
Depends on: 1987556
Depends on: 1987557
Depends on: 1988067
Depends on: 1988394
Depends on: 1988418
Depends on: 1988655
Depends on: 1989195
Depends on: 1989726
Depends on: 1990131
Depends on: 1990708
Depends on: 1991312
Depends on: 1991313
Depends on: 1991570
Depends on: 1991804
Depends on: 1991805
Depends on: 1992303
Depends on: 1992304
Depends on: 1992536
Depends on: 1992537
Depends on: 1992640
Depends on: 1992641
Depends on: 1992642
Depends on: 1992904
Depends on: 1992905
Depends on: 1992906
Depends on: 1993143
Depends on: 1993144
No longer depends on: 1992537
Depends on: 1993634
Depends on: 1993635
Depends on: 1993828
Depends on: 1993861
Depends on: 1994091
Depends on: 1994566
Depends on: 1992537
Depends on: 1995209
Depends on: 1995443
Depends on: 1995444
Depends on: 1995662
Depends on: 1995663
Depends on: 1995906
Depends on: 1996157
Depends on: 1996391
Depends on: 1996491
Depends on: 1996736
Depends on: 1997015
Depends on: 1997479
Depends on: 1997480
Depends on: 1998055
Depends on: 1998056
Depends on: 1998057
Depends on: 1998058
Depends on: 1998302
Depends on: 1998303
Depends on: 1998576
Depends on: 1999061
Depends on: 1999112
Depends on: 1999157
Depends on: 1999158
Depends on: 1999159
Depends on: 1999416
Depends on: 1999518
Depends on: 1999580
Depends on: 1999843
Depends on: 1999844
Depends on: 2000123
Depends on: 2000124
Depends on: 2000341
Depends on: 2000475
Depends on: 2001245
No longer depends on: 1999112
Depends on: 2001760
Depends on: 2002899
Depends on: 2004555
Depends on: 2048947
Depends on: 2048948
Depends on: 2048949
Depends on: 2048950
Depends on: 2048951
Depends on: 2049146
Depends on: 2049217
Depends on: 2050311
No longer depends on: 2048949
Depends on: 2050693
Depends on: 2050694
Depends on: 2051098
Depends on: 2051099
Depends on: 2051773
Depends on: 2052077
Depends on: 2052624
Depends on: 2053292
Depends on: 2053962
Depends on: 2054317
Depends on: 2054777
Depends on: 2055162
Depends on: 2055379
Depends on: 2055380
Depends on: 2055381
Depends on: 2055725
Depends on: 2055726
Depends on: 2055981
Depends on: 2055982
Depends on: 2056047
Depends on: 2056116
Depends on: 2056569
Depends on: 2058349
Depends on: 2058350
Depends on: 2059283
Depends on: 2059284
Depends on: 2059285
Depends on: 2059286
Depends on: 2059607
Depends on: 2059892
Depends on: 2059982
Depends on: 2059983
Depends on: 2059984
Depends on: 2060852
Depends on: 2061035
Depends on: 2061910
Depends on: 2061989
Depends on: 2062102
Depends on: 2062103
Depends on: 2062472
Depends on: 2063493
Depends on: 2063766
Depends on: 2063934
Depends on: 2063935
Depends on: 2064269
Depends on: 2064576
Depends on: 2064957
Depends on: 2066303
Depends on: 2066304
Depends on: 2066953
Depends on: 2067323
Depends on: 2067959
Depends on: 2068305
Depends on: 2072794
Depends on: 2073966
Depends on: 2074323
Depends on: 2074702
Depends on: 2074703
Depends on: 2074704
You need to log in before you can comment on or make changes to this bug.