Open Bug 1980560 Opened 1 year ago Updated 9 days ago

[META] Stress-tests created using LLMs (Chatgpt, Grok and others) where Firefox is slow or comparatively slower.

Categories

(Core :: Performance: General, task)

Product:
Core
Component:
Performance: General
Type:
task

Tracking

()

People

(Reporter: mayankleoboy1, Unassigned)

References

(Depends on 217 open bugs, Blocks 1 open bug)

Details

(Keywords: ai-involved, meta)

This is a meta-bug to track all the stress-tests I create using LLMs like ChatGPT. The testcases span across different components, so this META in Core:General seems appropriate.

Depends on: 1979594
Type: defect → task
Depends on: 1980757
Depends on: 1980991
Depends on: 1967068
Depends on: 1981000
Depends on: 1981650
Depends on: 1981659
Depends on: 1981690
Summary: [META] Stress-tests created using LLMs (Chatgpt, Grok and others) where Firefox is slow or comparitively slower. → [META] Stress-tests created using LLMs (Chatgpt, Grok and others) where Firefox is slow or comparatively slower.
Depends on: 1981799
Depends on: 1981909
Depends on: 1982196
Depends on: 1982216
Depends on: 1982220
Depends on: 1982245
Depends on: 1982247
Depends on: 1982487
Depends on: 1982536
Depends on: 1982547
Depends on: 1982596
Depends on: 1982611
Depends on: 1982758
Depends on: 1982510
Depends on: 1982771
Depends on: 1972259, 1972255, 1977111, 1977112, 1977116, 1978218
Depends on: 1983019
Depends on: 1983021
Depends on: 1983215
Depends on: 1983228
Depends on: 1983233
Depends on: 1983371
Depends on: 1983734
Depends on: 1983886
Depends on: 1982246
Depends on: 1985489
Depends on: 1973339
Depends on: 1974902
Depends on: 1986711
Depends on: 1986962
Depends on: 1977030
Depends on: 1965507
Depends on: 1968172
Depends on: 1952778
Depends on: 1957329
Depends on: 1987291

I wonder we could use our fuzzer infra to generate this kind of perf tests, because it looks like we are basically doing the same - generating random scripts, run them, and report in case there's an issue. Generating tests from our own Web IDL would be able to cover new APIs that LLM is not yet trained for, or to not cover what Firefox does not support.

Flags: needinfo?(mayankleoboy1)
Flags: needinfo?(jkratzer)

This bug has over 100 dependencies, and most of them are still open, so I'm not sure there's a big rush to generate more bugs that we're not going to fix. Finding quadratic behavior in the browser is fun, but I'm not sure whether we should prioritize it if no actual web pages are hitting it.

I can share what i have seen since the time i started this:

  1. The "number of iterations" is based on trial and error. For some APIs/functions/calls/whatever, number as low as 5000 is enough to show bad behaviour. For others numbers as large as 50000000 is needed. It depends on how the browser behaves for that APIs/functions/calls/whatever.
  2. In many cases, the browser will freeze for multiple minutes or use uncontrolled explosive memory. Or it will do neither, but will just use 100% of one CPU for multiple minutes which doesnt freeze the browser nor does it uses memory. The only way i have seen is to check the resource usage on Task Manager.
  3. What is the definition of 'bad' ? Whether bad in absolute terms, bad in comparison with other browser(s) ? Bad for realistic iteration counts? What is realistic iteration count?
  4. Some of the bugs which are "bad" are uninteresting because the iteration count is pathological, completely unrealistic. What would be the filter criteria for these bugs? If every issue that the fuzzers find are filed as bugs, i suspect the number of bugs will become explosively high.
  5. Root of a lot of bugs i have found are around (AFAIU) cleanup/cc-type-things, backpressure on IPC/IPDL, nsTarray ... These are sort of known in the sense that these are tuned towards realistic browser use (or realistic browser benchamrks like SP3). Optimizing for these extreme cases may be bad for the general use, if possible at all.
  6. IF we go the route of fuzzers doing this, these known issues must be fixed first. Else i suspect majority of bugs would be around these things only.
Flags: needinfo?(mayankleoboy1)

I think often times benchmarks log operations-per-second rather than the total time, rather than setting an iteration number and run the full iterations. That would solve many listed issues, except 3 - that still needs some definition.

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #1)

I wonder we could use our fuzzer infra to generate this kind of perf tests, because it looks like we are basically doing the same - generating random scripts, run them, and report in case there's an issue. Generating tests from our own Web IDL would be able to cover new APIs that LLM is not yet trained for, or to not cover what Firefox does not support.

This is something the fuzzers can do in some capacity and we have filed bugs like this in the past. The issue we've had is that there's varying level of appetite for fixing bugs like this.

Flags: needinfo?(jkratzer)
Depends on: 1995099
Depends on: 1995181
Depends on: 1995565
Depends on: 1996868
Depends on: 1997529
Depends on: 1997545
Depends on: 1997702
Depends on: 1997703
Depends on: 1997744
Depends on: 1997778
Depends on: 1997798
Component: General → Performance: General
Depends on: 1998179
Depends on: 1999120
Depends on: 1999164
Depends on: 1999608
Depends on: 1999610
Depends on: 1999639
Depends on: 2000163
Depends on: 2000355
Depends on: 2000414
Depends on: 2000444
Depends on: 2000450
Depends on: 2000532
Depends on: 1964311
Depends on: 1987175
Depends on: 2001109
Depends on: 2001164
Depends on: 2001168
Depends on: 2001307
Depends on: 1964988
Depends on: 2001544
No longer depends on: 2001544
Depends on: 2001853
Depends on: 2001858
Depends on: 2001563
Depends on: 2002701
Depends on: 2003217
Depends on: 2003227
Depends on: 2003408
Depends on: 2005459
Depends on: 1965872
Depends on: 2005979
Depends on: 2005982
Depends on: 2005983
Depends on: 2005989
Depends on: 2007073
Depends on: 2007229
Depends on: 2007248
Depends on: 2007250
Depends on: 2007251
Depends on: 2007724
Depends on: 2007726
Depends on: 2007727
Depends on: 2007841
Depends on: 2007970
Depends on: 2007992
Depends on: 2007994
Depends on: 2011013
Depends on: 2011080
Depends on: 2011593
Depends on: 2013749
Depends on: 2013750
Depends on: 1965426
Depends on: 1962839
Depends on: 2015006
Depends on: 2015256
Depends on: 2015317
Depends on: 2016070
Depends on: 2016905
Depends on: 2016958
Depends on: 2019566
Depends on: 2023352
Depends on: 2023499
Depends on: 2023831
Depends on: 2024134
Depends on: 2024309
Depends on: 2024829
Depends on: 2024849
Depends on: 2024863
Depends on: 2025088
Depends on: 2025141
Depends on: 2025143
Depends on: 2025236
Depends on: 2025245
Depends on: 2025253
Depends on: 2026846
Depends on: 2026860
Depends on: 2026861
Depends on: 2027230
Depends on: 2028357
Depends on: 2028759
Depends on: 2029107
Depends on: 2029396
Depends on: 2031160
Depends on: 2032667
Depends on: 2033303
Depends on: 2033376, 2033372
Depends on: 2033981
Depends on: 2033994
Depends on: 2034349
Depends on: 2034696
Depends on: 2034699
Depends on: 2034976
Depends on: 2035809
Depends on: 2036334
Depends on: 2036572
Depends on: 2036611
Depends on: 2038095
Depends on: 2038408
Depends on: 2038535
Depends on: 2038545
Depends on: 2038966
Depends on: 2039484
Depends on: 2039481
Depends on: 2039524
Depends on: 2040549
Depends on: 2040585
Depends on: 2040731
Depends on: 2041193
Depends on: 2041318
Depends on: 2041354
Depends on: 2041378
Depends on: 2041877
Depends on: 2041885
Depends on: 2042294
Depends on: 2043890
No longer depends on: 2043890
Depends on: 2043979
Depends on: 2044891
Depends on: 2044896
Depends on: 2044952
Depends on: 2045194
Depends on: 2045200
Depends on: 2045423
Depends on: 2045420
Depends on: 2045461
Depends on: 2045545
Depends on: 2045600
Depends on: 2045537
Depends on: 2045720
Depends on: 2046327
Depends on: 2046334
Depends on: 2047481
Depends on: 2048156
Depends on: 2048188
Depends on: 2045536
Depends on: 2048847
See Also: → 2048847
Depends on: 2048856
See Also: → 2048856
See Also: 2048847 →
See Also: 2048856 →
Depends on: 2048877
Depends on: 2048903
Depends on: 2049195
Depends on: 2049565
Depends on: 2049588
Depends on: 2050741
Depends on: 2051002
Depends on: 2051203, 2051197, 2051194, 2051189, 2051035, 2050729
Depends on: 2051215
No longer depends on: 2051194
No longer depends on: 2051035
No longer depends on: 2051203
No longer depends on: 2051215
Depends on: 2045455
You need to log in before you can comment on or make changes to this bug.