1942745 - 340.16 - 9.56% browser_ml_autofill_perf.js AUTOFILL-model-run-latency / browser_ml_autofill_perf.js AUTOFILL-model-run-latency + 1 more (Linux, Windows) regression on Fri January 10 2025

Status: RESOLVED WONTFIX

(Core :: Machine Learning, defect)

Description

[Treeherder Bug Filer]

Perfherder has detected a mozperftest performance regression from push b80614c34a34fb55a00d4713727ef315aec6b5ea. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
340%	browser_ml_autofill_perf.js AUTOFILL-model-run-latency	windows11-64-shippable-qr		10.17 -> 44.75
23%	browser_ml_summarizer_perf.js SUM-XENOVA-DISTILBART-CNN-12-6_BIG-total-memory-usage	linux1804-64-shippable		638.56 -> 788.46
10%	browser_ml_autofill_perf.js AUTOFILL-model-run-latency	linux1804-64-shippable		17.00 -> 18.62
3%	browser_ml_summarizer_perf.js SUM-XENOVA-DISTILBART-CNN-12-6_SMALL-model-run-latency	windows11-64-shippable-qr		8,903.50 -> 9,213.46

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
15%	browser_ml_summarizer_perf.js SUM-ONNX-COMMUNITY-QWEN2.5-0.5B-INSTRUCT_BIG-total-memory-usage	macosx1015-64-shippable-qr		1,828.54 -> 1,546.17

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 43318

The following documentation link provides more information about this command.

For more information on performance sheriffing please see our FAQ.

If you have any questions, please do not hesitate to reach out to [e-mail].

Comment 1

[Mayank Bansal]

Does form-filling with ML use WebGPU on Nightly?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1938659

Comment 3

[Erich Gubler [:ErichDonGubler]]

Mayank: I wasn't aware of ML-enhanced search in Firefox, but just based on the title, Mozilla might be experimenting with it. Note that this would be an issue for the folks working on ML at Mozilla, not the WebGPU team directly.

NI'ing Mardak and Tarek, who are listed as contacts for the Core: Machine Learning component. I presume that this "regression" is in response to support for WebGPU in dedicated and shared worker contexts landing, which makes sense, but ultimately, this seems like a new baseline that the Machine Learning team might need to accept.

Comment 4

[Ed Lee :Mardak]

cc vasish adding the ml_autofill_perf test in bug 1931883 with the largest regression but this is indeed nightly only for now

tarek, as in comment 3, i expect this likely will be a new baseline for the summarizer as well unless we shouldn't be running these with webgpu anyway (which would also be a different baseline)

Comment 5

[Vasish Baungally]

For form autofill, this is ok for now since it's only Nightly. We are currently only looking at accuracy metrics and the like and will look at next steps / wrapping up the expriment in the incoming week.

Comment 6

[Tarek Ziadé (:tarek)]

yeah we're refactoring a bunch of things and we're closely looking at perf, we can ignore those alerts for now thanks