1943132 - Add Translations Telemetry Performance Metrics

Status: RESOLVED FIXED

(Firefox :: Translations, task)

Description

[Erik Nordin [:nordzilla]]

Description

We should add telemetry data for things like words-per-second. This will help us track which language models are performing well, and which language models may need improvements across various user devices.

---

Steps to Implement

• Add a new telemetry event to Translations metrics.yaml file.
• Update the TranslationsEngine to accumulate and log performance metrics to telemetry when it is destroyed.

---

Tests to Implement

• Ensure relevant metrics are reported on existing telemetry tests.
• Ensure that additional metrics and logging do not significantly impact performance.

Comment 1

[Erik Nordin [:nordzilla]]

Attachment: Bug 1943132 - Add Initial Translations Performance Telemetry r=#translations-reviewers!

This patch adds some initial telemetry to measure the performance
of our translations models, including metrics like words per second
and total time spent in active inference.

Comment 2

[Erik Nordin [:nordzilla]]

Attachment: Bug 1943132 - Add Translations Performance Telemetry Tests r=#translations-reviewers!

Adds an assertion to all Translations telemetry test cases
that ensure all reported performance event metrics (or the
lack thereof) are expected and have valid data.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/828202167b04
Add Initial Translations Performance Telemetry r=translations-reviewers,gregtatum
https://hg.mozilla.org/integration/autoland/rev/31d7e227cbba
Add Translations Performance Telemetry Tests r=translations-reviewers,gregtatum

Comment 4

[Sandor Molnar[:smolnar]]

Backed out for causing multiple telemetry related perma failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/d0ee9be05dc38001ee88a6210fb81bd2e5484feb

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | browser/components/translations/tests/browser/browser_translations_select_telemetry_change_both_languages_together.js

Failure log -> browser/components/search/test/browser/telemetry/browser_search_telemetry_remote_settings_sync.js

 Buffered messages finished
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - TEST-UNEXPECTED-FAIL | browser/components/translations/tests/browser/browser_translations_select_telemetry_change_both_languages_together.js | There should be 4 telemetry events of type translations.engine_performance - Got 3, expected 4
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - Stack trace:
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - chrome://mochikit/content/browser-test.js:test_is:1627
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - chrome://mochitests/content/browser/toolkit/components/translations/tests/browser/shared-head.js:assertEvent:2017
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - chrome://mochitests/content/browser/toolkit/components/translations/tests/browser/shared-head.js:assertTranslationsEnginePerformance:2126
[task 2025-01-28T16:16:25.197Z] 16:16:25     INFO - chrome://mochitests/content/browser/browser/components/translations/tests/browser/browser_translations_select_telemetry_change_both_languages_together.js:test_select_translations_panel_change_from_and_to_languages_before_retranslation:327
[task 2025-01-28T16:16:25.198Z] 16:16:25     INFO - chrome://mochikit/content/browser-test.js:handleTask:1147
[task 2025-01-28T16:16:25.198Z] 16:16:25     INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1219
[task 2025-01-28T16:16:25.198Z] 16:16:25     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1360
[task 2025-01-28T16:16:25.198Z] 16:16:25     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1136
[task 2025-01-28T16:16:25.198Z] 16:16:25     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058

Comment 5

[Erik Nordin [:nordzilla]]

Ugh, I had a clean try push prior to review.

I took some optional feedback on a micro-optimization to the code that ended up causing the testing to have some race conditions.

I'm just going to revert the micro-optimization for now, since it's not really worth what I would need to do to make the testing reliable.

I've rebased on top of Bug 1942349, which is currently queued for landing and has conflicts with this stack. Now I am waiting on a new try run to complete successfully before attempting to re-land.

Sorry about that!

Comment 6

[Erik Nordin [:nordzilla]]

The latest try run is looking clean. Sorry about that!

Going to try relanding.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1bf051f9c3d8
Add Initial Translations Performance Telemetry r=translations-reviewers,gregtatum
https://hg.mozilla.org/integration/autoland/rev/227f10d54061
Add Translations Performance Telemetry Tests r=translations-reviewers,gregtatum

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/1bf051f9c3d8
https://hg.mozilla.org/mozilla-central/rev/227f10d54061