1869924 - Translation does not work, repeats single words in the target language

Status: UNCONFIRMED

(Firefox :: Translations, defect, P3)

Description

[Martin Weinelt [:hexa]]

Attachment: mozilla_org_translated.png

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0

Steps to reproduce:

Use the Bergamot "Translate page..." feature from the burger menu, to translate the page e.g. from english to german, but other combinations also fail.

Actual results:

The translation result is garbage. Sentences are replaced by one word each, which gets repeated multiple times.

Expected results:

The correct translation result should appear.

Comment 1

[Martin Weinelt [:hexa]]

Note that translation works on fresh profiles, but not on preexisting ones. We use the nixpkgs Firefox 120.0.1 source build, and can reproduce this issue on multiple machines and different users.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox::Translation' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Erik Nordin [:nordzilla]]

Attachment: translations-with-different-profiles.png

Hi Martin,

Thanks for reporting this.

The screenshot that you posted is definitely concerning, and I'm not sure yet what might be causing such an occurrence in a release build.

I tried to reproduce this on my end and was unable to.

---

Operating System
Ubuntu 22.04.3 LTS

User Agent
Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0

---

I opened up Firefox in two different profiles, new and pre-existing, and successfully translated https://mozilla.org from English to German on both profiles (see attached screenshot).

One difference, however, is that I did not get my Firefox build from nixpkgs.

I have some follow-up questions for you:

1. Are you able to reproduce on any Firefox builds that are not from nixpkgs?

2. Can you share any more anonymous detail about the what qualities the pre-existing profiles may have in common?

3. You said you were able to reproduce this on multiple machines with different users. Would you be able to give us a minimum steps to reproduce, especially if you are able to create a brand new profile and somehow modify it to become "pre-existing" in a ways that cause it exhibit the behavior you see?

Thanks!

Comment 4

[Martin Weinelt [:hexa]]

1. I'm afraid I don't have any machines that don't use NixOS/nixpkgs.

2. I'm not sure what makes these profiles special yet

3. What I meant is that this wasn't just a blip in a single profile for a single machine for a single user. I'm not sure yet how to reproduce this situation.

I tried creating a new profile at 118.0, upgrading to 119.0, 119.0.1, 120.0 and 120.1 - but translation keeps on working. So I'm stumped.

Comment 5

[Martin Weinelt [:hexa]]

I remembered that we have the Firefox binary distribution from Mozilla packaged up, and indeed I cannot reproduce it with that.

Seems the issue lies with our source build, but I wonder what could affect this.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:nordzilla, could you have a look please?

For more information, please visit BugBot documentation.

Comment 7

[Greg Tatum [:gregtatum]]

Due to the difficulty in reproducing and the workarounds, I'm marking as S3.

Comment 8

[Malte Jürgens [:maltejur]]

Attachment: Screenshot_20240125_123209.png

This is easily reproducible for me with the official Firefox source tarball and a new profile.

1. Download and extract https://ftp.mozilla.org/pub/firefox/releases/122.0/source/firefox-122.0.source.tar.xz
2. Add ac_add_options --without-wasm-sandboxed-libraries to the mozconfig (I am not sure if this is part of the problem, but it makes the compilation easier)
3. Run ./mach build && ./mach run in the extracted directory
4. Once Firefox is built, open https://google.de and accept the translation prompt

Expected result: The page gets translated into something readable
Actual result: Lots of repeating words that do not make sense, see screenshot

Comment 9

[ImperatorStorm]

Adding ac_add_options --without-wasm-sandboxed-libraries to mozconfig doesn't appear to be part of the issue, as NixOS' firefox doesn't appear to use that option.

Comment 10

[Erik Nordin [:nordzilla]]

:mjurgens,

I followed the steps building Firefox 122 from source on my Ubuntu machine, but I was unable to reproduce.

---

Greg, I know that this is somewhat related to Bug 1875542, where you discovered that there was an issue with the fallback gemm implementation, but that was not shipped to release channels before the issue was fixed.

Do you think it's possible that something in our the NixOS configuration is triggering this fallback gemm implementation defect?

Comment 11

[Greg Tatum [:gregtatum]]

It seems like the same behavior, so most likely it is an issue in the matrix math operations for whatever the configuration is. I'm not sure on the specific things to investigate without looking into it more and what codepath it is specifically going through. Someone could capture a performance profile and see the matrix math operations, for instance on translating a big page like a wikipedia page.

Comment 12

[Malte Jürgens [:maltejur]]

Here is a profile of this happening on a wikipedia article: https://share.firefox.dev/3UtDgLJ. I am not sure if it is of much help though, because I am only seeing wasm-function calls.

This is happening on my Arch Linux machine, so it does not seem to be limited to NixOS only. I'll try to reproduce it in a fresh Ubuntu VM as well.

Comment 13

[Greg Tatum [:gregtatum]]

So this is going down the fallbackGemm path. We could do a local debug build of the translation engine and get the stacks to see the specific call site. It seems worth doing.

https://searchfox.org/mozilla-central/rev/26dd94a024f21971e539a8650c2f177323c98fe2/toolkit/components/translations/bergamot-translator/bergamot-translator.js#3495-3507

Comment 14

[Greg Tatum [:gregtatum]]

To clarify, the wasm will be same on different platforms, so we don't need to reproduce the issue, just reproduce the same type of profile with the fallbackGemm call.

Comment 15

[Chas Belov]

I'm seeing something similar on the translate-to page where it is repeating words in the translate-from language on a page in Chinese Wikipedia:

Steps:

1. Go to 曹雅雯 (permalink to current version)
2. Click the translate icon in the address bar
3. Translate Chinese (Simplified) to English
4. Click Translate

Expected result: I get some manner of English translation

Actual result: One of the intro paragraphs reads:

2021 debuted for 10 years, self-reliance studio "good 聞音樂"[[3]. In the same year, he entered the 圍 八項獎 八項獎獎巨獎巨獎獎八獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎獎 with the best [4]嵂奪成成成成成成成

as a translation of:

2021年出道10年之際，自立門戶組工作室「好聞音樂」[3]。同年以專輯《自本》入圍第32屆金曲獎八項大獎，[4]最終奪得最佳台語女歌手獎及最佳台語專輯獎。

Part of the issue might be that the article is actually using Chinese (Traditional) characters, but I would expect a translation from Chinese to work with either Traditional or Simplified since they are usually the same Unicode code points.

Comment 16

[Ina Popescu, Desktop QA]

I was able to reproduce the behavior described in comment 15 on macOS 15 with Firefox 1430.1, but this doesn’t confirm the underlying issue, so I’m not updating the status for now. Please let us know if there is a simplified test case to reproduce the original issue.