1896093 - [Experiment] The Translate bottom sheet is not displayed on a translatable page if it was visited before enrollment

Status: NEW

(Fenix :: Translations, defect)

Description

[Ciprian Muresan [:cmuresan], Ecosystem QA]

[Affected versions]:

• Firefox Release 126.0 (Build #2016019338), hg-c23db1107632+, GV: 126.0-20240506203248, AS: 126.0
• Firefox Nightly 127.0a1 (Build #2016019818), hg-ee7969730d0c+, GV: 127.0a1-20240509094442, AS: 127.20240508173346

[Affected Platforms]:

• Android 13 - Samsung Galaxy A32.

[Prerequisites]:

• Have the nimbus-cli tool installed.
• Have adb (CLI) installed on your device.
• Have a device with USB or Wifi debugging enabled and connected to the device on which you installed nimbus-cli.
• Have the Nimbus CLI path added to the environment variable (Windows only).
• Have the latest Firefox Release app freshly installed on the device but left unopened.

[Steps to reproduce]:

1. Open the Firefox app and dismiss the onboarding.
2. Navigate to mozilla.org/fr.
3. Fully close the app.
4. Open the Firefox app using the following enroll command nimbus-cli -a fenix -c release enroll preview/full-page-translations-in-android --branch basic-translations-enabled.
5. Observe the bottom part of the screen.

[Expected result]:

• The Translation bottom sheet is displayed and the user is prompted to translate the page.

[Actual result]:

• The Translation option is displayed in the address bar, but it is not triggered.

[Notes]:

• The issue is no longer reproducible if the page is refreshed or if the same page is loaded in a separate tab.
• The issue is not reproducible with other pages after the first.
• Attaching a screen recording of the issue here.

Comment 1

[Olivia Hall [:olivia]]

Hi, thanks for reporting!

I have a question since I'm having trouble reproducing using the exact Nimbus CLI command (more info below).

Question:
In the video, I see a tab count of 1 on the main screen and usually no tabs have a tab count of 0. This might be something different between release and nightly, but could you confirm that when you tap 1 in the tab area when Nimbus relaunches the app, that the tab is not already open or loaded for mozilla.org/fr before tapping the Jump Back In suggestion? (Attached a screenshot below of how it usually looks when 0 tabs are present. I wonder if "Jump back In" has preloaded the tab.)

The reason I ask and for some more context, the bottom sheet automatically opens on these conditions:

• First visit in a session to the given host (to prevent popping up the bottom sheet too much)
• If the language of the page is detected as translatable
• If the user's preferred language is not the page language
  • Preferred languages are:
    • App Language
    • OS Language

If the tab is loaded first and redirected to the home screen, then the offer will not happen because the criteria First visit in a session to the given host won't have been met because that host was loaded once already.

When trying to use the exact command:
I get issues using nimbus-cli -a fenix -c developer enroll preview/full-page-translations-in-android --branch basic-translations-enabled of Error: Expected an array with key 'branches' in the JSONObject or Error: Expected a string with key 'slug' in the JSONObject with--no-validateset. I updated mynimbus-cli` as well. Let me know if you know the fix!

Even more oddly, I could not reproduce using a similar Nimbus command, which surprised me. I tried nimbus-cli --app fenix --channel developer test-feature translations --no-validate /Desktop/basic-translations-enabled.json where basic-translations-enabled.json is the same JSON as in the draft and the app offers a translation as expected. This makes me wonder if the other Nimbus command could be loading the tab.

Otherwise, this is very similar to bug 1893701. If the command isn't loading a tab ahead of time and changing the offer conditions, then the only solution option I can think of is we can add a GV API that calls resetHostsOffered when the experiment starts. This requires adding a GV API, AC API, and then connecting it in Fenix. The thought being, the opportunity to offer the translation has passed, so it won't happen for that reason when Nimbus starts. This is a pretty hefty API chain to add in, but one option.

Comment 2

[Olivia Hall [:olivia]]

Attachment: ZeroTabsUI.png

Added this as an example when nothing has been loaded. I'm suspecting the offer event didn't occur because the app already loaded the host.

Comment 3

[Olivia Hall [:olivia]]

Attachment: OfferWithNimbus.mp4

Similar Nimbus command using local JSON and flow not triggering the bug behavior. I wonder if the other Nimbus command is different than this one and causing the tab to pre-load based on the way it launches.

Comment 4

[Ciprian Muresan [:cmuresan], Ecosystem QA]

Hey @olivia, thanks for taking the time to look into this! I'll try my best to add as much context as I can.

I'll start with the command: the reason why it's not working now due to the experiment being moved out of the preview state so it's no longer available in the collection. I've uploaded a file here you can use this with some minor alterations. The command would be nimbus-cli -a fenix -c <channel> enroll --file <path-to-file> full-page-translations-in-android --branch basic-translations-enabled.

Moving on to the number of tabs. Initiually, there are 0 tabs as I'm attempting this as it's the first time I'm navigating to the page on the app. But then I fully close the app and reopen it. Once here, the 1 reflects the tab that I had open before and navigating to the page using the jump-back-in mechanic will expose the behavior.
I've tried using the same command that you used and I can reproduce the issue the same way. Load the page, close app, run command -> translation not offered. At this point I'm thinking it's either something tied to my specific device, or something that's on by default on the developer channel that might lead to us having different results.

Some more observations:

• I've noticed that if I navigate directly to a translatable page, the first one will not offer the translation, but a second one will (that or refreshing the same one)
• I've also noticed that if I navigate to a page before the translatable one, and navigate to it this way, the translation will be offered. E.g
  • Navigate to mozilla.org
  • Scroll down the page
  • Change language from english to french

| First visit in a session to the given host (to prevent popping up the bottom sheet too much)

Lastly, this gave me something to think about and I believe I found a way to hit this even more reliably.

• enroll in an experiment that turns the translations feature on
• navigate to mozilla.org/fr
• open a new tab and navigate to https://fr.m.wikipedia.org/
• leave both tabs in the tab tray and fully close the app
• reopen the app
• switch to the other tab

One of the tabs will offer to translate and the other won't. You can close the app again and restart it to have the same happen but the other way around. Now the tab that offered the translation does not offer it and the other one will.

Comment 5

[Olivia Hall [:olivia]]

Thanks so much for the detailed answer and explaining how the Nimbus preview state works! I'll check out the jump-back-in mechanic more, it sounds like a bigger factor than I originally thought. I'll link some leads to the bug to help find some connections too.

something that's on by default on the developer channel that might lead to us having different results.

Ohh, that is interesting, bug 1890763 landed towards the tail of v128 and also was a bug in offering. It wasn't as reproducible as this one, but has some similarities.

I've noticed that if I navigate directly to a translatable page, the first one will not offer the translation, but a second one will (that or refreshing the same one)

There could be a link to bug 1890763, which had "The translations pop-up is displayed only after a refresh or at the second site visit.". We thought cookies might be involved, but we discovered that a failure happened around the kiosk check sometimes and it was unclear why it failed sometimes and not others. I'll link the bugs in case there is some relation or something similar. Maybe this one is a consistent failure for the same reason.

One of the tabs will offer to translate and the other won't. You can close the app again and restart it to have the same happen but the other way around. Now the tab that offered the translation does not offer it and the other one will.

Interesting detail, thanks for mentioning!