1839641 - [Experiment] The browser toolbar is still displayed in the bottom part of the browser even if client is enrolled in the "treatment-a" branch of the "android-default-toolbar-placement" experiment

Status: RESOLVED DUPLICATE of bug 1872941

(Firefox for Android :: Toolbar, defect)

Description

[Marius Coman [:mcoman], Ecosystem QA]

Attachment: rec of the issue.gif

[Affected versions]:

• Firefox Beta 115.0b8 (Build #2015957699)

[Affected Platforms]:

• Android 12

[Affected Devices]:

• Samsung Galaxy A23 5G

[Prerequisites]:

• Have "nimbus-cli" installed from here.
• Have the mobile device connected to your PC.

[Steps to reproduce]:

1. Open a terminal and enter the nimbus-cli --app fenix --channel beta enroll preview/android-default-toolbar-placement --branch control --preserve-targeting --preserve-bucketing command.
2. Go to your mobile device and observe the browser's toolbar placement.

[Expected result]:

• The browser toolbar is displayed in the top part of the browser.

[Actual result]:

• The browser toolbar is still displayed in the bottom part of the browser.

[Notes]:

• This issue is also reproducible with a custom build created from the "releases_v115" branch.
• Attached a screen recording of the issue.

Comment 1

[[:skhan]]

The fix will be available in Fenix Beta 115.0b9 which happens later tonight. As of now it is available in the "releases_v115" branch.

Comment 2

[[:skhan]]

Ready to be tested in Firefox beta 115.0b9

Comment 3

[Marius Coman [:mcoman], Ecosystem QA]

Attachment: rec of the issue.gif

Hi, @skhan! I have retested this issue using Firefox Beta 115.0b9 (Build #2015958107) installed on Samsung Galaxy A23 5G (Android 12) and it seems that it’s not fully fixed.
Now, when the browser is opened for the first time, a client enrolled in the "treatment-a" branch will see that the "Top" option from the "Pick your toolbar placement" Onboarding section is selected but the toolbar is still placed in the bottom part of the browser.

However, the toolbar is moved to the top part of the browser if a New Tab is opened, the "Start browsing" button from the Onboarding screen is tapped, or if the "Top" option from the "Pick your toolbar placement" Onboarding section is tapped again.

Attached a screen recording of the new behavior.

Comment 4

[[:skhan]]

I suspect in your testing using nimbus-cli ( realized experiment isn't launched in beta, it was just in preview), it becomes a race for testing the first run experiment so even though toolbar and the radio button uses the same setting, toolbar gets the original default value (bottom) and as soon as experiment is downloaded (through nimbus-cli), the nimbus default value (top) is reflected and is shown on the radio button. I tested locally through Android Studio and the experimental json in beta and there is no issue.

STEPS TO REPRODUCE:

1. Add the experimental json to initial_experiments.json (attaching the json from the experiment) in the code
2. Add debuggable = true, autosignReleaseWithDebugKey to local.properties
3. Change the build variant to fenixBeta and build
4. Half of the time it will reflect bottom (control branch ), half the time top (treatment-a branch). Make sure to clear cache in between launches

Comment 5

[[:skhan]]

Attachment: Experiment json in beta

Comment 6

[[:skhan]]

The other way is to wait till the experiment is launched into beta and then test it.

Comment 7

[Marius Coman [:mcoman], Ecosystem QA]

Hi, Sarah, and thanks for addressing this issue so quickly!

I can confirm that after following the steps you provided in comment 4 the issue is no longer reproducible and the toolbar is correctly displayed in the top part of the browser while being enrolled in the "treatment-a" of the experiment.

However, considering that this race condition was caused by the "nimbus-cli" feature and the fact that we will use this tool a lot on future mobile experiments, we suggest changing the bug scope to either add capabilities in "nimbu-cli" to support this type of first-run experiments or change the feature configuration implementation to work together with nimbus-cli.

Comment 8

[James Hugman [:jhugman] [@jhugman]]

Hi :skhan;

We'd like to ensure that the nimbus-cli can be used to test these experiments.

Commenting out the first read of the shouldUseBottomToolbar pref causes the app to work as expected under nimbus-cli enroll.

This would suggest that either:

1. the startup and preference metrics should be recorded after visual completeness, instead of during initGlean AND/OR:
2. the LazyPreference.getValue() is actually doing a write when key is not in preferences already.

Rewriting LazyPreference.getValue() to check if the key is in preferences causes the app to work as expected under nimbus-cli enroll.

To resolve this bug, please could you make a judgment which approach to take.

Comment 9

[[:skhan]]

I will debug and take a look on exactly what is happening. Do a follow up here and open a new task/bug based on the results and approach.

Comment 10

[Roger Yang [:royang]]

(In reply to James Hugman [:jhugman] [@jhugman] from comment #8)

Commenting out the first read of the shouldUseBottomToolbar pref causes the app to work as expected under nimbus-cli enroll.

Does this means that anything that uses lazyFeatureFlagPreference will have the same issue if it's read to early?

Comment 11

[Roger Yang [:royang]]

Hi :jhugman;
As discussed, lazyFeatureFlagPreference cannot be modified because the write portion of the behaviour is important. This is to reserve consistency even if the user did not make a explicit choice.

As you stated above, the only other solution is to move where startup and preference metrics is recorded. Is this the only solution left?
If so we'll need to look into moving them and test to make sure nothing is broken.

Comment 12

[[:skhan]]

(In reply to Marius Coman [:mcoman], Ecosystem QA from comment #7)

Hi, Sarah, and thanks for addressing this issue so quickly!

I can confirm that after following the steps you provided in comment 4 the issue is no longer reproducible and the toolbar is correctly displayed in the top part of the browser while being enrolled in the "treatment-a" of the experiment.

However, considering that this race condition was caused by the "nimbus-cli" feature and the fact that we will use this tool a lot on future mobile experiments, we suggest changing the bug scope to either add capabilities in "nimbu-cli" to support this type of first-run experiments or change the feature configuration implementation to work together with nimbus-cli.

Can be directly tested in the latest RC build too : https://firefox-ci-tc.services.mozilla.com/tasks/BcL_gXuzQIyhE68i1Huc4Q

Comment 13

[James Hugman [:jhugman] [@jhugman]]

As you stated above, the only other solution is to move where startup and preference metrics is recorded. Is this the only solution left?

You could fix lazyFeatureFlagPreference so that it doesn't write, then do an explicit write in the places that depends on this behavior. e.g. at the time the old-style onboarding is dismissed.

Should we change the bug title to: "Make the toolbar feature work with nimbus-cli"?

Comment 14

[Roger Yang [:royang]]

Closing this issue as this will be fixed through the off train experiment meta. https://bugzilla.mozilla.org/show_bug.cgi?id=1872941