1830080 - Firefox builds can no longer switch back to the Prod collection after switching to the Nimbus Preview one

Status: VERIFIED FIXED

(Firefox for Android :: Experimentation and Telemetry, defect)

Description

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

[Affected versions]:

• Firefox Beta 113.0b7 (Build #2015946547), 0cbbf39d7a+, GV: 113.0-20230423180101, AS: 97.4.1
• Firefox Release 112.2.0
• Firefox Release 111.1.1

[Affected Platforms]:

• Android 13 - Samsung Galaxy A32.
• Android 11 - OnePlus 6T

[Prerequisites]:

• Have Nimbus experiments available in Prod preview.

[Steps to reproduce]:

1. Open the profile from the prerequisites.
2. Tap the Kebab Menu > Settings > About Firefox.
3. Tap 5 times on the Firefox logo.
4. Tap the Back button.
5. Scroll down to the “Secret Settings” option and tap on it.
6. Toggle the Nimbus Preview option.
7. Fully close and reopen the app.
8. Repeat Steps 2 through 4 and navigate to the "Nimbus Experiments" option.
9. Tap the option.
10. Fully close and reopen the app.
11. Repeat Steps 2 through 4 and navigate to the "Nimbus Experiments" option.
12. Tap the option.
13. Go back and scroll to the “Secret Settings” option and tap on it.
14. Toggle the Nimbus Preview option.
15. Fully close and reopen the app.
16. Repeat Steps 2 through 4 and navigate to the "Nimbus Experiments" option and observe the list of items.

[Expected result]:

• A list of experiments that are available in the regular collection are displayed.

[Actual result]:

• The list of experiments that are available in the Preview collection are displayed.

[Notes]:

• This issue severely impacts QA as we use the Preview collection to test upcoming experiments.
• Attaching a screen recording of the issue here.

Comment 1

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments (#1882)

Comment 2

[BMO Github Automation]

Authored by https://github.com/jhugman
https://github.com/mozilla-mobile/firefox-android/commit/22d398fe20df0b0d19fa3ca6de196ac74acb311c
[main] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Is this something we need to backport to v113?

Comment 4

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

I have verified that the issue is no longer reproducible using a custom Nightly build made from the latest changes available in the main branch (114.0a1 (Build #2015948579), c3dda7c59c+, GV: 114.0a1-20230503214103, AS: 114.20230504050432). I verified that the app can switch back from the Preview collection on the following devices:

• Samsung Galaxy S9 - Android 9
• OnePlus 5 - Android 10
• OnePlus 6T - Android 11
• Samsung Galaxy A23 - Android 12
• Samsung Galaxy A32 - Android 13

@James, if still possible can we get this uplifted to 113? This would help QA with any Android experiments during the Fx 113 cycle.

Comment 5

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments (backport #1882) (#1902)

Comment 6

[James Hugman [:jhugman] [@jhugman]]

Comment on attachment 9331583 [details] [review]
[mozilla-mobile/firefox-android] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments (backport #1882) (#1902)

Beta/Release Uplift Approval Request

• User impact if declined: This makes it harder for QA to test off-train experiments.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The changes have been tested, and this only affects codepaths exercised by QAs.
• String changes made/needed:
• Is Android affected?: Yes

Comment 7

[James Hugman [:jhugman] [@jhugman]]

:cmuresan ^^^ uplifted to v113.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9331583 [details] [review]
[mozilla-mobile/firefox-android] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments (backport #1882) (#1902)

Approved for 113.0rc2.

Comment 9

[BMO Github Automation]

Authored by https://github.com/jhugman
https://github.com/mozilla-mobile/firefox-android/commit/d360843eb14ae7b09ab089b5b5bd65f5d8112d64
[releases_v113] Bug 1830080 Add more careful handling of preview-collection in maybeFetchExperiments

Comment 10

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

I have verified that the issue is not reproducible with the latest Firefox Release 113.0 build (link). I have used the following devices for verification:

• Samsung Galaxy S9 - Android 9
• OnePlus 5 - Android 10
• Samsung Galaxy A32 - Android 13

Thanks for this fix and uplift @James!