1920702 - The "first Run" targeting criteria stopped working for mobile experiments and rollouts starting with Firefox 130.0 version

Status: VERIFIED FIXED

(Firefox for Android :: Experimentation and Telemetry, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Affected versions]:

• Firefox Release 130.0
• Firefox RC 131.0

[Affected Platforms]:

• Android 13

[Prerequisites]:

• Have a mobile first Run Nimbus experiment or rollout created and launched that targets version 130 (example: Android default toolbar placement - Treatment A Rollout).

[Steps to reproduce]:

1. Install Firefox for Android version 130.
2. Open Firefox and dismiss the Onboarding.
3. Tap the Settings button and go to the “About Firefox” option.
4. Tap the Firefox logo 5 times and return to the Settings menu.
5. Tap the “Nimbus Experiments” option and select the Android default toolbar placement - Treatment A Rollout.
6. Observe the rollout’s branch.

[Expected result]:

• A checkmark is displayed next to the branch’s name.

[Actual result]:

• The checkmark is NOT displayed next to the branch’s name, meaning the build is not enrolled in the rollout.

[Notes]:

• This issue is reproducible starting with the Firefox 130.0 version.
• We used mozregression to find the regression window, but due to the app limitations we were only able to reach the following pushlog:
  https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2024.07.26&enddate=2024.07.27

Comment 1

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1920702 - Correctly record is_first_run as boolean r?chumphreys

Bug 1898552 added the recorded context, which recorded the "isFirstRun" param
as a boolean. However, the recorded context is splatted into the targeting
attributes inside the Nimbus SDK 1 and overrides other targeting members,
which include isFirstRun: String and is_first_run: bool. This has resulted
in targeting evaluation that checks against isFirstRun == "'true'" failing,
causing first run experiments to not enroll correctly. Recording the value as
is_first_run instead, which is expected to be a boolean, addresses the issue.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac4bb55b2a14
Correctly record is_first_run as boolean r=chumphreys,android-reviewers,jonalmeida

Comment 3

[Atila Butkovits]

Backed out for causing failures at RecordedNimbusContextTest.kt.

Backout link: https://hg.mozilla.org/integration/autoland/rev/b17724c0569ac0450e22599328506a163e74569e

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=ddciy7gMS5qBsZSpm8AW7w.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=73f7e0b593988ed5492e4012611cf2ee0c19df72

Failure log: https://treeherder.mozilla.org/logviewer?job_id=475794035&repo=autoland&lineNumber=13

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0d87b199f26
Correctly record is_first_run as boolean r=chumphreys,android-reviewers,jonalmeida

Comment 5

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/c0d87b199f26

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:beth, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox131 to wontfix.

For more information, please visit BugBot documentation.

Comment 7

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on latest Firefox Nightly 132.0a1 (Build 2016046778), hg-519a5ec62f8c+ GV: 132.0a1-20240926212653 AS: 132.20240925050359 on Android 13, Google Pixel 4 device.

• In order to verify this issue, I used the following nimbus-cli command and can confirm that I got enrolled in the rollout that has the first Run flag set in the targeting criteria.

.\nimbus-cli.exe --app fenix --channel nightly enroll android-default-toolbar-placement-treatment-a-rollout --branch treatment-a --no-validate --preserve-targeting

Comment 8

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1920702 - Correctly record is_first_run as boolean r?chumphreys

Bug 1898552 added the recorded context, which recorded the "isFirstRun" param
as a boolean. However, the recorded context is splatted into the targeting
attributes inside the Nimbus SDK 1 and overrides other targeting members,
which include isFirstRun: String and is_first_run: bool. This has resulted
in targeting evaluation that checks against isFirstRun == "'true'" failing,
causing first run experiments to not enroll correctly. Recording the value as
is_first_run instead, which is expected to be a boolean, addresses the issue.

Original Revision: https://phabricator.services.mozilla.com/D223399

Comment 9

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: Users will not be able to enroll in first run experiments on Android
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: Already QE-ed
• Risk associated with taking this patch: Low
• Explanation of risk level: Patch has been verified to fix the issue by QE. At worst it would just continue to prevent people from enrolling in first run experiments (ie no change)
• String changes made/needed: None
• Is Android affected?: yes

Comment 10

[[e-mail]]

Hi there - what can we do to get this into the 131 dot release? This bug is currently blocking our first run experimentation.

Comment 11

[Dianna Smith [:diannaS]]

(edited- Sorry for the mistake!) This is being considered for the 131 mobile dot release which is slated to rollout next week.

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/d630881ae811

Comment 13

[Carmen Fat [:cfat] - Ecosystem QA]

I have verified this issue on the Android first Run recipes that are currently live:

• Android default toolbar placement - Treatment A Rollout
  -> Firefox RC 132.0 (Build #2016051562), hg-0e15e2edd460+ GV: 132.0-20241021175835 AS: 132.0.1
  -> Firefox Release 131.0.2
• Android Mobile Sync Screen Skip Copy Optimization V2
  -> Firefox RC 132.0 (Build #2016051562), hg-0e15e2edd460+ GV: 132.0-20241021175835 AS: 132.0.1

The users are now correctly enrolled in the experiments and rollouts that have the first Run targeting criteria in their recipes.