1920702 - The "first Run" targeting criteria stopped working for mobile experiments and rollouts starting with Firefox 130.0 version

Status: RESOLVED FIXED

(Fenix :: Experimentation and Telemetry, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

[Affected versions]:

• Firefox Release 130.0
• Firefox RC 131.0

[Affected Platforms]:

• Android 13

[Prerequisites]:

• Have a mobile first Run Nimbus experiment or rollout created and launched that targets version 130 (example: Android default toolbar placement - Treatment A Rollout).

[Steps to reproduce]:

1. Install Firefox for Android version 130.
2. Open Firefox and dismiss the Onboarding.
3. Tap the Settings button and go to the “About Firefox” option.
4. Tap the Firefox logo 5 times and return to the Settings menu.
5. Tap the “Nimbus Experiments” option and select the Android default toolbar placement - Treatment A Rollout.
6. Observe the rollout’s branch.

[Expected result]:

• A checkmark is displayed next to the branch’s name.

[Actual result]:

• The checkmark is NOT displayed next to the branch’s name, meaning the build is not enrolled in the rollout.

[Notes]:

• This issue is reproducible starting with the Firefox 130.0 version.
• We used mozregression to find the regression window, but due to the app limitations we were only able to reach the following pushlog:
  https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2024.07.26&enddate=2024.07.27

Comment 1

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1920702 - Correctly record is_first_run as boolean r?chumphreys

Bug 1898552 added the recorded context, which recorded the "isFirstRun" param
as a boolean. However, the recorded context is splatted into the targeting
attributes inside the Nimbus SDK 1 and overrides other targeting members,
which include isFirstRun: String and is_first_run: bool. This has resulted
in targeting evaluation that checks against isFirstRun == "'true'" failing,
causing first run experiments to not enroll correctly. Recording the value as
is_first_run instead, which is expected to be a boolean, addresses the issue.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac4bb55b2a14
Correctly record is_first_run as boolean r=chumphreys,android-reviewers,jonalmeida

Comment 3

[Atila Butkovits]

Backed out for causing failures at RecordedNimbusContextTest.kt.

Backout link: https://hg.mozilla.org/integration/autoland/rev/b17724c0569ac0450e22599328506a163e74569e

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=ddciy7gMS5qBsZSpm8AW7w.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=73f7e0b593988ed5492e4012611cf2ee0c19df72

Failure log: https://treeherder.mozilla.org/logviewer?job_id=475794035&repo=autoland&lineNumber=13

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0d87b199f26
Correctly record is_first_run as boolean r=chumphreys,android-reviewers,jonalmeida

Comment 5

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/c0d87b199f26