1897321 - Add minVersion variable to whatsNewPage Nimbus feature

Status: RESOLVED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Shane Hughes [:aminomancer]]

Right now, we have to use the confusing min version targeting in Nimbus. This is not ideal for two reasons:

1. It limits the audience to only users who are already upgraded to the major version before the version our WNP targets (say, min: 125, target: 126), but users who don't update frequently can just as easily upgrade from an earlier version to the latest version.
2. It's confusing for beta experiments, because min: 125 actually means any version that starts with 125. and that includes e.g. 125.0b1. So users who upgrade from 125.0b1 to 125.0 RC (remember that RCs are shipped to beta channels at the end of the cycle, the week before the 126 beta builds start shipping) will actually see a WNP and our experiment will modify it. So that forces us to change the min version to 126 for a beta experiment, so as to only include users who are already on a 126 beta build.

Punam and I discussed and analyzed this feature at length, and we realized that minVersion (which we don't have) may be more important than maxVersion (which we do have). Actually, I now think maxVersion is not particularly useful and may be unnecessary (not that we should remove it). This variable asserts that the update version is below the specified version, not the starting version. It's only evaluated after an update, just before a WNP is shown. This makes it unnecessary because we'd only run WNP experiments targeting the most recent release. If the latest release is 127, then that's the one we're targeting, and there's no need to exclude 128+ because 128 does not exist yet. And if 128 has been released, then why are we targeting 127? Once 128 has been released, it's no longer possible to upgrade to 127 (without manual hackery that we don't need to account for) because Firefox will always update to the most recent eligible version. And if an experiment targeting 127 runs so long that it covers 126, 127, and 128, then that means it needs to be ended anyway, because once 128 releases it's no longer realistically possible for the 127 WNP to show, as the 128 WNP will show instead.

However, minVersion really would be useful. Unlike enrollment targeting, minVersion would be evaluated after the update, but before the WNP shows. That means we could configure the recipe to enroll everyone, regardless of versions, but set minVersion to 127 so that you can only see it if you upgraded specifically to 127. If you were on version 99 for some reason, and then finally update while 127 is the latest release, then you will see the WNP for 127, and we want the experiment to modify that. This way we include all users who will see a WNP when upgrading to 127, rather than needlessly excluding users who started on an earlier version.

It's important that this be minVersion and not version because there are potentially multiple versions that can receive a given WNP. If you were on 126 beta and you upgrade to 127.0b1, we will show you a WNP. If you were on 126.0 and you wait around a while before upgrading, you'll end up updating to 127.0.1 (planned dot release), and we will show you a WNP. So we can't say version: 127.0 because that excludes dot releases. Keep in mind that WNPs don't only show on major releases like 127.0 - they show for any major upgrade, meaning any time your version number changes by an integer (and apparently also any time a beta build upgrades to an RC, though this needs more testing). So, the WNP is not simply part of the 127.0 release specifically. It's a consequence of upgrading from an earlier major version to a newer major version.

We can still include min and max version targeting in experiments if we feel a need to for some reason. And maxVersion may still be a useful precautionary feature to prevent an experiment from having negative effects if it is mistakenly left running too long. My point is just that the combination of maxVersion and min version targeting is inadequate to control who sees the experimental treatment.

I think this is a high priority. I would like to fix this ASAP, since we have plans to run release experiments. It's a small patch, 2 or 3 points.

Comment 1

[Hanna Alemu [:hanna_a]]

Attachment: Bug 1897321 - Add minVersion variable to whatsNewPage Nimbus feature r=#omc-reviewers

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3da3a7f2530f
Add minVersion variable to whatsNewPage Nimbus feature r=omc-reviewers,firefox-desktop-core-reviewers ,aminomancer

Comment 3

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3da3a7f2530f

Comment 4

[Marius Coman [:mcoman], Ecosystem QA]

Hi Hanna, thanks for bringing this enhancement to our attention!

Considering that this enhancement is part of the “whatsNewPage” feature config, the testing part falls under the Feature Configuration Testing, a process implemented by the Ecosystem QA to thoroughly test all the feature configurations, new and old.
For testing, we will be creating experiments and rollouts using the feature configuration in the test on the Nimbus Stage server, in order to validate its capabilities with different configurations. We will also be verifying the experiment and rollout interactions if they will be used at the same time.

Based on the above we will create a Jira ticket for the “whatsNewPage” feature config and will track the testing process there.

Comment 5

[Hanna Alemu [:hanna_a]]

@Marius Thank you! This is another bug: Bug 1898361 that can be verified along with this one as well.

Comment 6

[Marius Coman [:mcoman], Ecosystem QA]

@Hanna, thanks for the link! We have created a Jira ticket that will cover the testing process for the “whatsNewPage” feature config. You can find it here: QA-2475. Considering that both issues will be covered in this ticket I will remove the "qe-verify+" flag from this enhancement.

That being said, we will wait for Bug 1898361 to land before starting testing this feature config.