1756152 - [Experiment] The “VPN Promo” message is wrongly displayed more than once per week

Status: VERIFIED FIXED

(Firefox :: Messaging System, defect, P1)

Description

[Simona Rosu [:srosu], Ecosystem QA]

Attachment: VpnPromo_lifetime.gif

[Affected versions]:

• Firefox 98.0b6 Build ID: 20220217185748

[Affected Platforms]:

• Windows 10 x64
• MacOS 11.6
• Ubuntu 20.04

[Prerequisites]:

• Have the Firefox browser en-US build installed/extracted.
• Have the “browser.search.region” pref set to the “US” value.
• Have the user.js file copied to your PC.

[Steps to reproduce]:

1. Create a new Firefox profile but do not open it.
2. Go to the recently created profile folder and paste the "user.js" from the prerequisites.
3. Open the Firefox profile.
4. Open a new "Private Window".
5. Open a new tab and observe the displayed elements.
6. Open the third new tab and observe the displayed elements.

[Expected result]:

• Step 5 and 6: The “VPN Promo” message is NOT displayed.

[Actual result]:

• Steps 5 and 6: The “VPN Promo” message is displayed.

[Notes]:

• Based on the Mana page, the “VPN promo” message should be displayed only 1 time/ week.
• The “VPN Promo” message is displayed up to 3 times.
• Attached is a screen recording of the issue.

Comment 1

[Punam Dahiya [:pdahiya]]

Recipe seems to have correct frequency capping and lifetime capping set https://experimenter.services.mozilla.com/nimbus/validate-private-browsing-mode-pbnewtab-as-experimentation-surface/details

"custom": [
{
"cap": 1,
"period": 604800000
}
],
"frequency": {
"lifetime": 3
}, ...

NI @andreio for feedback if values look correct and help debug thanks

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

The problem was that the custom object was outside of the frequency object; I've fixed it in the experiment that is now in preview mode.

It's possible that having the schema set-up on experimenter and stage will help catch issues like this, and Andrei and I are planning on tightening up the schema to catch more issues as well. I'll be sure that we consider this too, if it's not already caught by the existing schema.

Comment 3

[Simona Rosu [:srosu], Ecosystem QA]

I‘ve verified this issue using Firefox Beta 98.0b8 (Build ID: 20220222185824) on Windows 10 x64, macOS 11.6, and Ubuntu 20.04.

• The “VPN Promo” message is not displayed more than once a week.