1756177 - [Experiment] The "VPN promo" message is wrongly displayed even if you are connected to "Firefox Sync" with a "Mozilla VPN" account

Status: VERIFIED FIXED

(Firefox :: Messaging System, defect, P2)

Description

[Marius Coman [:mcoman], Ecosystem QA]

[Affected versions]:

• Firefox 98.0b6 Build ID: 20220217185748

[Affected Platforms]:

• Windows 10 x64
• macOS 11.6.2
• Linux Mint 20.2 x64

[Prerequisites]:

• Have a Firefox profile with the "browser.search.region" pref set to "US" in the "about:config" page.
• Have the "user.js" file copied to your PC.
• You are connected to "Firefox Sync" with a FxA account that was used with "Mozilla VPN" in the last 30 days.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Go to the recently created profile folder and paste the "user.js" from the prerequisites.
3. Restart the browser.
4. Open a new "Private Window" and observe the displayed elements.

[Expected result]:

• The following elements are displayed:
• "Firefox" logo;
• In-content "Search Bar";
• The "Private window: Firefox clears your search and browsing history when you close all private windows. This doesn’t make you anonymous." string.

[Actual result]:

• The "VPN promo" message is also displayed

Comment 1

[Punam Dahiya [:pdahiya]]

Should be off train fix that can be handled in targeting

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

This is a missing targeting clause in the experiment, and something that needs to be exposed in nimbus.py in experiment, which I will be doing a pull request on shortly.

I think the targeting criteria I'm going to add will be related attachedFxAOAuthClients[.id == "e6eb0d1e856335fc"]|mapToProperty("lastAccessedDaysAgo"), however, my wife's machine (which has VPN on it and the VPN has not been accessed in at least 30 days, though she's still subscribed) has that property set to 0. So I need to dig into some pieces related to the semantics of that property first.

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

As it turns out, there had been a login in Mozilla VPN yesterday when I looked at that machine, which was why it was set to 0. So

!("e6eb0d1e856335fc" in attachedFxAOAuthClients|mapToProperty("id")) ||
(attachedFxAOAuthClients[.id == "e6eb0d1e856335fc"].lastAccessedDaysAgo > 29)

is going to be part of the targeting.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

As it turns out, unlike its JavaScript cousin, the JEXL || operator does not short-circuit. So I ended up using this for the message targeting.

"targeting": "('e6eb0d1e856335fc' in attachedFxAOAuthClients|mapToProperty('id')) ? (attachedFxAOAuthClients[.id == 'e6eb0d1e856335fc'].lastAccessedDaysAgo > 29) : true",

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

https://github.com/mozilla/experimenter/pull/7097 contains a patch for this so we can use it on the experimenter side, which, with luck, will be used for the release experiment. There's an integration test issue which still needs to be sorted there, however, and my local integration test setup is having some issues of its own.

Venetia and I have talked it over, and in the interest of getting the beta experiment out there sooner rather than later, we're going to switch it from "No enterprise and no VPN in the past 30 days" to "No enterprise and no VPN in the past".

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

As of right now, the experiment in preview is using "No enterprise and no VPN in the past", and I've updated the mana page to reflect that for the beta experiment.

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

After a long discussion with one of the VPN folks, it seems that the way we're detecting VPN use actually can't tell us whether it's been used in the last 30 days. I've talked to Venetia, and we'll be using exactly the same targeting that we've got on the beta experiment for the 98 release experiment, and updating another already-filed bug to make future sure future campaign actually do better, in particular by telling us whether that user is a VPN subscriber, and we may well want to expose this as an experiment targeting criterion as well. I'll update this bug with more details and links to the others in the coming week.

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

This was fixed by the change described in comment 7.

I've spun off bug 1765899 at making the code better at avoiding showing the promo to folks particularly unlikely to be interested.

Comment 9

[Marius Coman [:mcoman], Ecosystem QA]

Marking this issue as VERIFIED based on comment 7.