Closed Bug 1724076 Opened 3 years ago Closed 3 years ago

Enable Nimbus exposure ping for Firefox Suggest experiments

Categories

(Firefox :: Address Bar, enhancement)

enhancement

Tracking

()

VERIFIED FIXED
93 Branch
Tracking Status
firefox92 + verified
firefox93 --- verified

People

(Reporter: nanj, Assigned: nanj)

References

Details

Attachments

(1 file)

No description provided.
Assignee: nobody → najiang

Can this be prioritized so that it will make Firefox 92?

Yes, I am working on a patch for this right now, will get it uplifted to 92.

Thank you Nan!

Pushed by najiang@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e70a5656e054
Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 93 Branch

Steps for QA:

  • Enable Nimbus debug mode by setting nimbus.debug to true in "about:config"
  • Pick a Firefox Suggest experiment on Nimbus, and copy its Preview URL and paste it in Firefox to force-enroll into an experiment
  • Enter "about:telemetry#events-tab_parent" in the Urlbar to open the telemetry - event tab
  • Verify there is an event recorded with the following fields:
    • category: normandy
    • method: expose
    • object: nimbus_experiment
    • value: should be the experiment slug, such as optin-firefox-suggest-history-vs-online
    • extra object:
      • branchSlug: should be the branch slug, e.g. "control" or "treatment"
      • featureId: "urlbar"

Hi Cosmin,

Could you verify this new telemetry ping when you get back? Please see comment 7 for STRs. We might want to add this to the test cases for all Firefox Suggest experiments/rollouts as well.

Thanks!

Flags: needinfo?(cmuntean)

Hi Nan! Sure, I have verified the "expose" telemetry and also I will update our test cases.

The "expose" telemetry event is correctly registered while enrolling in any of the FIrefox Suggest experiments/rollouts. I have verified this on the latest Nightly 93.0a1 (Build ID: 20210810213316) on Windows 10 x64, macOS 10.15.7 and Linux Mint 20.

Nan, I have observed that the "expose" telemetry event is registered after every Firefox restart. Can you please confirm that this is intended?

Status: RESOLVED → VERIFIED
Flags: needinfo?(cmuntean) → needinfo?(najiang)

Nan, I have observed that the "expose" telemetry event is registered after every Firefox restart. Can you please confirm that this is intended?

Yes, that's the expected behavior of the exposure ping.

Flags: needinfo?(najiang)

Hi Nan -- I haven't understood everything on this thread, but I just want to make sure we're on the same page about what an exposure ping is. The exposure ping should be sent:

  • for a client enrolled in Firefox Suggest Offline or Firefox Suggest Online, when a Firefox Suggest is shown
  • for a client enrolled in No Firefox Suggest or Firefox Suggest History, when the user enters a search query that would have triggered showing a Firefox Suggest if they had been in Offline or Online

Within a Nimbus experiment, we only care about the first date when a client sent the exposure ping.

for a client enrolled in Firefox Suggest Offline or Firefox Suggest Online, when a Firefox Suggest is shown

The ping is recorded when FS is activated for an enrolled user other than when an FS suggestion is shown. Recording it when FS is shown would be expensive to do due to its high overhead.

for a client enrolled in No Firefox Suggest or Firefox Suggest History, when the user enters a search query that would have triggered showing a Firefox Suggest if they had been in Offline or Online

It's impossible to detect "when the user enters a search query that would have triggered ..." if the user is enrolled into No FS or FS History, since Firefox would only download the search suggestions and queries if FS is activated either naturally or by experiments. So, instead, we record an exposure ping for those users who would have FS activated if they were enrolled in the treatment branch.

Within a Nimbus experiment, we only care about the first date when a client sent the exposure ping.

To my understanding, this ping will be recorded whenever the user is (or would have been) exposed to an experiment feature for each browser session during the experiment. It (its API) does not support only recording it once during the experiment.

Let me know what you think.

Thanks for the clarifications, Nan. What does it mean that FS is activated? Is the activation visible to the user?

That activation is not visible to the user.

Activation takes place when Firefox believes a user is qualified to see Firefox Suggest. Currently, that qualification is determined by the Nimbus experiment recipe (whether or not enrolled in a particular branch). When FS is activated, Firefox will go ahead download all the suggestions and queries from RS to build a search data structure to answer user's search queries in the future.

I see. I'm trying to understand the difference between enrollment and activation in terms of client attributes -- what would be an example of a client who enrolled in Firefox Suggest Offline, but didn't activate?

It sounds like this ping would still be helpful though, even though it is based on activation and not "exposure" in the experiments sense.

(In reply to Rebecca BurWei from comment #15)

I see. I'm trying to understand the difference between enrollment and activation in terms of client attributes -- what would be an example of a client who enrolled in Firefox Suggest Offline, but didn't activate?

For example, the enrolled user might have disabled the whole search suggestion feature (in about:preferences#search), which also includes Firefox Suggest. In which case, Firefox will not activate FS for that user.

[Tracking Requested - why for this release]: This would facilitate the experiment analysis for Firefox Suggest.

Comment on attachment 9235405 [details]
Bug 1724076 - Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio

Beta/Release Uplift Approval Request

  • User impact if declined: This allows us to understand how many users are "exposed" to the feature for Firefox Suggest Nimbus experiments.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This only enables the Nimbus "exposure" event for the Firefox Suggest experiments, it has no effect on the user-facing features.
  • String changes made/needed: None
Attachment #9235405 - Flags: approval-mozilla-beta?

Comment on attachment 9235405 [details]
Bug 1724076 - Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio

Approved for 92.0b5.

Attachment #9235405 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

I have verified this on the latest Beta 92.0b5 (Build ID: 20210817185605) on Windows 10 x64, macOS 10.15.7 and Linux Mint 20.

  • The "expose" telemetry event is correctly registered while enrolling in any of the FIrefox Suggest experiments/rollouts.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: