Enable Nimbus exposure ping for Firefox Suggest experiments
Categories
(Firefox :: Address Bar, enhancement)
Tracking
()
People
(Reporter: nanj, Assigned: nanj)
References
Details
Attachments
(1 file)
48 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-beta+
|
Details | Review |
Assignee | ||
Updated•3 years ago
|
Comment 1•3 years ago
|
||
Can this be prioritized so that it will make Firefox 92?
Assignee | ||
Comment 2•3 years ago
|
||
Yes, I am working on a patch for this right now, will get it uplifted to 92.
Comment 3•3 years ago
|
||
Thank you Nan!
Assignee | ||
Comment 4•3 years ago
|
||
Pushed by najiang@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e70a5656e054 Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio
Comment 6•3 years ago
|
||
bugherder |
Assignee | ||
Comment 7•3 years ago
|
||
Steps for QA:
- Enable Nimbus debug mode by setting
nimbus.debug
to true in "about:config" - Pick a Firefox Suggest experiment on Nimbus, and copy its
Preview URL
and paste it in Firefox to force-enroll into an experiment - Enter "about:telemetry#events-tab_parent" in the Urlbar to open the telemetry - event tab
- Verify there is an event recorded with the following fields:
- category:
normandy
- method:
expose
- object:
nimbus_experiment
- value: should be the experiment slug, such as
optin-firefox-suggest-history-vs-online
- extra object:
- branchSlug: should be the branch slug, e.g. "control" or "treatment"
- featureId: "urlbar"
- category:
Assignee | ||
Comment 8•3 years ago
|
||
Hi Cosmin,
Could you verify this new telemetry ping when you get back? Please see comment 7 for STRs. We might want to add this to the test cases for all Firefox Suggest experiments/rollouts as well.
Thanks!
Comment 9•3 years ago
|
||
Hi Nan! Sure, I have verified the "expose" telemetry and also I will update our test cases.
The "expose" telemetry event is correctly registered while enrolling in any of the FIrefox Suggest experiments/rollouts. I have verified this on the latest Nightly 93.0a1 (Build ID: 20210810213316) on Windows 10 x64, macOS 10.15.7 and Linux Mint 20.
Nan, I have observed that the "expose" telemetry event is registered after every Firefox restart. Can you please confirm that this is intended?
Assignee | ||
Comment 10•3 years ago
|
||
Nan, I have observed that the "expose" telemetry event is registered after every Firefox restart. Can you please confirm that this is intended?
Yes, that's the expected behavior of the exposure ping.
Comment 11•3 years ago
•
|
||
Hi Nan -- I haven't understood everything on this thread, but I just want to make sure we're on the same page about what an exposure ping is. The exposure ping should be sent:
- for a client enrolled in Firefox Suggest Offline or Firefox Suggest Online, when a Firefox Suggest is shown
- for a client enrolled in No Firefox Suggest or Firefox Suggest History, when the user enters a search query that would have triggered showing a Firefox Suggest if they had been in Offline or Online
Within a Nimbus experiment, we only care about the first date when a client sent the exposure ping.
Assignee | ||
Comment 12•3 years ago
|
||
for a client enrolled in Firefox Suggest Offline or Firefox Suggest Online, when a Firefox Suggest is shown
The ping is recorded when FS is activated for an enrolled user other than when an FS suggestion is shown. Recording it when FS is shown would be expensive to do due to its high overhead.
for a client enrolled in No Firefox Suggest or Firefox Suggest History, when the user enters a search query that would have triggered showing a Firefox Suggest if they had been in Offline or Online
It's impossible to detect "when the user enters a search query that would have triggered ..." if the user is enrolled into No FS or FS History, since Firefox would only download the search suggestions and queries if FS is activated either naturally or by experiments. So, instead, we record an exposure ping for those users who would have FS activated if they were enrolled in the treatment branch.
Within a Nimbus experiment, we only care about the first date when a client sent the exposure ping.
To my understanding, this ping will be recorded whenever the user is (or would have been) exposed to an experiment feature for each browser session during the experiment. It (its API) does not support only recording it once during the experiment.
Let me know what you think.
Comment 13•3 years ago
|
||
Thanks for the clarifications, Nan. What does it mean that FS is activated? Is the activation visible to the user?
Assignee | ||
Comment 14•3 years ago
|
||
That activation is not visible to the user.
Activation takes place when Firefox believes a user is qualified to see Firefox Suggest. Currently, that qualification is determined by the Nimbus experiment recipe (whether or not enrolled in a particular branch). When FS is activated, Firefox will go ahead download all the suggestions and queries from RS to build a search data structure to answer user's search queries in the future.
Comment 15•3 years ago
|
||
I see. I'm trying to understand the difference between enrollment and activation in terms of client attributes -- what would be an example of a client who enrolled in Firefox Suggest Offline, but didn't activate?
It sounds like this ping would still be helpful though, even though it is based on activation and not "exposure" in the experiments sense.
Assignee | ||
Comment 16•3 years ago
|
||
(In reply to Rebecca BurWei from comment #15)
I see. I'm trying to understand the difference between enrollment and activation in terms of client attributes -- what would be an example of a client who enrolled in Firefox Suggest Offline, but didn't activate?
For example, the enrolled user might have disabled the whole search suggestion feature (in about:preferences#search), which also includes Firefox Suggest. In which case, Firefox will not activate FS for that user.
Assignee | ||
Comment 17•3 years ago
|
||
[Tracking Requested - why for this release]: This would facilitate the experiment analysis for Firefox Suggest.
Assignee | ||
Comment 18•3 years ago
|
||
Comment on attachment 9235405 [details]
Bug 1724076 - Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio
Beta/Release Uplift Approval Request
- User impact if declined: This allows us to understand how many users are "exposed" to the feature for Firefox Suggest Nimbus experiments.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): This only enables the Nimbus "exposure" event for the Firefox Suggest experiments, it has no effect on the user-facing features.
- String changes made/needed: None
Updated•3 years ago
|
Comment 19•3 years ago
|
||
Comment on attachment 9235405 [details]
Bug 1724076 - Add nimbus exposure event for Firefox Suggest experiments r=adw,andreio
Approved for 92.0b5.
Comment 20•3 years ago
|
||
bugherder uplift |
Comment 21•3 years ago
|
||
I have verified this on the latest Beta 92.0b5 (Build ID: 20210817185605) on Windows 10 x64, macOS 10.15.7 and Linux Mint 20.
- The "expose" telemetry event is correctly registered while enrolling in any of the FIrefox Suggest experiments/rollouts.
Description
•