1895714 - Improve Pref Experiment Workflow

Status: NEW

(Fenix :: Experimentation and Telemetry, task)

Description

[Beth Rennie [:beth] (she/her)]

Right now to run a pref setting experiment on Fenix, one needs to do the following:

1. Add the feature to the Fenix Nimbus manifest
2. Add a pref instance to GeckoRuntimeSettings
3. Add a getter and setter for that member
4. Add a member to the GeckoEngine settings object in android-components and assign it in the .apply block
5. Update the DefaultSettings ctor to set this new member based on a value from FxNimbus

On Desktop, we annotate the feature manifest with what prefs each variable will set, which is much simpler to (a) implement and (b) remove when your experimentation is complete.

I don't think this doesn't belong in the Nimbus SDK bug tracker. Any solution here will likely have to be in client code

Comment 1

[Jonathan Almeida [:jonalmeida]]

Sorry, I completely forgot that there is an alternative route for engine experiments. There is an ExperimentDelegate on the GeckoRuntime, that can fetch the experiment data from the embedding application.

This might reduce the steps needed for running the kinds of experiments that you want, but this issue is probably still valid to see if this option can also be improved.

Comment 2

[Beth Rennie [:beth] (she/her)]

Yeah that could likely be used to set prefs in a data driven manner like how its done for Desktop

Comment 3

[Andrew Creskey [:acreskey]]

ExperimentDelegate added in bug 1845824

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Fenix::Experimentation and Telemetry' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 5

[Andrew Creskey [:acreskey]]

I'm going to make some gecko preferences, currently exposed as Nimbus features, accessible for pref flip experiments in Fenix.

There are about a half dozen preferences in the networking and security components.

I don't need exposure counting, just the ability to have the pref's set via experiment enrollment.

I'm wondering what's a better route -- the steps Beth described in comment 0, which seem straightforward.

Or via the ExperimentDelegate which Olivia added in bug 1845824. I don't fully understand how it binds prefs but it seems interesting.

Olivia -- do you have any thoughts on this?

Comment 6

[Beth Rennie [:beth] (she/her)]

(In reply to Andrew Creskey [:acreskey] from comment #5)

I'm going to make some gecko preferences, currently exposed as Nimbus features, accessible for pref flip experiments in Fenix.

There are about a half dozen preferences in the networking and security components.

I don't need exposure counting, just the ability to have the pref's set via experiment enrollment.

I'm wondering what's a better route -- the steps Beth described in comment 0, which seem straightforward.

Or via the ExperimentDelegate which Olivia added in bug 1845824. I don't fully understand how it binds prefs but it seems interesting.

Olivia -- do you have any thoughts on this?

The ExperimentDelegate does not do that for you automatically. It lets you query feature values at runtime, so you could set prefs yourself with whatever API calls are available.

Comment 7

[Andrew Creskey [:acreskey]]

(In reply to Beth Rennie [:beth] (she/her) from comment #6)

(In reply to Andrew Creskey [:acreskey] from comment #5)

I'm going to make some gecko preferences, currently exposed as Nimbus features, accessible for pref flip experiments in Fenix.

There are about a half dozen preferences in the networking and security components.

I don't need exposure counting, just the ability to have the pref's set via experiment enrollment.

I'm wondering what's a better route -- the steps Beth described in comment 0, which seem straightforward.

Or via the ExperimentDelegate which Olivia added in bug 1845824. I don't fully understand how it binds prefs but it seems interesting.

Olivia -- do you have any thoughts on this?

The ExperimentDelegate does not do that for you automatically. It lets you query feature values at runtime, so you could set prefs yourself with whatever API calls are available.

Ah, I see.
Let me go through the Fenix Feature Manifest to improve my understanding of this workflow.

Comment 8

[Andrew Creskey [:acreskey]]

FYI, my first attempt at getting a pref-setting Nimbus experiment setup on Fenix, bug 1925390.