Closed Bug 1698684 Opened 3 years ago Closed 3 years ago

Consider leaving users enrolled in experiments if only a subset of prefs are changed

Categories

(Firefox :: Normandy Client, enhancement, P3)

enhancement

Tracking

()

RESOLVED FIXED
90 Branch
Tracking Status
firefox-esr78 --- wontfix
firefox87 --- wontfix
firefox88 --- wontfix
firefox89 --- wontfix
firefox90 --- fixed

People

(Reporter: mythmon, Assigned: mythmon)

References

(Depends on 2 open bugs)

Details

Attachments

(2 files)

Currently, if a user changes any preference that is being modified by an experiment, user-branch or default-branch, they are unenrolled from that experiment. Generally this is a reasonable approach for single preference experiments, though not right in all situations.

However, for multi preference experiments, it produces strange results. The intent is that on unenrollment, everything is set back to how it was before the experiment, respecting any user choice (though bug 1698683 describes how this is not quite true). This isn't the right choice for some experiments though, which would prefer to continue tracking partial treatments. It may be the case that the different preferences are independently useful, and so there is no reason to stop the N-1 other experiments just because 1 is not longer useful. Alternatively, we may be interested in how users change these preferences over time, and unenrolling makes this harder to determine.

We should consider if this is still the right choice to make. Other ideas include some combination of:

  • Only unenroll if all preferences have been changed, not matching the experiment values.
  • Allow changes to user branch prefs when the experiment is using the default branch.
  • Report when users change preferences, but never unenroll because of it.
  • Do nothing and keep the existing system.

I've written a more detailed proposal to never unenroll because of preference change here (public gdoc link): https://docs.google.com/document/d/18VkZrGBhfC8jVH6fjSX6kx5TI4lhL50iwhNENYwK0XU/edit?ts=605fcbf0#

My current disposition is to proceed with implementation of this proposal soon.

Apologies for not reading the doc sooner. I don't have permission to modify it, so I'll just comment here—by "not unenroll" I think this means both that a) users will retain whatever part of the user experience the experiment delivers that doesn't depend on the pref that changed, and b) unenroll telemetry will not be sent before the recipe is withdrawn. Instead, we'll see an expPrefChanged event.

On both counts I think that makes a lot of sense; the new behavior would be consistent with our analysis model and this also makes the user experience more consistent -- after this change, a user changing one setting won't lead to them crashing out of the experiment, which could have weird action-at-a-distance effects on settings they didn't mean to change.

I wonder if there are safety concerns; this might make it more likely for users to have any combination of pref values, and it would be sad if a combination we didn't test caused startup crashes or something. I think that sounds unlikely?, but maybe it's something to flag if someone is doing something exotic.

So: sounds great to me; thanks for writing it up.

Thanks for the feedback, Tim. Your summary of the behavior is correct, so we're on the same page.

Apologies for not reading the doc sooner. I don't have permission to modify it, so I'll just comment here

Oops. I've modified the preferences and now you (and anyone else following along) should be able to comment and make suggestions on the document. Hopefully not too many people were discouraged from giving feedback by this.

I wonder if there are safety concerns; this might make it more likely for users to have any combination of pref values, and it would be sad if a combination we didn't test caused startup crashes or something. I think that sounds unlikely?, but maybe it's something to flag if someone is doing something exotic.

That's a good point. I can't see a way to avoid hitting those problems (if they exist) besides review during the advising process. However, I agree it sounds unlikely. I think that if there is some combination of preference values that we sending out in a recipe that could produce a major problem, then we have a fairly large bug in the feature.

This reverses a decision that was made in the original design of Normandy's preference experiments. That decision was made assuming simple experiments that only change one preference.

Today, preferences are more complicated, have more interactions, and often change more than one preferences. Unenrolling users from experiments due to preferences changing causes bugs, unexpected experiences for users, and bad experiment outcomes.

This change will make Normandy act more in alignment with how experimenter owners and data analysts already treat it, improving the overall experience of the experiments program.

Assignee: nobody → mcooper
Status: NEW → ASSIGNED
Attachment #9215844 - Attachment description: WIP: Bug 1698684 - Leave users in Normandy experiments when prefs change → Bug 1698684 - Leave users in Normandy experiments when prefs change
Attachment #9215844 - Attachment description: Bug 1698684 - Leave users in Normandy experiments when prefs change → WIP: Bug 1698684 - Leave users in Normandy experiments when prefs change
Attachment #9217477 - Flags: data-review?(tdsmith)
Attachment #9215844 - Attachment description: WIP: Bug 1698684 - Leave users in Normandy experiments when prefs change → Bug 1698684 - Leave users in Normandy experiments when prefs change

Comment on attachment 9217477 [details]
data-request-1698684.md

Sorry, I was out last week and didn't change my bugzilla handle.

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in the probe definition files and the Probe Dictionary.

  1. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

  1. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, mythmon.

  1. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

I think under the theory that this reflects a human decision this is cat 2 but that doesn't affect the review.

  1. Is the data collection request for default-on or default-off?

Default-on.

  1. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

  1. Is the data collection covered by the existing Firefox privacy notice?

Yes.

  1. Does there need to be a check-in in the future to determine whether to renew the data?

No.

  1. Does the data collection use a third-party collection tool?

No.

Attachment #9217477 - Flags: data-review?(tdsmith) → data-review+
Pushed by mcooper@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c407d725866
Leave users in Normandy experiments when prefs change r=Gijs
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 90 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: