1634841 - No Show settings for messaging layer for Experiment Manager

Status: ASSIGNED

(Firefox :: Messaging System, enhancement, P1)

Description

[Su-Young Hong]

Summary

We need to be able to specify a no-show branch in all messaging experiments. The no-show branch is a branch that doesn't show any messages for the messaging layer / group being experimented on. It differs from control in that control shows the default setting for that message layer, while the no show shows nothing.

This is necessary because we need to be able to answer, for all messaging experiments, "this is the effect of message X versus not showing any message". That would allow us to learn long term (by having a consistent comparison across different experiments), and identify any issues with attention stewardship.

---

What we need:

We need the ability to do the following, in descending order of importance (order is my opinion):

• Turn off ALL messaging system messages for users in a branch
• Turn off all messaging system message of one (or more) particular group(s) (CFR, moments, etc) for users of a branch
• Turn off one (or more) messages (by message_id) for users of a branch

Comment 1

[Kate Hudson :k88hudson]

Shell, is this required for any of the 81 (shirley) experiments?

Comment 2

[:shell escalante]

Jim is working with Shirley on normal CFR experiments. For Shirley this will not be used - but should be added to future requirements.

Comment 3

[Beth Rennie [:beth] (she/her)]

This really isn't a Nimbus bug. If this is still needed it should be implemented as part of the Messaging System's integration with the Experiment Manager and/or configured as part of the messaging system features.

I'm kicking this backing to the fxms triage queue for priorization within OMC.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

This seems like it could be useful to think about high-level messaging approaches and optimize at that level.

Alex, I'd love to hear your thoughts here on whether we want to spend this time/effort, either sooner or later. This is already technically straightforward, though a small amount of added busy work.

Comment 5

[Alex Davis [:adavis]]

Sorry for the slow reply, still catching up from the holidays.

When we used to do this it provided three benefits:
1- Measure if onboarding was better than no onboarding. We still had some doubts then.
2- Allow us to measure our cumulative impact vs no onboarding at all.
3- It provided us with more sensitivity to move our metrics. It is somewhat easier to detect a change from "no-show" than the latest experience. This was valuable because we had a lot of trouble having impact back then with onboarding.

Today:
1- We know onboarding is better than no onboarding. When we regress our current experiences, we hurt our metrics.
2- It's more valuable for us to measure our cumulative impact vs the start of the year than no onboarding at all.
3- I think we enroll enough users to detect differences and we have a better understanding of what is more likely to improve the user experience vs what won't.
4- When we introduce new messages, our control normally has no message (so we do end up doing something along these lines when relevant).

I will acknowledge that always having a "no-show" makes it a bit easier to compare between experiments but there will always be doubt and nuance when comparing different periods. This is not something we regularly want to do.

As of Firefox 123, Nimbus will be offering the capability for a long-term holdback which will allow us to understand the cumulative impact of winning experiments over some time without resorting back to "no-show". This covers #2 from that list.

Should we add the capability? It wouldn't hurt but it's much less needed than it used to be. It will also require more enrollments for each experiment.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

Wonderful; thanks for the detailed info, Alex! While you make a compelling case for onboarding, I'm curious what you think about the other surfaces (spotlights, infobar, feature callout, etc). Spotlights, being the most intrusive, seem like a particularly interesting question to ask about in the large...

Comment 7

[Alex Davis [:adavis]]

I'm curious what you think about the other surfaces (spotlights, infobar, feature callout, etc).

The long-term holdback will include those too so it will allow us to understand the overall impact of messages in the browser.

When we introduce new messages, we normally already test them again "no-message" and we tend to only roll-out a message if it has had some positive and significant impact. We're looking to also survey some amount of the users to ensure they also find the messages valuable. (combine quant and qual)

From there on out, I'm not sure if it's valuable to always include a no-show. When optimizing an existing message, is it not already possible (if desired) to add a treatment that disables the message altogether and get the desired outcome of this feature request?