Bugzilla

Sorry for the slow reply, still catching up from the holidays.

When we used to do this it provided three benefits:
1- Measure if onboarding was better than no onboarding. We still had some doubts then.
2- Allow us to measure our cumulative impact vs no onboarding at all.
3- It provided us with more sensitivity to move our metrics. It is somewhat easier to detect a change from "no-show" than the latest experience. This was valuable because we had a lot of trouble having impact back then with onboarding.

Today:
1- We know onboarding is better than no onboarding. When we regress our current experiences, we hurt our metrics.
2- It's more valuable for us to measure our cumulative impact vs the start of the year than no onboarding at all.
3- I think we enroll enough users to detect differences and we have a better understanding of what is more likely to improve the user experience vs what won't. 
4- When we introduce new messages, our control normally has no message (so we do end up doing something along these lines when relevant).

I will acknowledge that always having a "no-show" makes it a bit easier to compare between experiments but there will always be doubt and nuance when comparing different periods. This is not something we regularly want to do.

As of Firefox 123, Nimbus will be offering the capability for a long-term holdback which will allow us to understand the cumulative impact of winning experiments over some time without resorting back to "no-show". This covers #2 from that list.

Sorry for the slow reply, still catching up from the holidays.

When we used to do this it provided three benefits:
1- Measure if onboarding was better than no onboarding. We still had some doubts then.
2- Allow us to measure our cumulative impact vs no onboarding at all.
3- It provided us with more sensitivity to move our metrics. It is somewhat easier to detect a change from "no-show" than the latest experience. This was valuable because we had a lot of trouble having impact back then with onboarding.

Today:
1- We know onboarding is better than no onboarding. When we regress our current experiences, we hurt our metrics.
2- It's more valuable for us to measure our cumulative impact vs the start of the year than no onboarding at all.
3- I think we enroll enough users to detect differences and we have a better understanding of what is more likely to improve the user experience vs what won't. 
4- When we introduce new messages, our control normally has no message (so we do end up doing something along these lines when relevant).

I will acknowledge that always having a "no-show" makes it a bit easier to compare between experiments but there will always be doubt and nuance when comparing different periods. This is not something we regularly want to do.

As of Firefox 123, Nimbus will be offering the capability for a long-term holdback which will allow us to understand the cumulative impact of winning experiments over some time without resorting back to "no-show". This covers #2 from that list.

Should we add the capability? It wouldn't hurt but it's much less needed than it used to be. It will also require more enrollments for each experiment.