1915917 - [Skylight] Move away from using heuristics to figure out if messages contain microsurveys

Status: NEW

(Firefox :: Messaging System, task, P3)

Description

[Sarah Chung [:sachung]]

Currently, we check for the 'survey' sub-string inside messages' and experiments' ids, slugs, descriptions, and user facing names to figure out if they contain a micro-survey. Ideally, we'd like to move away from this heuristic check to figure out if a message contains micro-surveys.

Comment 1

[Emily McMinn :emcminn]

We've discussed this with both the Nimbus team and with Shane and figured out there's not much to do here right now. The groups property as implemented exists inside the feature value on Experimenter recipes, and should already be properly exposed. And while there is a micro-surveys group, it's a little bit of a misnomer in that it is providing some trigger behaviour that we won't necessarily always want, even though the message may still contain a micro-survey.

We could consider including a tag or a property on all microsurvey-containing messages going forward to allow us to identify them better (which might be a next step for this) but we would still need to account for the existing microsurvey messages, which means we'll likely want to keep the work Sarah has done & add to it, rather than replacing it with another identification method.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

I'm concerned about it being a footgun to having something called "micro-surveys" not be 1:1 and onto microsurveys going forward. Perhaps we deprecate / rename? Emily has more context here...

Comment 3

[Shane Hughes [:aminomancer]]

micro-surveys is a message group created specifically for implementing the product requirement that surveys be limited to one per 6 months. It is never used for any messages that are not surveys. The issue is the opposite - that not every survey has used micro-surveys, because some surveys were implemented before the group was added, and some other surveys use triggers that need to show and hide the message many times, and therefore can't abide by the micro-surveys group's 1-per-6-months frequency cap. This is the case for the surveys we've run that visually look like they're inside the new tab page, but are actually rendered on top in the browser chrome.

Since those new tab surveys are not actually in the content, they need to hide when you switch tabs, but reappear when you switch back to the new tab page. Each time they reappear, it checks the frequency caps and saves an impression. So if there was a cap, it would not reappear when you switched back to the tab. We determined that it would intolerably reduce our reach for the message to only be visible for that one brief moment before navigating away from the new tab page.

On the other hand, if they genuinely appeared inside the content, then for that one tab, it would remain visible no matter how many times you switched tabs, as the content doesn't permanently disappear on visibilitychange events, and therefore the message doesn't need to be "re-shown" when its rendering context becomes visible again. It wouldn't show up in multiple tabs, but it would persist in any one individual tab. Once in-content feature callout is finished, these surveys will behave like that. But for the moment, they show on top of the content, so they technically need to "show" an arbitrarily large number of times.

That forces our hand on frequency capping and groups. But in the long run, our intention is still that all new surveys will eventually use the micro-surveys group, notwithstanding any special cases or extenuating circumstances (for example, a survey that needs to quickly follow-up on a specific experiment while it's fresh in users' minds).

In summary, all members of the micro-surveys group are and should be surveys. But not all surveys are in the micro-surveys group, which means that for the purposes of labeling/tagging experiments as surveys, we can't exclusively check for this group. In other words, not all surveys are in the group, but everything that's in the group is a survey. The group itself will only get more necessary, so we can't deprecate it. And there isn't any better name for it, since it is intended solely for micro-surveys.

So I do think a multi-pronged approach is best for this. Even when we overcome this issue of some surveys needing to forego frequency caps, we'll still always have this historical issue of some previous survey experiments not including the group. For those to be searchable and properly labeled in Skylight, we'll always need a somewhat bespoke method of identifying old surveys with unconventional/provisional configurations.

I don't know if heuristic is the right approach, though. I agree that searching for the substring "survey" constitutes some serious heavy-duty duct tape. But we will still need a bespoke solution, since there's no way to retroactively standardize these old, pre-standardization surveys. I proposed a multi-pronged approach like this:

Conditions for micro-survey status:

1. groups array includes micro-surveys
2. OR tags array includes micro-survey (not implemented, but this would provide for potential future extenuating circumstances I mentioned above)
3. OR message_id is present in a static list saved somewhere in Skylight (not implemented, but the list would be pretty short)

Those 3 should be able to cover everything without requiring heuristics. The first two would never be used for anything other than a survey, so this avoids false positives. And the third would avoid false negatives for surveys that predate the message group, while only making reference to unique messages. The number of surveys that fall outside the first condition should, in the long run, be very small. Perhaps 5 or 6 messages? So simply saving their ids isn't all that unwieldy, and it avoids the imprecision of searching for the "survey" substring.

One point in favor of this approach is that it's easy to imagine tags being generically useful on Skylight. We just discussed a messaging campaign for Profile Management. Those shouldn't belong to any unique group, and they'll probably be built directly into Firefox so they won't be united by an experiment id. So there's nothing to tie them together as a discreet campaign, nor separate them from other similar spotlights and callouts. So multi-message campaigns like this would benefit from a tags property that allows Skylight to label them as part of a campaign. Same goes for Review Checker, Device Migration, etc.

A static list of old survey ids is a bit icky, but not as bad (nor as slow) as substring searching, and it's better than failing to label old surveys as surveys. Though it wouldn't be the end of the world if we only implemented the first 2 conditions, and just accepted that some old surveys will slip through the cracks.

Comment 4

[Meg Viar [:mviar]]

NI from triage