Left some first thoughts in the doc via comments. The overall idea seems OK to me. To my mind, these are the big things to think about right now: 1. we have not done an a/b test on the monitor site yet. typically, engineering would put in the plumbing for this, and we would verify it with an a/a test before doing a "real" experiment on which decisions hinge. do we need to do that here? if so, we can file another bug with DS to help with the design and analysis of that. 2. the doc suggests we compare the click rate of an FPR CTA to that of similar lockwise and FPN CTAs. such a design would tell us, most directly, whether users are *more, less or similarly willing to view information about FPR than they are to view information about lockwise and FPN*. Let's say for the sake of argument that the FPR CTA does perform better than Lockwise or FPN. This would be consistent with any of the following (not mutually exclusive or exhaustive): a. Users are more interested in FPR than they are Lockwise or FPN b. The wording on the FPR CTA was more convincing/clear/easier to read than the other CTAs c. The FPR CTA is more novel than the Lockwise or FPN CTAs (especially a possibility if the user has already seen - and potentially clicked on - CTAs for those products on the monitor site or elsewhere) My guess is that, in reality, we would want to conclude (a) while eliminating (b) and (c) to the best of our ability. There probably isn't a design that does this perfectly, but we would want to give it our best shot. We can rely on our content experts to reduce the chance of (b) as much as possible. My biggest concern is therefore (c): We know that FPR will be totally new for all the users seeing that branch, but Lockwise and FPN may *not* be new to users in those branches. Therefore FPR may have an inherent advantage (or disadvantage?) due to novelty and that could be largely independent of how genuinely interesting the product is to users. I believe there is a good chance users seeing the Lockwise CTA may have already seen some promotion related to Lockwise on the monitor site. Maybe the problem is slightly less for FPN, I'm unsure if its promoted on monitor, but users may have been exposed via other channels - especially email if the are FxA (and the initial design calls for this to only be run on signed-in users, so that may increase the risk). What could we do about this? 1. only use FPN as the control - users are less likely to have been exposed to it already, and hey, it also starts with "Firefox Private...." in the title :P 2. instead of calling out FPN and Lockwise by name in the CTA, make the wording more vague, e.g. "Learn how to manage your passwords better with Firefox" or "Learn how you can more securely browse the web using Firefox", you see my point. 3. Other ideas? I'd like for the FPR promotion to be on as even a footing with its comparison as possible.
Bug 1606003 Comment 1 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Left some first thoughts in the doc via comments. The overall idea seems OK to me. To my mind, these are the big things to think about right now: 1. we have not done an a/b test on the monitor site yet. typically, engineering would put in the plumbing for this, and we would verify it with an a/a test before doing a "real" experiment on which decisions hinge. do we need to do that here? if so, we can file another bug with DS to help with the design and analysis of that. 2. the doc suggests we compare the click rate of an FPR CTA to that of similar lockwise and FPN CTAs. such a design would tell us, most directly, whether users are *more, less or similarly willing to view information about FPR than they are to view information about lockwise and FPN*. Let's say for the sake of argument that the FPR CTA does perform better than Lockwise or FPN. This would be consistent with any of the following (not mutually exclusive or exhaustive): a. Users are more interested in FPR than they are Lockwise or FPN b. The wording on the FPR CTA was more convincing/clear/easier to read than the other CTAs c. The FPR CTA is more novel than the Lockwise or FPN CTAs (especially a possibility if the user has already seen - and potentially clicked on - CTAs for those products on the monitor site or elsewhere) My guess is that, in reality, we would want to conclude (a) while eliminating (b) and (c) to the best of our ability. There probably isn't a design that does this perfectly, but we would want to give it our best shot. We can rely on our content experts to reduce the chance of (b) as much as possible. My biggest concern is therefore (c): We know that FPR will be totally new for all the users seeing that branch, but Lockwise and FPN may *not* be new to users in those branches. Therefore FPR may have an inherent advantage (or disadvantage?) due to novelty and that could be largely independent of how genuinely interesting the product is to users. I believe there is a good chance users seeing the Lockwise CTA may have already seen some promotion related to Lockwise on the monitor site. Maybe the problem is slightly less for FPN, I'm unsure if its promoted on monitor, but users may have been exposed via other channels - especially email if they are FxA (and the initial design calls for this to only be run on signed-in users, so that may increase the risk). What could we do about this? 1. only use FPN as the control - users are less likely to have been exposed to it already, and hey, it also starts with "Firefox Private...." in the title :P 2. instead of calling out FPN and Lockwise by name in the CTA, make the wording more vague, e.g. "Learn how to manage your passwords better with Firefox" or "Learn how you can more securely browse the web using Firefox", you see my point. 3. Other ideas? I'd like for the FPR promotion to be on as even a footing with its comparison as possible.