Bugzilla

I'm fine to do this test in this way — that is, to add a recommendation regarding email masking with a clickable CTA — but would like us to avoid pitting recommendations and various products/services against each other when we analyze the results. There are too many variables to isolate this as a test with a true control. 

Users clicking on product A versus product B doesn't mean that A is a winner and B is a loser. Product A and Product B solve strive to solve entirely different privacy-related problems. Users may or may not have a level of understanding of what those privacy problems even are. Those users may or may not be interested in learning about one right now. We also cannot know is this the first time a user is seeing breach recommendations (perhaps this is their first breach), the 10th (perhaps they've received many breach alerts and have seen this page without this product countless times), or any number in between. And all privacy threats are not created equal. The fact that someone selects recommendation A doesn't mean they don't care about the others. It only means that at this very isolated moment in time, they clicked that particular one. 

So with that context and nuance, I think see if people click recommendation A at all, without comparing it to recommendations B, C, D, etc., is the right way to go.

I'm fine to do this test in this way — that is, to add a recommendation regarding email masking with a clickable CTA — but would like us to avoid pitting recommendations and various products/services against each other when we analyze the results. There are too many variables to isolate this as a test with a true control. 

Users clicking on product A versus product B doesn't mean that A is a winner and B is a loser. Product A and Product B solve strive to solve entirely different privacy-related problems. Users may or may not have a level of understanding of what those privacy problems even are. Those users may or may not be interested in learning about one right now. We also cannot know is this the first time a user is seeing breach recommendations (perhaps this is their first breach), the 10th (perhaps they've received many breach alerts and have seen this page without this product countless times), or any number in between. And all privacy threats are not created equal. The fact that someone selects recommendation A doesn't mean they don't care about the others. It only means that at this very isolated moment in time, they clicked that particular one. 

So with that context and nuance, I think seeing if people click recommendation A at all, without comparing it to recommendations B, C, D, etc., is the right way to go.