1063124 - Build and deploy an iframe snippet to install an add-on from about:home

Status: RESOLVED FIXED

(Snippets Graveyard :: HTML5, defect)

Description

[Chris More [:cmore]]

While we are waiting on the ability to install an add-on directly via about:home, we are going to do a snippet using an iframe to addons.mozilla.org. We've done these iframe add-ons previously, but not for quite a while. This will require an HTML page on AMO and a snippet with an iframe.

More details coming.

Comment 1

[Chris More [:cmore]]

Attachment: Mock up of add-on snippet

Comment 2

[Chris More [:cmore]]

Attachment: Icon for snippet

Comment 3

[Chris More [:cmore]]

Copy: 

Title: LastPass Password Manager

Description: We love giving you awesome add-ons. We also love giving you free things. Check out LastPass, a password management add-on.

Comment 4

[Chris More [:cmore]]

Technical update: The + Add to Firefox green button is the only part of the snippet that needs an iframe. The rest of the snippet content and icon can be within the snippet. Thus, the only code that needs to be on AMO is an HTML with the green button pointing to: https://addons.mozilla.org/firefox/downloads/latest/8542/platform:3/addon-8542-latest.xpi

Comment 5

[Jean Collings]

Adding Jslater! John can you verify you're cool with the title as well?

Comment 6

[Chris More [:cmore]]

Also, note that the version number doesn't have to be after the title. That was just on AMO when I created the title.

Comment 7

[John Slater]

(In reply to Jean Collings from comment #5)
> Adding Jslater! John can you verify you're cool with the title as well?

Title isn't bad, but if we have the option it could be a way of calling out the spirit of the promo more. For example, the title could be "We love giving you awesome add-ons", and then the rest of the blurb below could start with the part about free things. Jean, any interest in pursuing something like that? If so, we can wordsmith. If not, I think what you have now (minus the version #) is ok with me.

Comment 8

[Chris More [:cmore]]

(In reply to John Slater from comment #7)
> (In reply to Jean Collings from comment #5)
> > Adding Jslater! John can you verify you're cool with the title as well?
> 
> Title isn't bad, but if we have the option it could be a way of calling out
> the spirit of the promo more. For example, the title could be "We love
> giving you awesome add-ons", and then the rest of the blurb below could
> start with the part about free things. Jean, any interest in pursuing
> something like that? If so, we can wordsmith. If not, I think what you have
> now (minus the version #) is ok with me.

Yeah, we could have a generic title, but then it makes the snippet almost too generic for something that is just about to be installed. For something that is not an external link snippet, we probably want to be more explicit. I believe the sensitivity was around logos and possibility less on text about something, but your question and concern has a good point.

Comment 9

[Jean Collings]

(In reply to Chris More [:cmore] from comment #8)
> (In reply to John Slater from comment #7)
> > (In reply to Jean Collings from comment #5)
> > > Adding Jslater! John can you verify you're cool with the title as well?
> > 
> > Title isn't bad, but if we have the option it could be a way of calling out
> > the spirit of the promo more. For example, the title could be "We love
> > giving you awesome add-ons", and then the rest of the blurb below could
> > start with the part about free things. Jean, any interest in pursuing
> > something like that? If so, we can wordsmith. If not, I think what you have
> > now (minus the version #) is ok with me.
> 
> Yeah, we could have a generic title, but then it makes the snippet almost
> too generic for something that is just about to be installed. For something
> that is not an external link snippet, we probably want to be more explicit.
> I believe the sensitivity was around logos and possibility less on text
> about something, but your question and concern has a good point.

Chris let's try not having the title and just have the specific add on called out within the copy.

Comment 10

[Jean Collings]

And maybe just bold the word "LastPass" in the copy.

Comment 11

[Chris More [:cmore]]

I talked to Winston and he's fine with keeping the title out for now and the rest of the UX for this V1. We can test later if the titles affects conversion with future tests.

Comment 12

[John Slater]

Sounds good to me. Thanks!

Comment 13

[Osmose [:osmose, :mkelly]]

Attachment: Screenshot of in-progress snippet

Still working on making it functional, but here's what I have implemented as far as how the snippet should look, based on the comments above and feedback from jcollings.

(Ignore the flicks icon and text, those are configurable like any normal simple snippet. The button text will also be configurable.)

Comment 14

[Jean Collings]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #13)
> Created attachment 8485965 [details]
> Screenshot of in-progress snippet
> 
> Still working on making it functional, but here's what I have implemented as
> far as how the snippet should look, based on the comments above and feedback
> from jcollings.
> 
> (Ignore the flicks icon and text, those are configurable like any normal
> simple snippet. The button text will also be configurable.)

+1 :)

Comment 15

[Osmose [:osmose, :mkelly]]

We're getting a fair amount of pushback from security (via an email thread) about the "page hosted on AMO" technique. This is understandable: any way we do this involves making it possible for anyone on the internet to trick users into installing the addon we feature, and security is also concerned about users installing addons they don't know much about.

Assuming we don't want to push back against security's opinion (which would make me sad if we did), we're down to two options:

1. Use a normal snippet, possibly with the big green button to catch the user's eye, that just links to the addon's page on AMO.

2. Use a snippet that installs the addon immediately after the user clicks the button, but in return we cannot change the text or icon of the snippet without making code changes to AMO (or spending the extra time to add some new interface to AMO, which is my absolute least favorite option).

I'm ready to do either option, I just need guidance on what is preferred here. 

I think we should go with option 1 mainly because we're already working on a bug to be able to install addons directly from about:home without AMO being involved, and am curious if even just a snippet pointing to an addon (with the green button) will help improve retention. Maybe after we get numbers from that we can decide if no-step installs is a good next move?

Comment 16

[Jean Collings]

I was not involved in the conversations with Chris Beard but I have a hunch that he would want to see add-ons being installed right from the about:home page (and he doesn't want to wait until the product team makes the changes). I will let Chris More and Winston jump in here. 

Just another to be aware of - if we go with #2, can we track clicks or installs?

Comment 17

[Chris More [:cmore]]

AMO doesn't have much Google Analytics on it outside of what can be captured via page views. While it is possible to add click/install tracking on AMO, that team has been focused on Marketplace. We can track traffic from a snippet to addons.mozilla.org via UTM tags, but what they do after that point won't be known without adding GA event tagging on addons.mozilla.org.

Still reviewing the recommendations.

Comment 18

[Chris More [:cmore]]

(In reply to Chris More [:cmore] from comment #3)
> Copy: 
> 
> Title: LastPass Password Manager
> 
> Description: We love giving you awesome add-ons. We also love giving you
> free things. Check out LastPass, a password management add-on.

Going to drop the title for this test and just use the description. Use the icon from comment 2.

Comment 19

[Osmose [:osmose, :mkelly]]

Attachment: Screenshot of finished iframe snippet

Here's a screenshot of the finished snippet using option 1 from comment 15. It's ready for being submitted for review to AMO, but I need confirmation that the design is good before I submit the PR.

Comment 20

[Osmose [:osmose, :mkelly]]

(In reply to Jean Collings from comment #16)
> I was not involved in the conversations with Chris Beard but I have a hunch
> that he would want to see add-ons being installed right from the about:home
> page (and he doesn't want to wait until the product team makes the changes).
> I will let Chris More and Winston jump in here. 
> 
> Just another to be aware of - if we go with #2, can we track clicks or
> installs?

In either case we can send data to the stats server (similar to what we did with the sync snippet and with the social API snippets) for tracking if the user clicks the button, as well as if the user installs the addon or cancels the install.

Comment 21

[Jean Collings]

Cmore, friendly ping on your recommendation on comment 15.

Comment 22

[Chris More [:cmore]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #15)
> We're getting a fair amount of pushback from security (via an email thread)
> about the "page hosted on AMO" technique. This is understandable: any way we
> do this involves making it possible for anyone on the internet to trick
> users into installing the addon we feature, and security is also concerned
> about users installing addons they don't know much about.
> 
> Assuming we don't want to push back against security's opinion (which would
> make me sad if we did), we're down to two options:
> 
> 1. Use a normal snippet, possibly with the big green button to catch the
> user's eye, that just links to the addon's page on AMO.
> 
> 2. Use a snippet that installs the addon immediately after the user clicks
> the button, but in return we cannot change the text or icon of the snippet
> without making code changes to AMO (or spending the extra time to add some
> new interface to AMO, which is my absolute least favorite option).
> 
> I'm ready to do either option, I just need guidance on what is preferred
> here. 
> 
> I think we should go with option 1 mainly because we're already working on a
> bug to be able to install addons directly from about:home without AMO being
> involved, and am curious if even just a snippet pointing to an addon (with
> the green button) will help improve retention. Maybe after we get numbers
> from that we can decide if no-step installs is a good next move?

Cbeard specifically wanted to add-ons installed directly from about:home and not just pointing to AMO.

I think we should make a blocking bug here to add a GA event to the +Add to Firefox button on AMO so we can measure the conversion rate. Without that GA event, we won't know the rate at which people click a snippet button, go to AMO and then click the Add to Firefox button on AMO. That's a critical piece of info to get the true conversion rate and without it, we can't really say one option is better or worse than another. I would be fine with doing option 1 if we created another bug that blocked this about adding a GA event to AMO to measure conversion rate.

Osmose: thoughts on filing a blocking bug and get :garethc to define the GA event for AMO? The conversion rate to visiting AMO isn't really useful at all as we want to see how many of those people actually add it to their browser. If we did that, we could then decide if we think we can improve that conversion rate by moving the install directly to about:home.

Comment 23

[Chris More [:cmore]]

Option 2 is what cbeard and others wanted, but I am fine with option 1 if we can get the GA event on AMO to measure the conversion rate of the additional click. If we did an option 2, we would have to find out how we would get conversion rate on how many saw the snippet and install the add-on from about:home. Technically, it is possible to fire of GA events from non-websites and associate those events/request to a specific web property by passing parameters to the GA gif.

Comment 24

[Osmose [:osmose, :mkelly]]

(In reply to Chris More [:cmore] from comment #22)
> (In reply to Michael Kelly [:mkelly,:Osmose] from comment #15)
> > We're getting a fair amount of pushback from security (via an email thread)
> > about the "page hosted on AMO" technique. This is understandable: any way we
> > do this involves making it possible for anyone on the internet to trick
> > users into installing the addon we feature, and security is also concerned
> > about users installing addons they don't know much about.
> > 
> > Assuming we don't want to push back against security's opinion (which would
> > make me sad if we did), we're down to two options:
> > 
> > 1. Use a normal snippet, possibly with the big green button to catch the
> > user's eye, that just links to the addon's page on AMO.
> > 
> > 2. Use a snippet that installs the addon immediately after the user clicks
> > the button, but in return we cannot change the text or icon of the snippet
> > without making code changes to AMO (or spending the extra time to add some
> > new interface to AMO, which is my absolute least favorite option).
> > 
> > I'm ready to do either option, I just need guidance on what is preferred
> > here. 
> > 
> > I think we should go with option 1 mainly because we're already working on a
> > bug to be able to install addons directly from about:home without AMO being
> > involved, and am curious if even just a snippet pointing to an addon (with
> > the green button) will help improve retention. Maybe after we get numbers
> > from that we can decide if no-step installs is a good next move?
> 
> Cbeard specifically wanted to add-ons installed directly from about:home and
> not just pointing to AMO.

Option 2 it is! Luckily that's what I coded today (In IRC I said option 1 but I was mistakenly thinking of the options I originally presented to the security people) so I can have the PR up tomorrow morning.

> I think we should make a blocking bug here to add a GA event to the +Add to
> Firefox button on AMO so we can measure the conversion rate. Without that GA
> event, we won't know the rate at which people click a snippet button, go to
> AMO and then click the Add to Firefox button on AMO. That's a critical piece
> of info to get the true conversion rate and without it, we can't really say
> one option is better or worse than another. I would be fine with doing
> option 1 if we created another bug that blocked this about adding a GA event
> to AMO to measure conversion rate.
> 
> Osmose: thoughts on filing a blocking bug and get :garethc to define the GA
> event for AMO? The conversion rate to visiting AMO isn't really useful at
> all as we want to see how many of those people actually add it to their
> browser. If we did that, we could then decide if we think we can improve
> that conversion rate by moving the install directly to about:home.

As I've currently implemented it for option 2, we can install directly from the snippet and measure the conversion rate via the snippets stats service (instead of passing a snippet ID for the impression, we pass a unique string, and it shows up alongside the impressions in jcollings' snippets reports). I've got it reporting attempted installs, as well as cancelled installs and successful installs. There's some edge cases (no way to detect if someone removes an addon after installing it, for example) but it's decent stats and should do well for us while we wait for bug 1046903.

Given that we can do option 2 until it becomes possible to install addons directly, I don't think trying to pull GA stats from AMO and linking them with the snippets is necessary.

I'm also assuming you're okay with the design as you haven't commented against it, so I'll go ahead and submit the PR for option 2 to AMO tomorrow. Yay!

Comment 25

[Osmose [:osmose, :mkelly]]

The snippet has been finished. cmore tested it for functionality and I tested it across Firefox versions back to Firefox 26. We're targeting Firefox 26 or higher simply to make testing easier.

The snippet is available on the production admin interface at https://snippets.mozilla.com/admin/base/snippet/4595/. I've set it to go to all English locales and only to versions of Firefox > 26. I've also set it to go to Beta, as I recommend at least 3 days of running this on Beta before we enable this for release in case there are unforeseen issues.

Comment 26

[Chris More [:cmore]]

Fabio: Can you update us on when we can release the LastPass add-on snippet to some Firefox channel? cbeard was asking about this before and I want to make sure we have tested it on some channel and got some data.

Comment 27

[Fabio Rios [:frios]]

Attachment: add-lastpass1.jpg

Comment 28

[Fabio Rios [:frios]]

Attachment: add-lastpass2.jpg

(In reply to Chris More [:cmore] from comment #26)

Let's run this on Beta now through November 1st. 

- Based on comments above, it appears the click and conversion metrics have been sorted out? This will be collected right?

- Ideally we'd have the lastpass logo as icon, but based on previous comments, we cannot change the icon. If that's still the case, an alternative is to remove the icon completely and add a reference to LastPass directly into the button. Sample image attached. Is this possible?

We don't necessarily need the above for Beta since we are just ensuring no technical issues, but it would be great to have them sorted about prior to release.

Comment 29

[Osmose [:osmose, :mkelly]]

(In reply to frios from comment #28)
> - Based on comments above, it appears the click and conversion metrics have
> been sorted out? This will be collected right?

Yes. The metrics will show up in the snippet impressions report in Tableau. Impressions with a snippet ID of "lastpass-addon-snippet-click" represent clicking the button, "lastpass-addon-snippet-install" represents successfully installing the addon, and "lastpass-addon-snippet-cancelled-install" represents an attempted install that a user cancelled. 

> - Ideally we'd have the lastpass logo as icon, but based on previous
> comments, we cannot change the icon. If that's still the case, an
> alternative is to remove the icon completely and add a reference to LastPass
> directly into the button. Sample image attached. Is this possible?
> 
> We don't necessarily need the above for Beta since we are just ensuring no
> technical issues, but it would be great to have them sorted about prior to
> release.

Can we run the snippet as-is and then, after we collect data from beta and prod, consider if we want to change the icon? If we were prepared to put the snippet out before without a LastPass icon, I'd rather not go and add more work for us to finish before putting the snippet on prod when we really just want to see how an addon-install snippet works. If the performance is compelling we can spend time later making small tweaks to perfect it if we really want.

Comment 30

[Fabio Rios [:frios]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #29)
> (In reply to frios from comment #28)

Thanks Mike. I'll set this up to go live today and confirm when they are up.

Comment 31

[Fabio Rios [:frios]]

Confirming this is live in Beta.

Comment 32

[Chris More [:cmore]]

(In reply to Fabio Rios [:frios] from comment #31)
> Confirming this is live in Beta.

Any preliminary data from beta? Are we planning to go to release channel?

Comment 33

[Fabio Rios [:frios]]

Hey guys, sorry for the delay. I'll get the data to share very soon. 

Assuming all looks good. I'd like to get this out on release in December. Two questions are, 

1. Is getting this on release in December realistic? Let's say running from the 1st
2. Can this be made available to multiple locales, I've got about 15 in mind (I'll need to cross check if LastPass is available in those specific of course)

In terms of what the snippet would like like I imagine we would remove the icon and customize the text on the snippet and the button (exact details TBC).

Comment 34

[Osmose [:osmose, :mkelly]]

(In reply to Fabio Rios [:frios] from comment #33)
> Hey guys, sorry for the delay. I'll get the data to share very soon. 
> 
> Assuming all looks good. I'd like to get this out on release in December.
> Two questions are, 
> 
> 1. Is getting this on release in December realistic? Let's say running from
> the 1st
> 2. Can this be made available to multiple locales, I've got about 15 in mind
> (I'll need to cross check if LastPass is available in those specific of
> course)
> 
> In terms of what the snippet would like like I imagine we would remove the
> icon and customize the text on the snippet and the button (exact details
> TBC).

Any changes to the text or icon require addons.mozilla.org to do another deploy, as for security reasons we were not able to implement this snippet in a way that didn't involve having the text and icon live there. And any subsequent tweaks would also require them to deploy.

I'm going to suggest that until bug 1046903 is resolved (which would let us install addons from about:home directly) this snippet is frozen (as in not changing but totally able to be sent out to any channel) and we're not going to do any more like it. The security issues as well as reliance on the AMO team make it unreasonable to expand until we solve the root issue of not being able to install from about:home.

Luckily SOMEONE just moved into a more focused role around Firefox Growth and should totally drive that bug to get it resolved. :D

Comment 35

[Chris More [:cmore]]

*someone* hmmmm. *cricket* *cricket*

Comment 36

[Chris More [:cmore]]

+1 to not doing anymore iframe add-ons until bug 1046903. The AMO work is a bit pailful and anytime with doing it via an iframe is time that could be done making it work within about:home directly without having to involve other teams. The iframe stop-gap solution was do an add-on snippet, gather data, to help inform us when we build one with bug 1046903.

Comment 37

[Winston Bowden]

+1.  Also very curious to hear how test performed.

Comment 38

[Fabio Rios [:frios]]

Attachment: Results!

20M impressions from Beta channel resulted in a 1.45% CTR. The attached shows the actions based on 292,610 clicks.

@cmore, let's touch on this during our development road map meeting next week.

Comment 39

[Osmose [:osmose, :mkelly]]

This looks done to me.