Closed Bug 760663 (track-real-downloads) Opened 12 years ago Closed 9 years ago

Track Real Firefox Downloads instead of Clicks

Categories

(Firefox Affiliates Graveyard :: affiliates.mozilla.org, defect)

Product:
Firefox Affiliates Graveyard
Component:
affiliates.mozilla.org
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: osmose, Assigned: osmose)

References

Details

(Whiteboard: [kb=1300313] [qa+][2.0 post-launch]) 

We'd like to be able to track how many real Firefox downloads have resulted from a user's Affiliate Banner.

Implementation-wise, one way we've considering doing this:

- Add a URL argument, such as "?affiliateId=335135" to URLs going to Firefox download pages.

- Submit a patch to Bedrock that alters the Download Button code to detect this argument and include an AJAX request back to Affiliates with the ID when the button is clicked.

This does not cover the issue of PHP-generated Download buttons, but we might be able to alter that code as well. 

There are a few issues with tracking Firefox downloads in general that we need answers to:

- How will we display Downloads vs Clicks? We can't just remove clicks, because not all of our banners lead to Firefox downloads.

- What other changes will happen to the site based on the new distinction between Firefox downloads and Clicks?
Severity: normal → major
(In reply to Michael Kelly [:mkelly] from comment #0)
 
> There are a few issues with tracking Firefox downloads in general that we
> need answers to:
> 
> - How will we display Downloads vs Clicks? We can't just remove clicks,
> because not all of our banners lead to Firefox downloads.
> 
> - What other changes will happen to the site based on the new distinction
> between Firefox downloads and Clicks?

I still think affiliates still get credit for clicks. Knowing how the number of downloads we generate is for us. We need this data so that we can understand how effective we're being and how many downloads we're actually generating from affiliate links.

It would be great if this was visible in the admin interface.
Depends on: 713040
+1 on making this statistic in addition to the leaderboard that we have now. It will be interesting to track over time clicks vs downloads. It will also help to know if future changes to Mozilla.org close the gap between clicks on banners and real downloads of our products. Ideally, each click of an Affiliates banner would result in a download, but we know that is probably not the case. Knowing the gap will help determine if improvements to our websites makes that better or worse.
Then we'll add a dashboard for downloads once the dashboard is live and we're tracking this.

malexis: I'll be doing development on this, is there anything we have to do to get this approved for addition to mozilla.org and what general timeframe would there be for its release once it's done?
Assignee: nobody → mkelly
Product: Websites → Firefox Affiliates
Since the Firefox download pages are going through an overhaul in the near future, this could happen after that work is live and they are all in bedrock.
That would be excellent!
Hey Chris and Mike and Mike, 

Now that the new download pages are up and running can we get back to this bug? 

Cheers, 
C
Whiteboard: [qa+]
(In reply to Chelsea Novak [:chelsea] from comment #6)
> Hey Chris and Mike and Mike, 
> 
> Now that the new download pages are up and running can we get back to this
> bug? 
> 
> Cheers, 
> C

The new download page is not live yet. We are still doing another A/B test till next week. It should be live in the next few weeks. The new download page should do what you want out of the box since I had pmac add the feature last week. We will be able to track campaigns to the download page and then taking an action of downloading. The Google Analytics UTM campaign tags will automatically pass through to the Google Analytics virtual page view that is set when a user clicks on a download button.

Stay tuned. :)
Soooo, how are things with the new download page coming along? Are we ready to get moving on this? :)
Comment 7 explains how it will work. We just need to add UTM tags to the affiliate buttons as we have GA on the button clicks on /firefox/new/. This will only work with /firefox/new/ at this point as /firefox/fx/ does not work the same way.
(In reply to Chris More [:cmore] from comment #10)
> Comment 7 explains how it will work. We just need to add UTM tags to the
> affiliate buttons as we have GA on the button clicks on /firefox/new/. This
> will only work with /firefox/new/ at this point as /firefox/fx/ does not
> work the same way.

I think you're thinking of getting general stats about how Affiliate banners turn into downloads, but I thought that this bug was for actually tracking if an Affiliate banner click resulted in a download, and storing that count on Affiliates to show users how many downloads their banner generated. 

If we use Google Analytics for that, we'd need it to have an API we can query for a specific banner, and also be sure that it's not sampling data and we're getting an exact number. Can it do that? (And even if it can, I don't think analytics software is the right choice for this task.)

At this point I thought we were waiting for the A/B tests on the download page to be done so we can add the ping back to Affiliates after a download occurs in bedrock's JavaScript code.
> If we use Google Analytics for that, we'd need it to have an API we can
> query for a specific banner, and also be sure that it's not sampling data
> and we're getting an exact number. Can it do that? (And even if it can, I
> don't think analytics software is the right choice for this task.)
> 

It is possible given that we have a GA API with unsampled data, but I agree that it is an overkill. 

The A/B tests for the /new page are done enough to add in additional functionality like this ping. We can do the ping back to affiliates like we are doing it with GA with a slight delay. If we don't have a delay or do it in an asynchronous way, the save-as dialog box will stop all JS execution when it focus is taken away from the DOM.
Hey guys. 

Can I get a status update on this one? Thanks!
No updates. It was not on our Q2 priority list. I would push this with WilliamR since he is the product owner of community tools and decide if we should back off of Reps and Mozillians to allow for developer bandwidth.
Our current thinking is along the lines of what was suggested in Comment 11; we're thinking of using a page on Affiliates with GA on it to redirect to mozilla.org, and we'll use the GA API to track visits coming from those pages to download events, and pull that data into Affiliates.
Blocks: affiliates-2.0
Target Milestone: --- → 5
Alias: track-real-downloads
Blocks: build-affiliates-2.0
No longer blocks: affiliates-2.0
Whiteboard: [qa+] → [kb=1300313] [qa+]
Gareth, can you add a comment specifying what tags we need to include on Affiliate links going to bedrock that will allow us to track users coming from a banner and ending up with a download?
Flags: needinfo?(garethcull.bugs)
Hey mkelly,

Currently, this is an example of the utm structure we use in our affiliates tracking in Google Analytics. 

http://www.mozilla.org/mobile/?utm_source=firefox-affiliates&utm_medium=banner&utm_campaign=aff-mobile-android.

Based on your comment 1 it looks like we could dynamically append a query parameter with an affiliate id. So we could keep our current parameters the same, and store the affiliate id appended at the end with a utm_content parameter.  

http://www.mozilla.org/mobile/?utm_source=firefox-affiliates&utm_medium=banner&utm_campaign=aff-mobile-android&utm_content=={affiliateID}.

Would that work? Or do you need the referring domain in addition to affiliateID for you stats?
Flags: needinfo?(garethcull.bugs)
(In reply to Gareth Cull [:garethc] from comment #18)
> Hey mkelly,
> 
> Currently, this is an example of the utm structure we use in our affiliates
> tracking in Google Analytics. 
> 
> http://www.mozilla.org/mobile/?utm_source=firefox-
> affiliates&utm_medium=banner&utm_campaign=aff-mobile-android.
> 
> Based on your comment 1 it looks like we could dynamically append a query
> parameter with an affiliate id. So we could keep our current parameters the
> same, and store the affiliate id appended at the end with a utm_content
> parameter.  
> 
> http://www.mozilla.org/mobile/?utm_source=firefox-
> affiliates&utm_medium=banner&utm_campaign=aff-mobile-
> android&utm_content=={affiliateID}.
> 
> Would that work? Or do you need the referring domain in addition to
> affiliateID for you stats?

We actually need an id that is specific to each link a user generates, but since those are tied to a user, we can just use that value in place of the affiliateID value in your link above.

So I think we should be good, then. Thanks for the help!
Gareth: Does the user from bug 975258, comment 1 (716087457957-tqst57dkgpil0pj5tjj3u4s56mc61q6l@developer.gserviceaccount.com) have read access to the bedrock analytics profiles? Can we give it read access to the www-dev.mozilla.org profile?
Flags: needinfo?(garethcull.bugs)
mkelly,

Do you mean www-dev.allizom.org? We don't have a GA profile for www-dev.mozilla.org. I've added the user to the allizom profile if that works.

Let me know. Thanks.

Gareth
Flags: needinfo?(garethcull.bugs)
(In reply to Gareth Cull [:garethc] from comment #21)
> Do you mean www-dev.allizom.org?

Yep. Thanks!
We've discovered sampling issues with getting download data from bedrock via GA using the normal API. I think it'd be wise to cut downloads as a stat from the first release, and stick with clicks only (which is what the current site has, anyway).

We're close, we just need to change how we're using the API, but there's not enough time left to do so and still launch the site in our timeframe. hoosteeno and craigcook were discussing on IRC and favor replacing download stats in the UI with the user's leaderboard position.

jessosorio: Are you okay with cutting download counts from the feature set for the first release, and replacing public stat displays of downloads with leaderboard positions?
Flags: needinfo?(josorio)
Confirmed during a meeting that jessosorio is okay with us punting on downloads for launch; it was implied that we would very much like to follow up with download tracking as a feature for shortly after launch.
No longer blocks: build-affiliates-2.0
Flags: needinfo?(josorio)
Whiteboard: [kb=1300313] [qa+] → [kb=1300313] [qa+][2.0 post-launch]
Target Milestone: 5 → ---
Depends on: 1025962
No longer going forward with the strategy outlined in 1025962.

mKelly - Can you update this bug with the new strategy or link to a different bug and mark this one resolved?
(In reply to ekariv from comment #25)
> mKelly - Can you update this bug with the new strategy or link to a
> different bug and mark this one resolved?

Sure!

The current plan is to modify mozilla.org (Bedrock) to add an extra query parameter to Firefox download links when a user arrives from an Affiliates banner. Then, we'll get Affiliates access to the logs for Bouncer, which handles Firefox downloads, and process the logs to find those extra parameters and count them as Firefox downloads.
Blocks: 1071242
Hey mkelly! 

We're going to open this up again - with an effort to get it done sometime in Q4 (cmore mentioned that is can be included in the pipeline). 

Do you think we can continue to work off of this bug? Remember we chatted about this - the tracking downloads issue?

Please confirm if we can continue to work off of this bug to get this one fix done :) 

Thank you!
(In reply to Jessica Osorio from comment #27)
> Hey mkelly! 
> 
> We're going to open this up again - with an effort to get it done sometime
> in Q4 (cmore mentioned that is can be included in the pipeline). 
> 
> Do you think we can continue to work off of this bug? Remember we chatted
> about this - the tracking downloads issue?
> 
> Please confirm if we can continue to work off of this bug to get this one
> fix done :) 
> 
> Thank you!

Hey Mike. Let us know the scope of this change and if we can fit it into Q4. Are you thinking about just passing along the UTM parameters that we use now on affiliates to d.m.o and then just query d.m.o logs?

While Affiliates is on-hold from major development, smaller tasks that make the website better and provide data for future development activities can still continue when we have time.
Flags: needinfo?(mkelly)
(In reply to Chris More [:cmore] from comment #28)
> Hey Mike. Let us know the scope of this change and if we can fit it into Q4.
> Are you thinking about just passing along the UTM parameters that we use now
> on affiliates to d.m.o and then just query d.m.o logs?
> 
> While Affiliates is on-hold from major development, smaller tasks that make
> the website better and provide data for future development activities can
> still continue when we have time.

Passing along parameters to d.m.o isn't terribly hard, but getting the data from the d.m.o logs is the hard part. I'd scope this out as a 2-3 week project, which I think is a bit larger than we want to support. Thoughts?
Flags: needinfo?(mkelly)
Jessica: After scoping this one, this is a bit more than a minor project and almost none of the work has been previously done. It would require a decent amount of engineering work and support from IT to be able to get more accurate numbers. Let's discuss when you are in MV next.
osmose: What happened to the feature gareth mentioned in comment 18 about adding &utm_content=[bannerID or affiliate ID] to all Affiliates banners? Was that ever added? If we had this, Gareth could make a Google Doc spreadsheet that pulled data from www.mozilla.org and created a dashboard that would include downloads, and even installs by banner/affiliate including other segment meta data. If we could combine it with GA data from affiliates.mozilla.org, we could probably get click-to-download conversion rates in the spreadsheet too.

What is the scope of adding utm_content=[some ID] to all affiliate links? If seems more trivial than querying d.m.o and trying to display that on affiliates.mozilla.org.

If it is added, we can track in Google Analytics sessions and any following page/experience they visit even if those following page don't have UTM parameters on it. We have a virtual page view when people download that would be associated with the session and we don't need to have those parameters passed to d.m.o.

Adding utm_content=ID is a "good enough" approach to create an internal leaderboard for competitions and there is one coming up soon that needs this. We'll just need to have Gareth create a gdoc dashboard that pulls in the data and summaries it.
Flags: needinfo?(mkelly)
(In reply to Chris More [:cmore] from comment #31)
> osmose: What happened to the feature gareth mentioned in comment 18 about
> adding &utm_content=[bannerID or affiliate ID] to all Affiliates banners?
> Was that ever added? If we had this, Gareth could make a Google Doc
> spreadsheet that pulled data from www.mozilla.org and created a dashboard
> that would include downloads, and even installs by banner/affiliate
> including other segment meta data. If we could combine it with GA data from
> affiliates.mozilla.org, we could probably get click-to-download conversion
> rates in the spreadsheet too.

I had looked into it, and hit an un-fixable roadblock, but I don't remember exactly what the issue was. I know I've looked into variations of that plan more than once over the years, and at least one time we hit an issue that GA couldn't trigger an event after a download button was clicked because any in-progress HTTP requests got cancelled once the download began. Your comments make it sound like we currently do have some form of GA tracking around downloads on mozilla.org; do we?

Another issue was that we've always approached the problem as wanting to track downloads as a number users can see on the site, and my experience with GA on Affiliates thusfar is that the numbers are right for the purposes of trends and measuring impact and changes, but not suitable for the kind of counting we need for user-facing features; if QA can't click a button, trigger a download, and see the download number go up by one, it's not suitable.

Your plan seems to just want to get the numbers in a Google spreadsheet for a human to review. Is that accurate? If so, that changes the purpose of this bug.

> What is the scope of adding utm_content=[some ID] to all affiliate links? If
> seems more trivial than querying d.m.o and trying to display that on
> affiliates.mozilla.org.

It wouldn't be difficult. A day or two for it to hit prod, assuming there's no other features waiting to be pushed out (I don't think there are but need to check).

> If it is added, we can track in Google Analytics sessions and any following
> page/experience they visit even if those following page don't have UTM
> parameters on it. We have a virtual page view when people download that
> would be associated with the session and we don't need to have those
> parameters passed to d.m.o.
>
> Adding utm_content=ID is a "good enough" approach to create an internal
> leaderboard for competitions and there is one coming up soon that needs
> this. We'll just need to have Gareth create a gdoc dashboard that pulls in
> the data and summaries it.

This is the first I've heard of this competition. Is there a bug for it?
Flags: needinfo?(mkelly)
We have had download tracking on mozilla.org for a few years now and it is pretty accurate. We can monitor downloads by source and campaign and all the way to successful installs (only if the user imports their cookies). If the Affiliate links had the utm-content populated, we could create an internal dashboard that shows the most successful banners. 

The competition is being ran by Kristin, CBT, and user engagement and they are trying to use all of our existing channels/tools to run a download competition (a legal one) for the 10th anniversary and hand out swag. The current gaming of Affiliates leaderboard prevents the contest from giving out sway. They would rather have an internal spreadsheet that they can view downloads. We have a dashboard now and if we have the utm_content, we could make a filter to only show downloads from Affiliates. The competition is a new thing as of a week ago and I just heard about it.
Osmose: Don't proceed with any dev work on adding utm_content. I was just looking at the feasibility and scope of adding that query parameter. From your messages above, it looks doable on adding the parameter and would probably take a week or so to get out the door. No pull request is needed at this moment given the above feedback.

As for the competition, it could be a bit of a complex UX to tell users that they have to sign up for affiliates and then use an affiliates banner URL to be entered into the competition. There has to be other ways without creating a piece of new technology to do this. I would prefer not inventing something for a contest given the time left to the 10th anniversary.
All work on Firefox Affiliates is on hold as the Affiliates program is slated to be phased out in 2015.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Product: Firefox Affiliates → Firefox Affiliates Graveyard
You need to log in before you can comment on or make changes to this bug.