1008390 - Add UTM parameters to locale redirects on bedrock

Status: RESOLVED INVALID

(www.mozilla.org :: Bedrock, defect)

Description

[Chris More [:cmore]]

We get a lot of direct traffic to mozilla.org and we feel this is partially the cause of the locale redirect. For example, if someone links to www.mozilla.org from another website, the traffic should appear to be "referral" with a referring URL, but instead, www.mozilla.org redirects to www.mozilla.org/[locale]/ and that redirect drops the http_referrer header value.

For just the locale redirects, can we add UTM parameters to pass this information through the redirect?

For example:

www.mozilla.org

would redirect to:

www.mozilla.org/[locale]/?utm_source=locale-redirect&utm_medium=referral&utm_content=%{HTTP_REFERER}

Comment 1

[Chris More [:cmore]]

Code here:

https://github.com/mozilla/bedrock/blob/master/lib/l10n_utils/__init__.py#L47-L50

Also, we should only add the HTTP_REFERER header if it exists and is not null. Then the HTTP_REFERER  should be url encoded to be able to be passed via a query parameter. If it is null or doesn't exist, we should append the utm_content parameter. The source, and medium are the only required parameters.

Comment 2

[Kohei Yoshino [:kohei]]

Maybe the correct code to redirect URLs w/o a locale is in funfactory:
https://github.com/mozilla/funfactory/blob/master/funfactory/middleware.py#L64

Comment 3

[Chris More [:cmore]]

(In reply to Kohei Yoshino [:kohei] from comment #2)
> Maybe the correct code to redirect URLs w/o a locale is in funfactory:
> https://github.com/mozilla/funfactory/blob/master/funfactory/middleware.
> py#L64

That's an interesting idea. So, maybe push the appended UTM tags back funfactory instead of making it a bedrock-specific instance. If that is the case, we should probably have a flag in settings.py to enable (or disable) that option.

Comment 4

[Paul [:pmac] McLanahan]

:cmore I think it's possible to ping GA directly from the python side. Would that be preferable to making the URLs really long and ugly in these cases?

Comment 5

[Kohei Yoshino [:kohei]]

WIP, still needs some tests.

https://github.com/kyoshino/bedrock/compare/bug-1008390-locale-redirect-utm
https://github.com/kyoshino/funfactory/compare/bug-1008390-locale-redirect-utm

or PYGA: Python Google Analytics - Data Collection API

https://github.com/kra3/py-ga-mob

Comment 6

[Chris More [:cmore]]

(In reply to Paul McLanahan [:pmac] from comment #4)
> :cmore I think it's possible to ping GA directly from the python side. Would
> that be preferable to making the URLs really long and ugly in these cases?

I was concerned about the long-urls too, but the UTM parameters are much easier technically and don't require any API connectivity. We really don't know how much traffic this would affect, because we don't know how people are redirected to a locale on a daily basis. What if we did the UTM parameters to gauge how big of a problem this is and then decide to go with a more complex solution? This shouldn't affect organic traffic as those always have locales attached. It is mainly sources that link to webpages without locales, but we don't know the scale of this problem. All I know is that it is a portion of direct traffic. While the URLs would be longer for visitors, it shouldn't affect SEO given the <link rel="alternate" canonical tags.

Comment 7

[Chris More [:cmore]]

Important note that we just thought of. Don't add UTM variables to the redirect *if* utm_medium or utm_source is already defined in the query parameter.

Comment 8

[Kohei Yoshino [:kohei]]

I'll finish my pull requests.

Comment 9

[Josh Mize [:jgmize]]

Based on the conversation in the Mozilla.org status meeting, several developers voiced concerns about potential issues with this approach. It may be possible to obtain this data via analysis on the load balancer logs. :aphadke is this something that we could do using our existing log analysis infrastructure and data collection, and if so, how difficult would this data be to obtain?

Comment 10

[Kohei Yoshino [:kohei]]

Can Bug 1018511 solve the issue? It only works on modern browsers though.

Comment 11

[Anurag Phadke[:[e-mail]]]

:jgmize - it's possible to do it via our log analysis, we can modify zeus to log header information for mozilla.org (already do it for few other properties).

jgmize - wanna sync up via vidyo and hash out the details?

Comment 12

[Josh Mize [:jgmize]]

:aphadke I'd be happy to sync up on vidyo later if this is the solution we decide to pursue; mainly I wanted to confirm that this is possible and not too difficult.  :cmore, would you be interested in setting up a brief meeting to discuss this and make sure we're all on the same page, perhaps including :garethc as well?

Comment 13

[Chris More [:cmore]]

This will more likely only affect non-search traffic because search engines don't link to redirects in their results. Users who visit our websites from search (22-63% depending on the page) will not be redirected to UTM URLs.

Yes, the URLs will be more messy, but we normally aren't concerned with this for campaign traffic, so why are worried about this more now? We drive millions of visits to our websites that contain UTM parameters every day regardless of this redirect and we haven't seen negative impacts. This won't affect SEO as search engines ignore these type of parameters.

We may be trying to make this more complex than it needs to be. We also could turn this off in the future once we understand how much of direct traffic comes from locale redirects. Currently, we have no idea and this insight will help uncover the "black box" that is direct traffic. How about we do this change, turn it on for two weeks, and then decide if we want to remove it? We should get info in those few weeks that should answer this question.

Comment 14

[Josh Mize [:jgmize]]

:cmore to be clear, I personally have no objections to adding the UTM params, I was merely suggesting a potential alternative after hearing objections other developers voiced about them in our meeting earlier today. As I recall, there were concerns that modifying the middleware in this way has a non-negligible risk of accidentally introducing errors into the system due to unforeseen interactions with other parts of our system that might only show up in production, and also concerns about possible PR impact of making tracking of users more visible given the objections that some users have to any GA use at all. I consider both of these to be relatively small risks; I'm merely providing them here for context. 

Completely independent of the risk avoidance though, there is the potential advantage that if we are already storing the referrer headers from the logs in hadoop, then there would not need to be a multi week delay in getting the data; there would only be the time needed to write and execute the queries against existing historical data-- this is actually the main reason I requested more information from :aphadke.

Comment 15

[Chris More [:cmore]]

(In reply to Josh Mize [:jgmize] from comment #14)
> :cmore to be clear, I personally have no objections to adding the UTM
> params, I was merely suggesting a potential alternative after hearing
> objections other developers voiced about them in our meeting earlier today.
> As I recall, there were concerns that modifying the middleware in this way
> has a non-negligible risk of accidentally introducing errors into the system
> due to unforeseen interactions with other parts of our system that might
> only show up in production, and also concerns about possible PR impact of
> making tracking of users more visible given the objections that some users
> have to any GA use at all. I consider both of these to be relatively small
> risks; I'm merely providing them here for context. 
> 
> Completely independent of the risk avoidance though, there is the potential
> advantage that if we are already storing the referrer headers from the logs
> in hadoop, then there would not need to be a multi week delay in getting the
> data; there would only be the time needed to write and execute the queries
> against existing historical data-- this is actually the main reason I
> requested more information from :aphadke.

Ah, got it. Valid concern! Don't want to break something fundamental to bedrock. I would have been in the meeting this morning to discuss, but I was at the "Bay Area Cookies and Online Trackers" meeting that I attend quarterly. Unfortunately, no real cookies. :-/ 

Let's see what anurag has to say. The main goal is to get a percentage measure of how much of the direct traffic is locale redirects.

Comment 16

[Chris More [:cmore]]

Quick note: The ROI of this bug is currently unknown. We have over 50% of traffic coming into mozilla.org that is considered direct traffic. We don't know what percentage of that is from locale redirects. A complex solution may not be worth it, so let's see what the minimal amount of work that we can do to understand how many locale redirects happen daily. We don't have to create the perfect solution.

I will schedule a meeting with jgmize and anurag. thanks!

Comment 17

[Chris More [:cmore]]

Some information I found:

http://webmasters.stackexchange.com/questions/2911/google-analytics-and-301-redirects

http://ilian.i-n-i.org/google-analytics-referring-sites-and-https/

Gareth is also going to see if we could rewrite this with a profile filter:

www.mozilla.org/en-US/?lr=1

to:

www.mozilla.org?utm_source=locale-redirect&utm_medium=referral

If we can do the rewrite on the GA side, we can have something much cleaner in the URL.

Comment 18

[Chris More [:cmore]]

Sheeri: Before Anurag left, he mentioned to me that you should be able to help here. Thanks in advance! :)

Comment 19

[Chris More [:cmore]]

Sheeri: wrong bug, check out bug 1040188.

Comment 20

[Mike Alexis [:malexis]]

Chris is this bug still valid and worthy of investigation or can we resolve wontfix?

Comment 21

[Chris More [:cmore]]

(In reply to Mike Alexis [:malexis] from comment #20)
> Chris is this bug still valid and worthy of investigation or can we resolve
> wontfix?

Yes, the blocking bug is still being worked on by jgmize and sheeri.

Comment 22

[Sheeri Cabral [:sheeri]]

As the blocking bug is complete, I'm going to take myself off the cc list for this bug. If you need me, please re-cc me.

Comment 23

[Mike Alexis [:malexis]]

looks like this is potentially blocked and ready to move forward. Josh, can you confirm?

Comment 24

[Josh Mize [:jgmize]]

(In reply to Mike Alexis [:malexis] from comment #23)
> looks like this is potentially blocked and ready to move forward. Josh, can
> you confirm?

I can confirm that we did some log analysis in bug 1040188, but I'm uncertain about whether :cmore got the answers he was looking for or whether further data and/or analysis is needed. :cmore, do you think the data we could get from the additional UTM params might still be valuable?

Comment 25

[Chris More [:cmore]]

(In reply to Josh Mize [:jgmize] from comment #24)
> (In reply to Mike Alexis [:malexis] from comment #23)
> > looks like this is potentially blocked and ready to move forward. Josh, can
> > you confirm?
> 
> I can confirm that we did some log analysis in bug 1040188, but I'm
> uncertain about whether :cmore got the answers he was looking for or whether
> further data and/or analysis is needed. :cmore, do you think the data we
> could get from the additional UTM params might still be valuable?

Your log analysis looks pretty promising and locale redirects account for about 57-60%, which could explain why our (direct) traffic is so big.  

Josh: are adding any parameters (UTM or anything else) to locale redirects still a technical risk to the middleware mentioned above?

What if we just introduce a code change to a demo server and we just test the heck out of it and ensure we are not creating more problems. If we have problems or if the risk is too big, that could be a reason to just do nothing with this bug.

Comment 26

[Josh Mize [:jgmize]]

(In reply to Chris More [:cmore] from comment #25)
> (In reply to Josh Mize [:jgmize] from comment #24)
> > (In reply to Mike Alexis [:malexis] from comment #23)
> > > looks like this is potentially blocked and ready to move forward. Josh, can
> > > you confirm?
> > 
> > I can confirm that we did some log analysis in bug 1040188, but I'm
> > uncertain about whether :cmore got the answers he was looking for or whether
> > further data and/or analysis is needed. :cmore, do you think the data we
> > could get from the additional UTM params might still be valuable?
> 
> Your log analysis looks pretty promising and locale redirects account for
> about 57-60%, which could explain why our (direct) traffic is so big.  
> 
> Josh: are adding any parameters (UTM or anything else) to locale redirects
> still a technical risk to the middleware mentioned above?
> 
> What if we just introduce a code change to a demo server and we just test
> the heck out of it and ensure we are not creating more problems. If we have
> problems or if the risk is too big, that could be a reason to just do
> nothing with this bug.

To clarify the risks, here is the section of the middleware that we're talking about modifying: https://github.com/mozilla/funfactory/blob/master/funfactory/middleware.py#L48-L72 -- there are a couple of branches that might need different UTM params depending on how you want to track it. The code changes themselves would be fairly straightforward and small, but the number of tests that this change would break in both the bedrock and mcom-tests suites is possibly large, which makes determining the level of effort in advance somewhat difficult, but I could put a PoC in a demo environment to help find out. Can we use the shorter params suggested in comment #17 or do we need the longer version?

Comment 27

[Chris More [:cmore]]

> Can we use the shorter params suggested in comment #17 or do we need the longer version?

Possible, but we just spreading the logic among disparate systems. Here's a shorter method:

no additional parameters:

1) visit www.mozilla.org
2) detects locale, redirects to www.mozilla.org/en-US/?icn=locale (r=redirect l=locale)
3) Rewrite the icn=locale URL in Google Analytics to ?utm_source=locale-redirect&utm_medium=referral

or with existing parameters before redirect:

1) visit www.mozilla.org?foo=bar
2) detects locale, redirects to www.mozilla.org/en-US/?icn=locale (r=redirect l=locale)
3) Rewrite the icn=locale URL in Google Analytics to ?foo=bar&utm_source=locale-redirect&utm_medium=referral

Comment 28

[Josh Mize [:jgmize]]

Is this issue still relevant, or should it be closed as WONTFIX? If the former, I will defer to :pmac on both implementation decisions and prioritization.

Comment 29

[Chris More [:cmore]]

Let's get :pg and :garethc to see if they are still interested in this given they run analytics now. The purpose of this bug was to help eliminate so much of the direct traffic on the site caused by locale redirects that drop the referring header.

Comment 30

[u570438]

+Jen Bertsch
If the work required to complete the solution listed in comment 27 is minimal, it would be good to complete it.

Then if we see that there is a large volume of traffic coming in listed in this way, then we know a more elaborate solution is appropriate.

Comment 31

[Paul [:pmac] McLanahan]

Attachment: Link to Github pull-request: https://github.com/mozilla/bedrock/pull/4536

Here's a start to the work. It'll be running on my demo[0] shortly. We still likely have a bunch of integration tests to fix, but we can figure that out after you've tried and approved these changes.

Comment 32

[Paul [:pmac] McLanahan]

Meant to link to the demo I mentioned in comment #31:

https://bedrock-demo-pmac.us-west.moz.works/

Comment 33

[pgerman]

I'm not seeing the parameters from comment #27: If we need to, I can be free to connect for most of the day.

Comment 34

[Paul [:pmac] McLanahan]

My reading of that comment is that locale redirects should get "?icn=locale" appended to them. This is happening on my demo.

Comment 35

[pgerman]

It should get:
?icn=locale&utm_source=locale-redirect&utm_medium=referral

or

?utm_source=locale-redirect&utm_medium=referral

Comment 36

[Paul [:pmac] McLanahan]

According to cmore's comment you can do that in GTM. We don't want that many people seeing awful URLs.

Comment 37

[pgerman]

I've made a rule in GA that looks for 'icn=locale' and then replaces it with 'utm_source=locale-redirect&utm_medium=referral'

Leaving open until it has taken effect on real time data and I can promote this to production.

Comment 38

[Paul [:pmac] McLanahan]

It seems we've let this one drop. All of the recent talk about experiment redirects and dropping referrers reminded me about this. Is this still a problem we should try to solve?

Comment 39

[Chris More [:cmore]]

(In reply to pgerman from comment #37)
> I've made a rule in GA that looks for 'icn=locale' and then replaces it with
> 'utm_source=locale-redirect&utm_medium=referral'
> 
> Leaving open until it has taken effect on real time data and I can promote
> this to production.

pg: is this rule still in place? would that mean that the moz.org team would have to append (?|&)icn=locale to the URL when a redirect happens and then GTM would take that single parameter and pass it into GA, so it is properly categorized as referral traffic? What if there are already UTM parameters on the site and the GTM rule sets it back to referral? 

For example.

acme.com links to: https://www.mozilla.org/?utm_campaign=foobar&utm_medium=display

then www.mozilla.org does a 301 redirect to www.mozilla.org/en-US/?utm_campaign=foobar&utm_medium=display&icn=locale, GTM would then change the medium from display to referral due to this rule.

Is that what we want?

Comment 40

[pgerman]

Hi Jon,

I don't know all that much about how traffic cop works, so depending on the nuances of the solution, we may need to figure something custom out.

By default GA will look to the following properties to set the referrer:
-documentLocation
-documentReferrer

If you can figure out a way for the redirect to pass these along, we're in good shape. If not, then we'll need to develop something custom.

This flow chart shows how referrer in GA is set(see flow chart): https://support.google.com/analytics/answer/6205762?hl=en#flowchart

Comment 41

[pgerman]

C40 was meant for another bug. Multiple windows open... urg. Please disregard.

Comment 42

[pgerman]

(In reply to Chris More [:cmore] from comment #39)
> (In reply to pgerman from comment #37)
> > I've made a rule in GA that looks for 'icn=locale' and then replaces it with
> > 'utm_source=locale-redirect&utm_medium=referral'
> > 
> > Leaving open until it has taken effect on real time data and I can promote
> > this to production.
> 
> pg: is this rule still in place? would that mean that the moz.org team would
> have to append (?|&)icn=locale to the URL when a redirect happens and then
> GTM would take that single parameter and pass it into GA, so it is properly
> categorized as referral traffic? What if there are already UTM parameters on
> the site and the GTM rule sets it back to referral? 
> 
> For example.
> 
> acme.com links to:
> https://www.mozilla.org/?utm_campaign=foobar&utm_medium=display
> 
> then www.mozilla.org does a 301 redirect to
> www.mozilla.org/en-US/?utm_campaign=foobar&utm_medium=display&icn=locale,
> GTM would then change the medium from display to referral due to this rule.
> 
> Is that what we want?

Currently, there are no rules in place to rewrite the utm parameters for ?icn=locale. TBH, I think we would want this traffic to retain its original source. 

If we want to mark this traffic as being redirected, then we would need to set something additional up involving a custom dimension instead of overwriting values contained in utm_source/medium/campaign/content.

Comment 43

[Chris More [:cmore]]

pgerman: what happened to the rule you mentioned in comment 37?

Comment 44

[pgerman]

I've found the rules that I made, they never got pushed. Based on the discussions with Gareth from last week, I've made the following changes:
1. Created a custom dimension called "Locale Redirect Session" the value for redirected traffic will be "TRUE"
2. Modified GA Advanced filter rule "Identify Locale Redirect Session"
3. Pushed to GA profile > www.mozilla.org (testing)

@Cmore - Tomorrow or Wednesday, please confirm that this is working in a way that you're comfortable with and I'll move it to the production GA view.

Comment 45

[Chris More [:cmore]]

(In reply to pgerman from comment #44)
> I've found the rules that I made, they never got pushed. Based on the
> discussions with Gareth from last week, I've made the following changes:
> 1. Created a custom dimension called "Locale Redirect Session" the value for
> redirected traffic will be "TRUE"
> 2. Modified GA Advanced filter rule "Identify Locale Redirect Session"
> 3. Pushed to GA profile > www.mozilla.org (testing)
> 
> @Cmore - Tomorrow or Wednesday, please confirm that this is working in a way
> that you're comfortable with and I'll move it to the production GA view.

For this rule, will it still require the mozill.org team to still provide a URL parameter (?icn=locale) when a locale redirect happens?

Comment 46

[Chris More [:cmore]]

Justin: please provide the steps here what you did to verify that maybe this is no longer needed and the referring header should get passed. Maybe we are good! We should also double check that this works outside of just Google searches.

Comment 47

[Justin Crawford [:hoosteeno] [:jcrawford]]

301 redirects (for locales and otherwise) appear to retain the document.referrer value that the session started with. Here's a test to prove it:

1. Visit http://bit.ly/2qsP1SK, which is a Google interstitial page that asks if you really want to go to a page on www.mozilla.org.

2. Confirm by clicking the URL. You will go to the URL https://www.mozilla.org/firefox/mobile/, which is double 301'd to https://www.mozilla.org/en-US/firefox/android/.

3. Open up the developer console and type `document.referrer`. You should see the long Google URL you just came from -- even though your browser jumped through a 301 redirect to /en-US and also a 301 redirect to /android.

If this doesn't work for you, try it in a browser where you don't have a lot of plugins installed. :cmore had to use Safari (but it works fine for me in Fx).

In other words, I don't think we need to jump through additional hoops to retain the referrer on 301 redirects. 

However, if a session comes in with a utm_source parameter, that will probably override any document.referrer in GA.

Comment 48

[Justin Crawford [:hoosteeno] [:jcrawford]]

Turns out, Bugzilla doesn't hide the referrer like many websites do. So just click here and then check document.referrer:

https://www.mozilla.org/firefox/mobile/

Comment 49

[Paul [:pmac] McLanahan]

Confirmed works for me in Nightly. Thanks :hoosteeno! Very nice work. Closing this out as invalid since it turns out that browsers do handle this appropriately. It is true that one can easily turn off sending referrer in browser settings, or have an addon that does that for you, and it's possible that moz.org sees more of those users than average. Also all in-product links and links from pages like about:home would not have a referrer. I'm not sure this explains all of our direct traffic, but it seems clear that it's not the locale redirects. Thanks all!

Comment 50

[Chris More [:cmore]]

Awesome! Thanks Justin