1485429 - Preserve original referrer on vanity domain referrals

Status: RESOLVED FIXED

(Infrastructure & Operations :: Runtime, task)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

When people request firefox.com/getfirefox.com, they are 301 redirected to the Firefox download page with some utm parameters (for GA) that indicate a referral. Here are the URLs they are 301 redirected to:

https://www.mozilla.org/en-US/firefox/new/?utm_medium=referral&utm_source=firefox-com

https://www.mozilla.org/en-US/firefox/new/?utm_medium=referral&utm_source=getfirefox-com

Google analytics uses these utm parameters to classify the source of traffic. But firefox.com and getfirefox.com are not the source of traffic: They are owned domains that get referrals and direct traffic from all over. We don't know where that traffic comes from, though, because it's lost in the redirect.

We should reconfigure or redesign the redirect for these vanity domains so we can retain the true source and medium of the referral as well as the fact that it came through the vanity domain. 

The specific implementation of this requires some discussion. I see two clear options, and want to hear what other domain experts think is best.

1) Add parameters for the original referrer, e.g. ?utm_medium=referral&utm_source=getfirefox-com&original_medium=referral&original_source=arstechnica.com


2) Add parameters for the vanity domain and remove it from utm, e.g. ?utm_medium=referral&utm_source=arstechnica&vanity_referral=firefox.com

I am partial to the latter (#2 above) because it gives us the ability to see the entire scope of our referral channels using standard GA reports, while retaining our ability to segment by vanity domain in the infrequent case where that is critical information. The former would forever isolate these referrals from standard reporting on referrals in GA. :cmore and :pg, would you please weigh in on this?

I'm also interested in any feasibility questions -- is this possible with the current redirecting mechanism or does it require a redesign?

Comment 1

[Chris More [:cmore]]

This is interesting and would be helpful as these vanity domain uses are a big blind spot. Here's some related bugs bug 897545 and bug 896570 to help understand previous changes.

Comment 2

[Justin Crawford [:hoosteeno] [:jcrawford]]

:cmore followed up with me separately and said he prefers option 2 above.

Comment 3

[Paul [:pmac] McLanahan]

I just did a test:

1. Go to a page on a popular domain (I chose https://en.wikipedia.org/wiki/Firefox)
2. Use dev tools to change a link on the page to `https://getfirefox.com`
3. Click the link and be taken to:
   https://www.mozilla.org/en-US/firefox/new/?utm_medium=referral&utm_source=getfirefox-com
4. Open devtools console and type `document.referrer`
5. Result is `"https://en.wikipedia.org/"`

This means that GA should be able to record both the UTM params saying that it came from the vanity domain and the referrer indicating the domain from which they clicked to get to the vanity domain. So I'm wondering if this is more of an issue with the data collection method than with the redirector.

Comment 4

[Justin Crawford [:hoosteeno] [:jcrawford]]

Great test, :pmac.

It's possible that we could configure GA to do something nonstandard in this instance, e.g. to map the document.referrer to the source. 

That said, GA's default behavior is to use the utm_source for the source of referrals, if that parameter is present. I think this document (and particularly the flowchart) explains it: https://support.google.com/analytics/answer/6205762?hl=en. I can't find any secondary source information using GA's standard report fields; I wonder if document.referrer is simply dropped on the floor when utm_source is present.

My preference would be to not reconfigure GA's behavior to accommodate what appears to be a mistaken use of utm parameters in a redirecting system we control. I'd rather get utm parameters set up more pragmatically with the actual source in the utm_source. 

I'm going to ask our analytics team to review this and encourage them to opine on the best way forward.

Tagging :erenaud in, since he's our conduit to that team. :erenaud, could you ask Analytics Pros to consider how best to accomplish the ask in this bug?

Thanks!

Comment 5

[Paul [:pmac] McLanahan]

Then it sounds like the quickest and easiest solution might be to have the redirector stop adding a utm_source parameter and to instead put e.g. "getfirefox-com" in some other parameter. In this way GA can record the referrer as the source like it normally would and potentially record the getfirefox.com bit in another place.

Alternately we could do some magic in the client code on bedrock to look for "getfirefox-com" as the source and when there is also a referrer to have GA record that in some way.

I prefer the first option myself as it seems like it would do what we want with minimal effort and chance for error.

Comment 6

[Chris More [:cmore]]

Justin: check out the "user defined value" field in GA when looking at the firefox.com or getfirefox.com redirects. See if the full refer is in there.

I added that filter that pulls in the full refer into the separate field. It may still over overwritten by the UTM parameters, but probably worth checking out.

Comment 7

[Justin Crawford [:hoosteeno] [:jcrawford]]

I agree with comment 5: Google will handle the referral link in the expected way if we just stop clobbering it with a utm parameter. So the implementation of this bug should be to change the current referral behavior of firefox.com/getfirefox.com to put the vanity URL in a different parameter. 

* Let's keep the current redirect method for both domains (301)
* Let's keep the current redirect regex for both domains (applying it to the same incoming requests) 
* Let's change the redirect target. It should no longer have utm* parameters; instead, it should get a parameter, "redirect_source=firefox.com" or "redirect_source=getfirefox.com".

Regarding comment 6, I checked that field and it's hard to understand what's landing in there now. The majority of referrals have  (not set) in that field, and a sizeable remainder have the redirect target, not a referral source. Debugging that is likely more work than doing the above, and doing the above instantly makes our data better, so let's do the above.

Comment 8

[Justin Crawford [:hoosteeno] [:jcrawford]]

One addition to the specs above:

We need to _retain_ any utm parameters that the original request includes. So, if someone visits firefox.com/?utm_source=example.com, we should pass that through the redirect.

Comment 9

[Firefox Bug Husbandry Bot]

Bulk reset of QA contacts in Infra and Ops bugs. 

https://bugzilla.mozilla.org/show_bug.cgi?id=1495849

Comment 10

[Eric Ziegenhorn :ericz]

Attachment: redirects config patch to fix utm_source and QSA as desired

Attaching a patch to switch parameter name from utm_source to redirect_source for firefox.com and getfirefox.com as well as preserve any incoming parameters such as utm_source.  Thanks to Josh for helping!

Comment 11

[Josh Mize [:jgmize]]

Comment on attachment 9034292 [details] [diff] [review]
redirects config patch to fix utm_source and QSA as desired

Review of attachment 9034292 [details] [diff] [review]:
-----------------------------------------------------------------

as discussed in slack, this is close but we need to remove all of the utm* params from the target as requested in comment #7

Comment 12

[Eric Ziegenhorn :ericz]

Attachment: redirects config patch to fix utm_source and QSA as desired

Updated to not set any additional utm parameters (but still preserve existing ones), as per comment 7, thanks for catching that Josh.

Comment 13

[Josh Mize [:jgmize]]

Comment on attachment 9034299 [details] [diff] [review]
redirects config patch to fix utm_source and QSA as desired

Review of attachment 9034299 [details] [diff] [review]:
-----------------------------------------------------------------

For consistency, I think we should add ?redirect_source=firefox-com to the targets in lines 167-173. Other than that this LGTM.

Comment 14

[Eric Ziegenhorn :ericz]

I had those redirect_sources in there at one point but lost them, good catch.  Re-added, and deployed.  To summarize we are now making firefox.com redirects add a 'redirect_source=firefox-com' parameter and we're passing along any existing parameters such as utm_source unchanged.  Anything else here?

Comment 15

[Eric Ziegenhorn :ericz]

I forgot to mention we also stopped adding utm parameters such as utm_medium in the redirect itself.

These changes also applied to getfirefox.com and mozillafirefox.com, which begs the question what other vanity urls are we interested in?   We have other domains that simply redirect to somewhere on mozilla.org.  There are a number of variations such as getfirefox.co.uk or firefox.lt, do we want to do the same with all of them?

Comment 16

[Justin Crawford [:hoosteeno] [:jcrawford]]

(In reply to Eric Ziegenhorn :ericz from comment #15)

> These changes also applied to getfirefox.com and mozillafirefox.com, which
> begs the question what other vanity urls are we interested in?   We have
> other domains that simply redirect to somewhere on mozilla.org.  There are a
> number of variations such as getfirefox.co.uk or firefox.lt, do we want to
> do the same with all of them?

We probably do! And, we shouldn't block this bug for them, since the domains addressed here are the largest sources of redirects. :ericz, what's the best way to get a list of such domains? Can you share it with me in an email? If so, I will file a secondary bug (cc'ing you) with specific recommendations per domain. Thanks for working on this.

Comment 17

[Eric Ziegenhorn :ericz]

I'll make a list and share it with you Justin.

Comment 18

[Eric Ziegenhorn :ericz]

Pinged on Slack and list emailed. Closing this bug.