1832715 - Multistage Spotlight OPEN_URL actions have UTM params duplicated by aboutwelcome bundle

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P2)

Description

[Shane Hughes [:aminomancer]]

A Spotlight message that specifies its own utm params, will get duplicate aboutwelcome-specific params added to any OPEN_URL or SHOW_FIREFOX_ACCOUNTS action. This probably isn't critical because it seems likely most consumers will read the first param value and discard the duplicate, so the message's params should win between the two.

So this probably isn't a high priority but it would be nice to remove any uncertainty. The solution I'm thinking of is that we iterate over the BASE_PARAMS, and for each: if there is no existing param, we add the base param specified in addUtmParams.js. If there is an existing param, we do nothing.

Comment 1

[Shane Hughes [:aminomancer]]

Attachment: Bug 1832715 - Allow multistage spotlight OPEN_URL actions to include UTM params. r=#omc-reviewers

Comment 2

[Punam Dahiya [:pdahiya]]

(In reply to Shane Hughes [:aminomancer] from comment #0)

Right now it's impossible for a spotlight message to specify its own utm params, because the aboutwelcome bundle overrides them with aboutwelcome-specific values for any OPEN_URL or SHOW_FIREFOX_ACCOUNTS action. I wish I'd noticed this before since we really need this functionality in 114 for bug 1832696.

UTM params specified in url in config JSON should be retained as is and not overridden , see Mobile Download screen in about:welcome. As we have more campaigns using these params it will be nice to avoid duplication but shouldn't be a blocker for analytics due to duplicate utm params received.

Comment 3

[Shane Hughes [:aminomancer]]

My mistake, you're totally right. JS' searchParams.get() apparently uses the first/earliest param value when there are duplicates. And the first one would be whatever we specify in the message configuration, with aboutwelcome values being added later. So I think you're right this isn't an urgent problem, unless SUMO's analytics parses it in some other way that reverses the order.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/13dcb9e0987f
Allow multistage spotlight OPEN_URL actions to include UTM params. r=omc-reviewers,negin

Comment 5

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/13dcb9e0987f

Comment 6

[Marius Coman [:mcoman], Ecosystem QA - PTO until 8th of November, for any requests ping :cmuresan]

I have verified this enhancement using the latest Firefox Nightly 116.0a1 (Build ID: 20230611214645) installed on Windows 8.1 x64 and I can confirm the following:

• The support.mozilla.org link contains the following UTM parameters and no duplicates are encountered:
  • utm_source=spotlight-default
  • utm_medium=firefox-desktop
  • utm_campaign=migration
  • utm_content=how-to-backup-data
• The follwing UTM parameters are not displayed in the support.mozilla.org link:
  • utm_source=activity-stream
  • utm_campaign=firstrun
  • utm_medium=referral

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

From looking at the code and this patch is that in spotlights (and maybe feature callouts?), utm_{source,campaign,medium} values specified in an experiment recipe or messaging system itself will be overwritten with these values (respectively) in Firefox 115 and earlier:

    utm_source=activity-stream
    utm_campaign=firstrun
    utm_medium=referral

Is there any expectation that any of the experiments done on 115(eg device migration & maybe others) will need/want to specify these keywords that way for attribution tracking?

If so, we'll want to uplift this to 115 and/or 115esr...

Comment 8

[Shane Hughes [:aminomancer]]

Copied from slack for some context:

We didn't uplift this before because the params aren't overridden, it just gets duplicate params appended. At the time, we thought the first params would take precedence over the duplicate params, so it wouldn't matter. We learned later that it actually seems the later params take precedence, at least with SUMO's google analytics.

So our spotlight analytics were getting messed up for the device migration spotlight experiment. but that was targeting 114, and by the time we discovered this, 114 was already shipped. So, I landed in 116 and there didn't seem any point to uplift to 115, since the audience included 114. And we were able to just treat the duplicate param values as "synonyms" for the experiment's intended param values in our analysis, because that experiment's 1 spotlight was the only thing sending users to that URL with the duplicate params.

Although google analytics picked up the wrong param values, the data was effectively the same. For that experiment there was only 1 kind of spotlight leading to that page, so there wasn't anything to disambiguate. In other cases, it could be possible to have 2 spotlights with URLs to the same page but with different UTM params, used to distinguish 1 branch from the other for example. And then it would be a problem if that experiment was targeting 115 or earlier.

Sorry if this is a bit confusing. It basically means there won't be a problem unless:

1. Multiple Spotlight experiments, or multiple branches within an experiment, contain URLs leading to the same page, but with different UTM param values;
2. And the experiment(s) in question are all targeting 115 or earlier.

If the above happens, then for example, branch 1 and branch 2 will send users to the same URL, with the same UTM params, making it impossible to distinguish one from the other in google analytics. I'm not aware of any experiments like this, but if anyone has any suspicions, send me the experiment links and I can try to figure out if it's going to be a problem.

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

I discovered this from the last (quickly aborted) VPN spotlight experiment, where the looker data had utm_campaign incorrectly set to firstrun, FWIW.

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

I should add that I believe/assume this affected the VPN experiment because later duplicated UTM params also seem to take precedence with our VPN subscription dashboard on Looker (and presumably Looker more generally?).

Comment 11

[Shane Hughes [:aminomancer]]

Comment on attachment 9333237 [details]
Bug 1832715 - Allow multistage spotlight OPEN_URL actions to include UTM params. r=#omc-reviewers

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Windows 7/8 users will end up in ESR 115, which increases the likelihood that we may target this audience with a Nimbus experiment. If that experiment involves about:welcome, Spotlight, or Feature Callout messaging, and includes affordances that send users to pages (e.g. on mozilla.org) with UTM parameters for Google Analytics, then this bug may cause the parameters to be wrong, potentially ruining the analytics data. Uplifting this to ESR 115 will ensure the analytics data is correct.
• User impact if declined: It will be harder to perform messaging experiments targeting Windows 7/8 users.
• Fix Landed on Version: 116
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The patch is just a very small amount of code that only affects whether UTM parameters are automatically added to URLs when the user clicks a button in about:welcome or in a message. Also, the file in question is rarely edited and it is a clean graft.

Comment 12

[Dianna Smith [:diannaS]]

Comment on attachment 9333237 [details]
Bug 1832715 - Allow multistage spotlight OPEN_URL actions to include UTM params. r=#omc-reviewers

Approved for 115.1esr

Comment 13

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/43db2f5e45bb

Comment 14

[Shane Hughes [:aminomancer]]

Sorry, I should have attached a comment as well. We were hoping this could be verified as fixed in ESR 115 if possible. Thanks!