Open Bug 1712062 Opened 3 years ago Updated 2 years ago

New attribution code referral does not overwrite existing cookied attribution code

Categories

(www.mozilla.org :: Bedrock, defect)

Production
Desktop
Windows 10
defect

Tracking

(firefox88 affected, firefox89 affected, firefox90 affected)

Tracking Status
firefox88 --- affected
firefox89 --- affected
firefox90 --- affected

People

(Reporter: romartin, Unassigned)

References

Details

[Notes]

  • The issue is very easily reproducible in Windows Sandbox mode (more details here).
  • We have observed that the “attribution.ua” is successfully overwritten by being a chrome switcher first and then an edge switcher.

[Affected Versions]:

  • Firefox Release 88.0.1 (Build ID: 20210504152106)
  • Firefox Beta 89.0b14 (Build ID: 20210518190425)
  • Firefox Nightly 90.0a1 (Build ID: 20210519214756)

[Affected Platforms]:

  • Windows 10 x64
  • Windows 8.1 x64
  • Windows 7 x64

[Prerequisites]:

  • Have a VM where Firefox was never installed or have a Windows Sandbox ready.

[Steps to reproduce]:

  1. Open the pre-installed Microsoft Edge browser, download, and install the latest version of Firefox Release.
  2. Navigate to the “about:telemetry” page.
  3. In the “Home” section of the page search for “attribution” and observe the values.
  4. Quit Firefox and open Microsoft Edge.
  5. Navigate to the “addons.mozilla.org/en-US/firefox/extensions/” page.
  6. Click any of the “Recommended extensions”, download, and install Firefox using the “Only with Firefox--Get Firefox Now” button.
  7. Navigate to the “about:telemetry” page and observe the “attribution” values.
  8. Navigate to the “about:welcome” page and observe the 1st slide.

[Expected results]:

  • Step 7: The “attribution” values are now overwritten with the ones for the Firefox extension.
  • Step 8: The RTAMO “about:welcome” page is displayed.

[Actual results]:

  • Step 7: The “attribution” values are the ones from the 1st Firefox install.
  • Step 8: The default “about:welcome” page is displayed.

[Additional notes]:

  • The attribution code can be successfully overwritten from the “about:newtab#devtools-targeting” page.
  • The issue is reproducible using Google Chrome, Opera, and Internet Explorer.
  • The issue is also reproducible vice-versa, if the 1st install of the Firefox is by using an extension, any subsequent installation will display the RTAMO “about:welcome” page and the RTAMO attribution code.

Changing component to Installer team to help triage the issue, thanks

Component: Messaging System → Installer

I haven't spent much time working on the attribution system. I'm not really sure how serious of a problem this is, so I'm having trouble triaging.

@nalexander - I think that you are more familiar with attribution than I am. Could you help me determine how serious of a problem this is?

Flags: needinfo?(nalexander)

I was able to see newer attribution overwriting older one. Just tested with a windows 10 1809 firefox had this at c:\Users\User\AppData\Local\Mozilla\Firefox\postSigningData when installing from edge after searching from bing dated 2021/05/21 9:21am

campaign%3D%2528not%2Bset%2529%26content%3D%2528not%2Bset%2529%26dltoken%3Dc0374fa4-0e4e-4bcf-ba92-39299f05dde1%26experiment%3D%2528not%2Bset%2529%26medium%3Dreferral%26source%3Dwww.bing.com%26ua%3Dedge%26variation%3D%2528not%2Bset%2529

Then went to addons.mozilla.org which had a top extension recommended with a download firefox link leading from

https://addons.mozilla.org/en-US/firefox/addon/foxy-gestures/?utm_source=addons.mozilla.org&utm_medium=referral&utm_content=homepage-primary-hero

to

https://www.mozilla.org/en-US/firefox/new/?experiment=20210404_download_cta_experiment&utm_campaign=non-fx-button&utm_content=rta:e2U4MzljM2Y5LTI5OGUtNGNkMC05OWUwLTQ2NDQzMWNiN2MzNH0&utm_medium=referral&utm_source=addons.mozilla.org&utm_term=amo-fx-cta-803317-new-cta&variation=new-cta

and postSigningData correctly updated with timestamp 2021/05/24 3:44pm and content:

campaign%3Dnon-fx-button%26content%3Drta%253Ae2U4MzljM2Y5LTI5OGUtNGNkMC05OWUwLTQ2NDQzMWNiN2MzNH0%26dltoken%3Da32ba817-9c13-4afc-bb29-2bd343cedd36%26experiment%3D20210404_download_cta_experiment%26medium%3Dreferral%26source%3Daddons.mozilla.org%26ua%3Dedge%26variation%3Dnew-cta

with about:welcome showing RTAMO which also satisfied the old and new attribution check from bug 1707038.

romartin, do you see the postSigningData file before/after installing firefox and if so, share their contents (open in Notepad)?

Flags: needinfo?(romartin)
See Also: → 1707038

I also saw correctly attribution updated after installing from opera (searched "firefox" resulted in google search result page):

campaign%3D%2528not%2Bset%2529%26content%3D%2528not%2Bset%2529%26dltoken%3De3653d0a-1fd7-4a02-8162-673997d9de04%26experiment%3D%2528not%2Bset%2529%26medium%3Dreferral%26source%3Dwww.google.com%26ua%3Dchrome%26variation%3D%2528not%2Bset%2529

The ua was set to chrome based on Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 OPR/76.0.4017.123 which does match the logic in https://github.com/mozilla/bedrock/blob/d9eb07e8c47afd866a611ec73cc0394c3a22a482/media/js/base/stub-attribution.js#L179-L198 (it doesn't try detecting opera and notices the "Chrome" portion of UA) added in bug 1406005 -- not a bug.

See Also: → 1406005

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #2)

Could you help me determine how serious of a problem this is?

I believe the main uses of attribution are for about:welcome, general telemetry and experiment targeting. Functionality like bug 1703327 to pre-select source import browser based on attribution has resulted in improved days of use. And I believe AMO is currently running experiments that rely on attribution per bug 1707038 comment 5. So if this is generally broken, it would likely have impact on retention.

Although from my quick testing in comment 3 and comment 4, at least the stub installer seems to be correctly writing to postSigningData, so let's hear back from romartin and maybe this is broken in certain configurations.

Flags: needinfo?(nalexander)
See Also: → 1703327

Hey Ed,

I've recorded the behaviors in both Windows Sandbox and a Windows 10 x64 20H2 OS.

  • For the Windows Sandbox I've made a recording that can be seen here.
  • For the Windows 10 x64 20H2 I've made a recording that can be seen here.

For the "postSigninData" I've the following values:

  • The Google Chrome downloaded firefox stub-installer: campaign%3D%2528not%2Bset%2529%26content%3D%2528not%2Bset%2529%26dltoken%3D79a1f3aa-0321-4ef3-b34c-a639b2868cb6%26experiment%3D%2528not%2Bset%2529%26medium%3Dreferral%26source%3Dwww.google.com%26ua%3Dchrome%26variation%3D%2528not%2Bset%2529
  • The Google Chrome downloaded RTAMO (laser cat extension) firefox stub-installer:
    campaign%3D%2528not%2Bset%2529%26content%3D%2528not%2Bset%2529%26dltoken%3D517743e6-5303-4f80-b7e7-289f1ddfad8e%26experiment%3D%2528not%2Bset%2529%26medium%3Dreferral%26source%3Dwww.google.com%26ua%3Dchrome%26variation%3D%2528not%2Bset%2529
Flags: needinfo?(romartin)

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #2)

I haven't spent much time working on the attribution system. I'm not really sure how serious of a problem this is, so I'm having trouble triaging.

@nalexander - I think that you are more familiar with attribution than I am. Could you help me determine how serious of a problem this is?

Certainly. It's not a 5 alarm fire but attribution is a critical piece of data for us, so I think that S2 is appropriate.

Now, to the technical details. I think the question is "what happens with attribution data on a pave-over install", and I'm not sure I know the answer.

If you have an existing install with some attribution, and you pave-over to update it, and the new install has different attribution, what's the expected outcome? The AMO team expects the new attribution to "win". My instinct is contrary to that: we want attribution to be stable; if you came to Firefox from a particular source moons ago, we shouldn't forget that when you re-engage via different source. Especially because we are likely to serve pave-over installs that are not particularly informative, i.e., that aren't a campaign or anything like that but instead just saying "you came from an Edge browser".

@bytesized: you've done the most work on pave-over installs: do you know if we (Install/Update team, I guess) have a position on this behaviour? Has this come up before?

Flags: needinfo?(ksteuber)

(In reply to Nick Alexander :nalexander [he/him] from comment #7)

@bytesized: you've done the most work on pave-over installs: do you know if we (Install/Update team, I guess) have a position on this behaviour? Has this come up before?

I don't know that we, as a team, have a strong opinion on this exact behavior. In general, we do tend to consider a paveover install to be a bit more like an update than an installation. One reason for this is that if update is sufficiently broken (for whatever reason), we prompt the user to download a new copy of the installer. If we always overwrote attribution data on a paveover install, I'd guess that this action would lose the attribution data, which seems like it is probably undesirable.

Flags: needinfo?(ksteuber)

romartin, I think I might have figured out what's going on. Because the page triggering the download can be different from the download button, bedrock uses cookies to remember the data for attribution:
https://github.com/mozilla/bedrock/blob/7f6e3ea9e002360a62ceeb3aa6ed76477b55da7b/media/js/base/stub-attribution.js#L324-L330

So I believe attribution is not updating because your cookies are set from the first time you visited while testing. So the second installer you download is using the cookie data from the first download. If you clear all cookies or use a different browser to download the second installer, do you still see this bug?

Flags: needinfo?(romartin)

Hey Ed,

I've tried out what you suggested and the results are as follows:

  • Windows 10 x64 20H2:
    • Downloading the firefox stub with edge and then downloading the RTAMO stub with chrome works, the RTAMO attribution code will overwrite the edge-switcher attribution code.
    • I've cleared the cookies from both edge and chrome after the first stub was downloaded and then downloaded the RTAMO stub resulting in the attribution code being correctly updated to the RTAMO attribution code.
  • Windows Sandbox:
    • Same results as the ones on Windows 10 x64 20H2.
Flags: needinfo?(romartin)

From comment 9 and comment 10, moving this to bedrock

Component: Installer → Bedrock
Product: Firefox → www.mozilla.org
Summary: New attribution code does not overwrite already existing attribution code set by the 1st Firefox install → New attribution code referral does not overwrite existing cookied attribution code
Version: Trunk → Production

[Actual results]:

Step 7: The “attribution” values are the ones from the 1st Firefox install.

As far as stub-attribution goes, the actual results (Step 7) are largely the expected results here. The attribution system was designed so that the first touch point of a marketing campaign (i.e. the first landing page someone visits) is what gets attributed during the install. This is because people can either (a) navigate through several pages before deciding to download, or (b) come back to the site a bit later on once they have formed a decision and are ready to install. And also (c) try to download the product twice on the same day...

The cookie we set for attribution is 24 hours, so if anyone downloads a second time within this window, the first source of attribution will be used. Using a cookie in this way also helps to limit the number of auth requests we need to make to the system.

We could potentially try and clear the cookie once a download begins, but this doesn't guarantee that someone actually runs the installer right there and then. I'm not sure on how much data that might effect.

But I guess the bigger questions are: how likely is it that someone tries to download Firefox twice within 24 hours? And if they did, would we want to attribute their first install, or the second? (I know this bug is about RTAMO, but that's not the primary thing stub attribution was built for - we also have to consider new user attribution).

If you have an existing install with some attribution, and you pave-over to update it, and the new install has different attribution, what's the expected outcome? The AMO team expects the new attribution to "win". My instinct is contrary to that: we want attribution to be stable; if you came to Firefox from a particular source moons ago, we shouldn't forget that when you re-engage via different source. Especially because we are likely to serve pave-over installs that are not particularly informative, i.e., that aren't a campaign or anything like that but instead just saying "you came from an Edge browser".

Since it hasn't come up explicitly: if attribution is not stable, the download token (dltoken) loses much of its value.

You need to log in before you can comment on or make changes to this bug.