Closed Bug 1316136 Opened 4 years ago Closed 3 months ago

Allow for attribution on the full installer

Categories

(Firefox :: Installer, enhancement, P2)

enhancement

Tracking

()

RESOLVED FIXED
Firefox 77
Tracking Status
firefox52 --- wontfix
firefox77 --- fixed

People

(Reporter: mkaply, Assigned: agashlin, NeedInfo)

References

(Depends on 1 open bug, )

Details

User Story

As a product manager, I want to bring exe full installer new profiles originating from moz.org downloads to the light funnel so I can better understand the share of new profiles that originate from this part of the funnel and optimize adequately.

As a product manager, I want to make the process of distributing Firefox through 3rd party partner sites simpler whilst retaining the ability to identify Firefox profiles originating from downloads made through distribution partner websites, so that setting up new distribution partners is less work.

Acceptance criteria:
- exe full installer attribution (MSI will be another phase) is supported on the full installer through addition of a dummy certificate when signing
- A new partner repack mode is added to support attribution
- Attribution is automatically added for any full installer download links used on moz.org pages (Bedrock) - should cover https://www.mozilla.org/en-US/firefox/installer-help and https://www.mozilla.org/en-US/firefox/all/#product-desktop-release

Attachments

(2 files)

There is work going on in bug 1259607 to add attribution to the stub installer.

It would also be useful to add attribution to the full installer.

Right now when we give builds to other websites to distribute, if we want to track it, we have to give them a custom distribution/partner build.

I'd much rather have it work the same way as the stub installer and be able to create a custom build that just has the campaign stored in the full installer.
Component: General → Installer
Too late for firefox 52, mass-wontfix.
Very little of the work involved for this would be in the installer; all that's needed from that side is to move the function from the stub installer into shared.nsh and insert a call to it in the full installer. There would be much more work involved in getting full installers with attribution data created in the first place.
Priority: -- → P5
> There would be much more work involved in getting full installers with attribution data created in the first place.

Can you provide more info? I thought the stub installers attribution was added dynamically.

We can't do that with the full installers?
I think I implied more certainty there than I actually have; sorry about that. I didn't closely follow what was going on with stub attribution on the backend once the client issues were resolved, so I don't know those details. I just wanted to point out that, while there would be very little work involved in the installer/client code to make this happen (assuming the existing telemetry is sufficient for the uses you're thinking about), hooking up all the backend services is probably harder and certainly involves more people (cloud ops and web would have work to do, might need to involve product also).
Depends on: 1453613
This is something that is interesting for the China folks as well (and will be part of a new distribution strategy if we get one together)

The primary usecase here is being able to pass some sort of distribution ID to a full installer just so when it runs for the first time, it stores that distribution ID internally so it can later be queried (I assume using the same mechanism as the campaign ID).

Is this practical at all?
Well, attribution is really intended (and best-suited) for data that isn't available at build time. I don't know anything about the new distribution strategy you mentioned, but if it's structured so that the distribution becomes something that's sort of layered on top of a normal full installer, then attribution would be a good way to get the distribution ID and any other required information into that full installer. We'd have to add a better way to query that information individually within Firefox, because right now it really only supports telemetry, but that would be very simple to do. If the distribution stuff is still baked into the installer as part of the application files, then attribution probably isn't the right method.
I definitely want to move away from baking the distribution stuff in the installer.

I picture a world where you download Firefox with a distribution ID passed to the installer and then the distribution is added dynamically.
Type: defect → enhancement

After discussion with Matt, we think this is straightforward.

Move detection of attribution file from stub to full installer.

Figure out how easy it is to have partner repackaging bundle the attribute file.

(In reply to Mike Kaply [:mkaply] from comment #8)

Figure out how easy it is to have partner repackaging bundle the attribute file.

Can you tell me more about this part ?

Can you tell me more about this part ?

Sure. The easy attribution works is there is a file in the directory where Firefox is installed.

So similar to how distribution directory works, we would repackage Firefox with this one file.

(This hasn't been prioritized yet which is why I haven't brought it up. This was a discussion Matt and I had at all hands)

More specific detail on how this would work.

Attribution currently works because the stub installer puts a file called postSigningData into the AppData\Local\Mozilla directory. This is created from signature data in the stub installer.

Then Firefox reads that file to get the attribution

The thought is that for the full installer, the postSigningData file would be bundled directly into the installer and then the installer would copy the file over if it exists.

So we would be able to reuse the existing functionality for full installer.

We really need to revisit this. As we're trying to look at dark funnel problems, this would be extremely useful.

Assignee: nobody → agashlin
Priority: P5 → P2
User Story: (updated)
Depends on: 1630154

Spelling out some caveats of the design in case any of this might be unexpected:

In the common case, the full installer will just write its attribution/post-signing data where it can be read via the same Firefox mechanisms that were added for the stub. This will use the same telemetry pipeline and generally be in the same dataset as the currently collected data for stub attribution, so the attribution data itself will have to indicate which is stub and which is full if those need to be distinguished.

The full installer has an install ping similar to the stub if it is run standalone (bug 1453613, using a different telemetry ingestion method). The stub ping includes the post-signing data, and I will add this into the full installer ping in the "attribution" field that was reserved for this purpose.

Both the stub installer and the full installer may have their own attribution data, but since the user manually downloaded the stub, I will explicitly ignore the full installer's attribution data if it is run by the stub. This seems like the only reasonable thing to do, but it should be expected that any attribution on those stub-driven full installer downloads will never appear further down the funnel. Note that this will not affect any download from the "Your download was interrupted" page, as that is entirely independent of the stub; any attribution data on those installers will be used when the installer is run.

Similarly this will be ignored with an MSI install, since the full installer within the .msi won't have relevant attribution. We haven't implemented a mechanism for attributing .msi files.

This reuses some logic from the stub installer, moved into common.nsh.

Pushed by agashlin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7690541a2c14
Support post-signing/attribution data in full installer. r=mhowell

Landing in progress. Once this lands, we will start to see "0" in the "attribution" field for full installer pings, indicating that the installer supported attribution but didn't find postSigningData. It is ready to use the dummy certificate once that is included when signing.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 77

The dummy cert change has landed too.

Hey all, I'm trying to find the new attribution value in our telemetry. I can't find any users with attribution.source = 0 in the main ping (example). Should I be looking somewhere else? Thanks!

Flags: needinfo?(agashlin)

The "0" I mentioned in comment 16 is only expected in the installer ping, you can see that show up in this query: true means attribution = 0 (supported but no attribution present), false is real attribution data, null means nothing was submitted (older installers or possibly an error).

We don't add anything that would let Firefox distinguish full installer attribution from stub installer attribution, so I wouldn't expect to be able to recognize full installer attribution in the main ping, unless the attribution data included in the installers is different between stub and full.

Flags: needinfo?(agashlin)

Awesome. Thanks for the explanation, Adam.

To confirm my understanding, we don't expect this change to give us any new information about user attribution, correct? For context, we have a KR set to increase retention for attributed new users. I want to understand whether we expect to see additional attributed users.

Flags: needinfo?(agashlin)

I expect more clients to report attribution after this change, all of those that are installed with an attributed full installer should now report that attribution in their pings, while formerly attribution was only included with the stub installer. The "false" line in the query I linked indicates that there are a substantial number of installations using an attributed full installer, I'd expect most of those to show up as new attributed clients in the Firefox pings. (I'm not sure what the attributed you quote refers to.)

My point was just that Firefox doesn't know what type of installer was used, so it wouldn't be straightforward to look at an attributed client and say "this wouldn't have been attributed without bug 1316136". If that wasn't what you asked I'm sorry to confuse!

Flags: needinfo?(agashlin)

@Ryan See https://sql.telemetry.mozilla.org/queries/70644#177786 that shows about 80k incremental installs with attribution from release full installer.

Seems like our data are conflicting then. GUD shows stable counts for Attributed New Users (source). If I understand correctly, we'd expect to see a spike of ~80k users in early June (a ~25%-50% boost).

To estimate attributed new profiles more conservatively, I added a restriction of NOT had_old_install, to eliminate re-installs (likely no new profile), and new_launched to only count those that started Firefox from the installer (this cuts out a surprising ~50%, likely an undercount, but this way we know that Firefox started and so is very likely to create a profile and send telemetry on shutdown). This query tops out at 12K per day. As a sanity check, I downloaded a full 79.0 installer from a search, and the install submitted the correct attribution in the new-profile ping (according to about:telemetry), while a download directly from archive.mozilla.org did not submit any attribution.

It's beyond my expertise how to recognize this among larger trends in the GUD data set. Do you know if this dashboard counts only a specific part of the attribution value?

Thanks, Adam. That's super informative.

It's beyond my expertise how to recognize this among larger trends in the GUD data set. Do you know if this dashboard counts only a specific part of the attribution value?

GUD calls a user attributed if either attribution.source or attribution.campaign are non-NULL (source).

The 12k/day attributed new user count is ~5% of total Windows new attributed user. I'd be surprised if this wasn't visible in the GUD graph, but I suppose it's possible. Would you mind verifying that one of the above fields is non-NULL when installed from the full installer?

Flags: needinfo?(agashlin)

(In reply to Ryan Harter [:harter] from comment #26)

Would you mind verifying that one of the above fields is non-NULL when installed from the full installer?

Yes. For completeness, steps to reproduce on a fresh Windows Sandbox image, in Edge:

  1. google.com, typed "firefox", clicked the first result
  2. clicked the Download Firefox button on https://www.mozilla.org/en-US/firefox/
  3. cancelled the download of the stub installer
  4. clicked the "Download in another language or for another operating system" link
  5. clicked Download Now (the defaults are (Firefox, Windows 64-bit, English (US))
  6. ran the installer
  7. left the check box checked to run Firefox at the end of the install, let it start for the first time
  8. restarted Firefox
  9. about:telemetry shows attribution in the archived new-profile ping (Environment Data section):
attribution.medium 	referral
attribution.source 	www.google.com
attribution.ua 	edge

The other attribution fields were %2528not%2Bset%2529.

Flags: needinfo?(agashlin)

Thanks for the clarification, Adam.

CC: ethompson. This should be creating a 5% increase in attributed new users but does not appear to be. Can we have someone from the DS team take a second look?

Flags: needinfo?(ethompson)

So according to this it appears we could be trading attributed stub installs for attributed full installs, such that the net number of attributed installs overall has not moved much (maybe a tad bit up). Also the fraction of installs that are attributed overall does not seem to be moving THAT much either.

Is this expected behavior? The decline in attributed stub installs seems to coincide pretty much exactly with the release of the patch. Could it be that we are now just recording the attribution for full installs in cases where both the full installer and stub are present?

Flags: needinfo?(ethompson)

Interesting... I wouldn't expect this to happen when both are present, but it sure does look like that from the data, doesn't it? There is a check so that the full installer run by the stub installer doesn't report its own attribution, and the full installer doesn't even send telemetry if it was launched from the stub. If the stub has its own attribution, it would later overwrite whatever the full had written anyway.

Nick, do you know if there was a change on the server side, sending full installers when it would have previously sent stub installers, perhaps coming along with the changes that allowed attributed full installers?

Flags: needinfo?(nthomas)

I'll forward that query on to Justin for his Bedrock expertise.

For the related issue of how download.m.o and the related attribution service is configured, I'm not aware of any changes there. eg The stub prefs for the full installer URL still point to an un-attributed build.

Flags: needinfo?(nthomas) → needinfo?(hoosteeno)

(In reply to Leif Oines [:loines] from comment #29)

So according to this it appears we could be trading attributed stub installs for attributed full installs, such that the net number of attributed installs overall has not moved much (maybe a tad bit up). Also the fraction of installs that are attributed overall does not seem to be moving THAT much either.

Is this expected behavior? The decline in attributed stub installs seems to coincide pretty much exactly with the release of the patch. Could it be that we are now just recording the attribution for full installs in cases where both the full installer and stub are present?

Installs and new profiles decrease during summer time son maybe what we're observing on the stub installs is just the summer drop?
Stub installs decline seem to align well with last year's drop: https://sql.telemetry.mozilla.org/queries/3651#7209

Yes that's also a possibility. Could just be (very) coincidental timing of the patch. Not all seasonal patterns have been "normal" this year though. Normally I'd say we'd know if its seasonality in 1-2 months but now.... I'm not sure.

I'm going to ask :agibson to look at comment 30. Alex, there is a shift in attribution from stub to full installers, and the thread above is trying to understand it. One idea proposed is that a change on bedrock in the last couple weeks caused downloaders who previously would've received a stub install to get a full install instead. I'm aware of nothing like that; are you?

Flags: needinfo?(hoosteeno) → needinfo?(agibson)

The image attached here is a good illustration of the mystery the latter comments are concerned with.

You need to log in before you can comment on or make changes to this bug.