1372307 - At 100% user activation, re-ship follow on search count addon to 54 via GoFaster

Status: RESOLVED FIXED

(Firefox :: Search, enhancement, P1)

Description

[Shilpi Gupta [:shilpi]]

+++ This bug was initially created as a clone of Bug #1371807 +++

This bug is to ship to 54 the same system addon for follow on counts that was shipped to 53.

Comment 1

[Shilpi Gupta [:shilpi]]

Goal:
Ship follow on count search addon (modified code with perf fizes) to 54 via GoFaster to 100% users by 6/22

Plan: 

1. Mark Banner to provide updated code (with perf fixes and removed 10% annotation) early next week.
2. Justin to provide QA sign off on updated add on mid next week
3. Shilpi to send Intent to ship email to r-d mid next week 
4. Rehan to ship via GoFaster by end of next week (6/22)

Comment 2

[Panos Astithas (he/him) [:past] (please ni?)]

(In reply to [e-mail] from comment #1)
> 1. Mark Banner to provide updated code (with perf fixes and removed 10%
> annotation) early next week.

Clarification: make the add-on available for 100% of the population and remove the TelemetryEnvironment.setExperimentActive call that makes the data pipeline take a more expensive path.

Comment 3

[Mark Banner (:standard8)]

Oops, I forgot about this bug when filing bug 1375564.

Comment 4

[Panos Astithas (he/him) [:past] (please ni?)]

Dependencies are cleared, so closing this.

Comment 5

[Ryan Harter [:harter]]

Hey All,

This query [1] suggests that only 45% of all users are reporting searches via our addon. Rehan, can you confirm that this addon is rolled out to 100% of users?

[1] https://sql.telemetry.mozilla.org/queries/5398#table

Comment 6

[Rehan Dalal (Inactive)]

Attachment: Balrog.jpg

I just checked and Balrog has been configured to ship the Addon to 100% of users.

Comment 7

[Panos Astithas (he/him) [:past] (please ni?)]

Mark, any ideas?

Comment 8

[Mark Banner (:standard8)]

Unfortunately no. Ryan, I believe there's a way to check if the add-on is installed in the profile or not from the telemetry data. So it might be worth comparing with that.

Also we will only be enabled for profiles without a distribution id, I doubt that would account for 50% but might be worth checking anyway.

Comment 9

[Ryan Harter [:harter]]

Thanks, all. I put together a couple of analyses that measure the add-on uptake by counting the number of client_ids reporting '[e-mail]' in `active_addons`. These analyses suggest uptake is unusually low.

This query [1] reports similar uptake (42.66%) for a 1% sample of data between 2017-07-01 and 2017-01-04 inclusive. 

I also have a more involved analysis here [2]. That analysis verifies this number is stable (42.49%) for a 100% sample of data over the same period. That analysis also stratifies over a bunch of factors (e.g. locale, channel, build_id, distribution_id). It looks like we have higher uptake for more recent builds and versions. However, very few groups report uptake >50%.

Rehan, would you expect this add-on to be installed for more than 50% of all `client_id`s by now?

[1] https://sql.telemetry.mozilla.org/queries/5464
[2] https://github.com/mozilla/search-adhoc-analysis/blob/master/follow-on-addon-uptake/Follow%20On%20Uptake%20Investigation.ipynb#overall%20numbers

Comment 10

[Rehan Dalal (Inactive)]

I would definitely expect it to have been installed for more than 50% of all client ids by now. 

Perhaps Ben Hearsum has some insight?

Comment 11

[Mark Banner (:standard8)]

(In reply to Ryan Harter [:harter] from comment #9)
> This query [1] reports similar uptake (42.66%) for a 1% sample of data
> between 2017-07-01 and 2017-01-04 inclusive. 
> 
> I also have a more involved analysis here [2]. That analysis verifies this
> number is stable (42.49%) for a 100% sample of data over the same period.
> That analysis also stratifies over a bunch of factors (e.g. locale, channel,
> build_id, distribution_id). It looks like we have higher uptake for more
> recent builds and versions. However, very few groups report uptake >50%.
> 
> Rehan, would you expect this add-on to be installed for more than 50% of all
> `client_id`s by now?

I think we should ignore any client_id where the version is less than 54.0. That's what we're really interested in at the moment. From your second link, it is looking like around 80% of those users have the add-on installed. FF 55 has it being shipped by default so is near 100% (0.995277) which feels about right.

We should also ignore any client_id where there is a distribution id - we don't really care about them, although in theory they should be getting the updates anyway.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Rehan Dalal [:rehan, :rdalal] from comment #10)
> I would definitely expect it to have been installed for more than 50% of all
> client ids by now. 
> 
> Perhaps Ben Hearsum has some insight?

It sounds like Mark may have solved the mystery in comment #11? Balrog certainly seems configured correctly as far as I can tell.

Comment 13

[Ryan Harter [:harter]]

> I think we should ignore any client_id where the version is less than 54.0.

Mark, can you expand on this? Why are we not interested in these clients?

> That's what we're really interested in at the moment. From your second link,
> it is looking like around 80% of those users have the add-on installed. FF
> 55 has it being shipped by default so is near 100% (0.995277) which feels
> about right.

We still only see an uptake of 56% for all clients with app_version >= 54 and with a NULL distribution_id. It looks like ~100% of *pings* from 55+ have the addon installed, however only ~66% of *clients* have the addon installed. I'd expect the client-level uptake to be higher. 

To me, it looks like there's a group of low-activity users that haven't gotten the addon yet. It looks like ~92% of all searches are from clients that have the addon installed, which is reassuring. The differences in ping/client_id weighted uptake seem to support this as well. 

I'm still interested in understanding uptake. Could it be that this uptake delay isn't unusual? Rehan, do you have an example addon previously shipped using a similar method? I can take a look at telemetry data for that period if you share an addon id and when it was shipped.

Comment 14

[Rehan Dalal (Inactive)]

The Shield System Addon (shield-recipe-client) was shipped to 53.* on May 1st.

Comment 15

[Ryan Harter [:harter]]

> We still only see an uptake of 56% for all clients with app_version >= 54
> and with a NULL distribution_id. 

This is incorrect, due to an error on my part when aggregating. Thanks, Dave, for pointing this out. We actually see 85% of clients have the addon if they are on 54+ with no distribution_id. That makes a lot more sense. Thanks all!

Comment 16

[Dave Zeber [:dzeber]]

For reference, here is a notebook validating the v54 add-on uptake across population segments: https://metrics.mozilla.com/protected/dzeber/tmp/follow-on-search-54.html

(This is an updated version of the report I prepared for the v53 add-on.)

The main conclusions are that the system add-on has been deployed to v54+ only, and uptake is consistent with the v53 add-on (no other issues stand out).