1627286 - Always send a baseline ping at foreground

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, enhancement, P1)

Description

[Michael Droettboom [:mdroettboom]]

It would be helpful for comparative analysis of Fennec and Fenix activity data to always send the baseline ping at startup (in addition to the baseline ping sending schedule that currently exists).

Comment 1

[Frank Bertsch [:frank]]

This ping should have a different reason, e.g. session-start.

Comment 2

[Michael Droettboom [:mdroettboom]]

Just to make sure we're all on the same page and thinking this through further --

Is this new baseline ping schedule to be on application startup or session start (on foreground)?

I think it needs to be on foreground to really address the underlying issue.

If sent only at startup, that doesn't help us if we have five short sessions spread out over a few days while the app remains running, the last of which drops the on-background baseline ping. Additionally, when we start up again, we'd get a dirty startup ping anyway, so there's no additional benefit.

Comment 3

[Frank Bertsch [:frank]]

Is this new baseline ping schedule to be on application startup or session start (on foreground)?

Agreed, this is session-start, including foreground.

Comment 4

[Chris H-C :chutten|PTO (back March 23)]

(Lean Data Hat On) Can this be made to expire after the migration?

Comment 5

[Alessio Placitelli [:Dexter]]

I think I had a clear picture of th(In reply to Michael Droettboom [:mdroettboom] from comment #2)

Just to make sure we're all on the same page and thinking this through further --

Is this new baseline ping schedule to be on application startup or session start (on foreground)?

Oh, good thing you mentioned, because I thought we agreed on application start!

I think it needs to be on foreground to really address the underlying issue.

Do you mean sending the ping every time we go to foreground in addition to sending the ping every time we go to background?

If sent only at startup, that doesn't help us if we have five short sessions spread out over a few days while the app remains running, the last of which drops the on-background baseline ping. Additionally, when we start up again, we'd get a dirty startup ping anyway, so there's no additional benefit.

This would give us a signal for the first start, which is currently lacking unless we crashed before. In the 5 short sessions scenario: the first time it starts, it would send a baseline (reason: startup). When going to background, it would send baseline (reason: background). When going back to foreground again nothing, then if going to background the usual baseline (reason: background). If, for some reason, the process is killed, then next time we start we get baseline (reason: dirty_startup or reason startup).

I thought the problem was really 2nd day retention here, which was hard to compute due to the missing initial signal?

But, if this choice of sending a ping on background is really a concern, we can just always send it on "foreground" and trade the ping latency with some inconsistent reporting of the "duration" field (which would always refer to the previous ping..).

Comment 6

[Marissa Gorlick]

+1 on foreground. On application start will not totally solve the issue. The problem is daily activation. Imagine X could be any positive day in someone's lifecycle.

https://docs.google.com/document/d/13t0m0JAUc2FthMWhOAaw3e0bcDtIy4tMYLbrLUXOKWE/edit#heading=h.bdxpnitcxq0e

Comment 7

[Michael Droettboom [:mdroettboom]]

Ok. Sounds like moving "background" to "foreground" with the session length from the last session is the thing.

Comment 8

[Frank Bertsch [:frank]]

(In reply to Michael Droettboom [:mdroettboom] from comment #7)

Ok. Sounds like moving "background" to "foreground" with the session length from the last session is the thing.

Hey Mike, I thought we were going to keep everything else as-is, and simply add another reason? So send at both foreground and background.

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/glean/pull/806

Comment 10

[Michael Droettboom [:mdroettboom]]

Yeah -- I think I may have misunderstood from :dexter's comment. It's easy enough to add foreground in addition to existing behavior, especially if there's no bandwidth concerns.

I'll leave the PR up as-is for now, as I'm almost EOD, but happy to refactor it to just add the new ping reason.

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/glean/pull/807

Comment 12

[Michael Droettboom [:mdroettboom]]

Ok -- this discussion got a bit circuitous above, not the least of which due to my own confusion. I will try to summarize the plan of action below (also based on some 1:1 conversation with :dexter):

1. We are going to add a baseline ping at every foreground, with it's own reason "foreground". This is in addition to any baseline pings that Glean already sends.

2. We will come back and validate that this had the desired effect. Since the new pings have their own reason code, it should be easy to filter just for them and see how they work on their own or in conjunction with the other baseline pings.

3. While we are ok with the extra pings for now, given Fenix's smaller population, we do not plan to send all of these baseline pings forever. (We should also consider the amount of user bandwidth we are using). Instead we plan to use the new data to determine the best design going forward. Bug 1627981 has been filed to track this follow-up work.

?ni -- confirm that this is the best course of action?

Comment 13

[Marissa Gorlick]

A few notes.

1. This is not an urgent patch today as I believe we will be using the activity dates (start_date) for our short-run success metric (Fennec to Fenix transition experiment). Instead, we should consider this the long-term solution to measuring Daily Activation in a way that is persistent in Glean. We will always need to know Activation at a daily interval, particularly for the first 90 days of a user's lifecycle (and beyond!). We will need to know this on other products too. If we do not feel this is our long-term solution maybe we should keep brainstorming. IMO we should only deploy a short term solution if it is necessary for ideating on our long-term solution.

2. We need to consider downstream impact and communications for our high-level KPIs. These might change how active people appear and on which days. We need to proactively communicate to Data Engineering ( klukas, frank ) so their ETL can consider the reason code to filter out new data. The new foreground ping data should not impact our high-level metrics until the change is approved by leadership and has been communicated to the org.

Comment 14

[Jeff Klukas [:klukas] (UTC-4)]

The new foreground ping data should not impact our high-level metrics until the change is approved by leadership and has been communicated to the org

I will include reason filtering in https://github.com/mozilla/bigquery-etl/pull/887 so that what we show in GUD does not take the new startup pings into account until we've vetted it.

Comment 15

[Alessio Placitelli [:Dexter]]

(In reply to Michael Droettboom [:mdroettboom] from comment #12)

?ni -- confirm that this is the best course of action?

I feel this is the best course of action. Given Marissa's comment 13, we might also want to add a step 0: "writing about the problem and the potential solution".

Comment 16

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Foreground pings are included in aggregate tables and for activation metrics as of https://github.com/mozilla/bigquery-etl/pull/1253.