1474295 - Validate "event" ping data for the Beta channel

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Chris H-C :chutten]

+++ This bug was initially created as a clone of Bug #1463410 +++

Similar to Alessio's analyses in bug 1351396 and bug 1351402 we will need to do a broad-spectrum validity analysis on the "event" ping on Beta after it lands.

Things to check (a non-exhaustive list):
Ping counts
Ping frequency
Ping size
session id link
subsession id link
Do the "main" ping events match the "event" ping events (allowing for them being sent at different times and in different subsessions)
Does the meta-telemetry (histogram TELEMETRY_EVENT_PING_SENT) send and correspond correctly to subsessions containing events. Does its counts match the counts of received "event" pings?

(( Note that this analysis is in addition to the preliminary realtime plugin that ensures we're sending pings at all: https://docs.telemetry.mozilla.org/cookbooks/new_ping.html ))

Work can be based on the Nightly notebook I wrote here: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/19038/command/19453

Comment 1

[Chris H-C :chutten]

Current schedule is that "event" ping will be uplifted for b7's build tomorrow.

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 62 so we will remember to follow up and check on these uplifts in 62 beta.

Comment 3

[Alessio Placitelli [:Dexter]]

I re-ran a slightly modified version of the notebook that was executed for the Nightly channel. Here are the key takeaways:

- All reasons were valid and no pings had no reason (an handful of pings reached the event cap and had the "max" reason).
- 95th percentile of event pings per day is 2, 4% of clients are sending more than 2 event pings per day.
- ~0.14% of pings were created within 10min of each other on the same session (similarly to Nightly).
- Quite a few pings report lost events: 'shutdown' 52388, 'max', 4710.
- The 99th percentile of events in the event ping is 18. The ratio of pings containing more than 18 events is ~0.008.
- Pings that have events spanning longer than 65min are 0.003%.
- ~99.8% of main pings linked by subsession to event pings have TELEMETRY_EVENT_PING_SENT set.
- When linking by sessionId, this figure drops to ~73.32%, which is a bit better than Nightly.

I think the event ping is behaving consistently on Beta. It's a bit concerning that we're loosing events (we weren't on Nightly), but maybe that's due to the fact that we're running SHIELD studies that use events on Beta more than Nightly?

r?Sunah on the results/notebook for sanity since you know more about events :)


[1] - https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/21288/command/21648

Comment 4

[Alessio Placitelli [:Dexter]]

(In reply to Alessio Placitelli [:Dexter] from comment #3)
> - Quite a few pings report lost events: 'shutdown' 52388, 'max', 4710.

Which corresponds to the following ratios, by reason:

"shutdown": 0.055
"max": 0.005
"periodic": 0.000

Comment 5

[bugzilla]

Notebook looks good (I have a few quibbles re: variable naming if we were using this as a template for posterity but since we're not, it's fine :)) Re: ping loss, we should try re-running the nightly notebook for the same time period first, maybe the disparity will disappear.

Comment 6

[Alessio Placitelli [:Dexter]]

Moving to :gfritzsche due to my PTO.

Comment 7

[Alessio Placitelli [:Dexter]]

After discussing this with Sunah, this seems to look good on Beta. I've filed bug 1477292 for investigating the event loss, which is not a blocker for this on beta.