Validate "event" ping data for the Beta channel

RESOLVED FIXED

Status

()

defect
P1
normal
RESOLVED FIXED
10 months ago
9 months ago

People

(Reporter: chutten, Assigned: gfritzsche)

Tracking

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox62+ fixed, firefox63 fixed)

Details

(Reporter)

Description

10 months ago
+++ This bug was initially created as a clone of Bug #1463410 +++

Similar to Alessio's analyses in bug 1351396 and bug 1351402 we will need to do a broad-spectrum validity analysis on the "event" ping on Beta after it lands.

Things to check (a non-exhaustive list):
Ping counts
Ping frequency
Ping size
session id link
subsession id link
Do the "main" ping events match the "event" ping events (allowing for them being sent at different times and in different subsessions)
Does the meta-telemetry (histogram TELEMETRY_EVENT_PING_SENT) send and correspond correctly to subsessions containing events. Does its counts match the counts of received "event" pings?

(( Note that this analysis is in addition to the preliminary realtime plugin that ensures we're sending pings at all: https://docs.telemetry.mozilla.org/cookbooks/new_ping.html ))

Work can be based on the Nightly notebook I wrote here: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/19038/command/19453
(Reporter)

Updated

10 months ago
Assignee: nobody → alessio.placitelli
Status: NEW → ASSIGNED
(Reporter)

Updated

10 months ago
No longer depends on: 1460595
(Reporter)

Updated

10 months ago
Depends on: 1460595
(Reporter)

Comment 1

10 months ago
Current schedule is that "event" ping will be uplifted for b7's build tomorrow.
Tracking for 62 so we will remember to follow up and check on these uplifts in 62 beta.
I re-ran a slightly modified version of the notebook that was executed for the Nightly channel. Here are the key takeaways:

- All reasons were valid and no pings had no reason (an handful of pings reached the event cap and had the "max" reason).
- 95th percentile of event pings per day is 2, 4% of clients are sending more than 2 event pings per day.
- ~0.14% of pings were created within 10min of each other on the same session (similarly to Nightly).
- Quite a few pings report lost events: 'shutdown' 52388, 'max', 4710.
- The 99th percentile of events in the event ping is 18. The ratio of pings containing more than 18 events is ~0.008.
- Pings that have events spanning longer than 65min are 0.003%.
- ~99.8% of main pings linked by subsession to event pings have TELEMETRY_EVENT_PING_SENT set.
- When linking by sessionId, this figure drops to ~73.32%, which is a bit better than Nightly.

I think the event ping is behaving consistently on Beta. It's a bit concerning that we're loosing events (we weren't on Nightly), but maybe that's due to the fact that we're running SHIELD studies that use events on Beta more than Nightly?

r?Sunah on the results/notebook for sanity since you know more about events :)


[1] - https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/21288/command/21648
Flags: needinfo?(ssuh)
(In reply to Alessio Placitelli [:Dexter] from comment #3)
> - Quite a few pings report lost events: 'shutdown' 52388, 'max', 4710.

Which corresponds to the following ratios, by reason:

"shutdown": 0.055
"max": 0.005
"periodic": 0.000
Notebook looks good (I have a few quibbles re: variable naming if we were using this as a template for posterity but since we're not, it's fine :)) Re: ping loss, we should try re-running the nightly notebook for the same time period first, maybe the disparity will disappear.
Flags: needinfo?(ssuh)
Moving to :gfritzsche due to my PTO.
Assignee: alessio.placitelli → gfritzsche
Depends on: 1477292
After discussing this with Sunah, this seems to look good on Beta. I've filed bug 1477292 for investigating the event loss, which is not a blocker for this on beta.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.