1851034 - PingCentre sending more of a certain message_id+event than Glean

Status: RESOLVED WONTFIX

(Firefox :: Messaging System, task, P1)

Description

[Jeffrey Silverman [:jsilverman]]

Attachment: Untitled.png

As part of the Messaging System Onboarding Data Validation, I found that PingCentre is sending more events with message_id=MR_WELCOME_DEFAULT and event=SESSION_END than Glean is. This combination of message_id and event is one of the 10 most often seen combos in the Glean data from July 2023 (and Fx 115+), so that's why it's concerning that PC is sending more of them than Glean. This also holds true for Aug 2023 (Fx 115+) and Aug 2023 (Fx 116+).

Comment 1

[Alessio Placitelli [:Dexter]]

Hey Perry and Chris, would you please take a look?

Comment 2

[Chris H-C :chutten]

I notice some interesting features here:

• The plot appears to be an uptake chart (or related to version uptake in a population) as you see the flat left (no/few clients on that version) and the rough increase over the weeks.
• The difference between the two systems disappears on weekends and by week four

I've taken a look at Fx115+ across the month of August (now completed) and found that there are day-by-day differences in ping counts with message_id of MR_WELCOME_DEFAULT and event of SESSION_END, but they're as often leaning to one system's side as the other. Wild Speculation Time: mayyyyyybe there's a small population of pings coming in on PC not Glean and this manifests more clearly in an upgrading population than a stable population where Glean's retries of previous versions' undeliverable pings can make up the difference? (It could also be that I'm seeing bug 1847950 everywhere)

Jeff, is this difference one of sufficient concern that should fail validation? Or is this a "document it, file a bug for looking into it, and deal with it" level of issue?

Comment 3

[Jeffrey Silverman [:jsilverman]]

My official opinion is that this should not lead to a validation fail and that the difference is the latter, i.e., "document it, file a bug, make sure people are aware of it and that it's different behavior than other ping counts (where Glean is consistently a few percent more than PC), but mostly deal with it".

Comment 4

[Chris H-C :chutten]

Attachment: pr 206: bug 1851034 - Annotate odd volume in certain circumstances

Comment 5

[Chris H-C :chutten]

Alrighty. This bug's a good one to keep around, then, and let's See Also it to bug 1847950 in case my suspicion ends up panning out.

I've put a Gleannotation up for review that ought to help with the "make sure people are aware of it" angle for anyone who doesn't get to read your Data Validation report.

Comment 6

[Chris H-C :chutten]

chutten merged PR #206: "bug 1851034 - Annotate odd volume in certain circumstances" in 1e32692.

Comment 7

[Chris H-C :chutten]

We're pretty much done thinking about PingCentre.