Closed Bug 1765280 Opened 4 years ago Closed 4 years ago

Brief - Evaluate FOG/Glean's client id capabilities

Categories

(Toolkit :: Telemetry, task, P1)

Product:
Toolkit
Component:
Telemetry
Type:
task

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: Dexter, Assigned: chutten)

Details

We should improve on this doc or write a separate brief to describe if and how FOG/Glean provides consumers with a better accounting of client ids compared to legacy telemetry.

Hey Chris,

sorry I forgot to file the bug before going on PTO. Here's the thing we talked about last time we met.

Flags: needinfo?(chutten)

So I've been looking into this, and I'm still wrapping my brain around whether this is actually a thing that is possible to work out without going all the way deep on each of the specific clients that have too many (or too few) Legacy client_ids per Glean client_id, or vice versa.

To give an example, is it a sign that Glean is better or worse to find that a single Glean client_id has multiple Legacy cilent_ids associated with it?

To resolve this I'd need some way to tell

  • If a single client_id (Glean or Legacy) has been cloned
  • If a group of client_ids (Glean or Legacy) are from the same profile or not (say, across multiple deletion requests that haven't yet resolved)

And then from there I could see if Glean's count of clients is more or less accurate than Telemetry's for a given assumption of how to detect clones and sequential-multiple-client_id profiles.

This... this might take a while. But! For now, for a given day in April, the client count as reported by "metrics" pings is quite a bit lower than the client count reported by "main" pings. Whether that's to do with the fact that we send 3x as many "main" pings or whether that's Glean being more stable over the span of a day than Legacy (or anything else) remains to be seen.

Assignee: nobody → chutten
Severity: -- → N/A
Status: NEW → ASSIGNED
Flags: needinfo?(chutten)
Priority: -- → P1

(In reply to Chris H-C :chutten from comment #2)

This... this might take a while. But! For now, for a given day in April, the client count as reported by "metrics" pings is quite a bit lower than the client count reported by "main" pings. Whether that's to do with the fact that we send 3x as many "main" pings or whether that's Glean being more stable over the span of a day than Legacy (or anything else) remains to be seen.

Interesting about the main ping. Does this hold across ping "reasons"? I suspect the count mismatch could be due to the 'shutdown' pings.

There'd be nothing to rule that in or out as an explanation, unfortunately, as on most platforms "shutdown"-reason "main" pings predominate. I'd have to link "main" pings to "metrics" pings, but that's outside the scope of the linking proposal and this investigation, which is settling for linking clients, not pings. Plus, the scheduling differences between "main" and "metrics" might mean that it wouldn't even help : |

To determine if a given client_id is a clone (and thus impacting our ability to accurately count clients) I used the following criteria:

  • If a ping of a given doctype is received from a given client_id with the same sequence number, it is evidence that the client_id may have been cloned.

Rationale:

  • Within a doctype, sequence numbers only ever increase and do not overflow or saturate until about 4B. At the current rate of send of "main" pings, this will take nearly four thousand years.
  • Thus, within a doctype, a given {client_id, sequence number} pair should be unique if the data collection system is functioning correctly

Details:

  • We're only looking at release-channel Firefox Desktop versions 96+ because that's a fairly reason set of users. This lets us ignore some of the obvious weirdness caused by old clients as well as levels the playing field by ensuring we've been sending data using both Glean and Legacy Firefox Telemetry.
  • We're looking at the past 28-(ish-)day period from March 22 to April 19 (inclusive), just to pick a modern interval.
  • "sequence number" means for Glean ping_info.seq and for Legacy Telemetry "main" pings payload.info.profile_subsession_counter

Results:

  • (query) A 1% sample of clients' Legacy Firefox Telemetry "main" pings sent from Firefox Desktop 96+ in a 28-day period revealed 73,872 "clone candidate" client_ids sending an excess 612,140 pings.
    • Un-sampled, that's about 7M "clone candidate" clients sending 61M excess pings.
  • (query) Clients' Glean "metrics" pings sent from Firefox Desktop 96+ in the 28-day period revealed 462,849 "clone candidate" clients sending 9,127,405 excess pings.
  • (query) Clients' Glean "baseline" pings sent from Firefox Desktop 96+ in the 28-day period revealed 835,548 "clone candidate" clients sending 67,252,515 excess pings.

Conclusions:

  • Glean and Firefox Telemetry both send a number of duplicate pings from "clone candidate" clients.
  • Firefox Telemetry reports an order of magnitude more "clone candidate" clients than reported by Glean.
    • This effect is not due to Firefox Telemetry "main" pings reporting more clients overall, as Glean "baseline" pings report even more clients than "main" pings.
  • Given the total reported clients over that period these "clone candidate" clients, if actually clones, amount to a proportional client inflation of
    • 3.95% for Telemetry via "main" pings
    • 0.24% for Glean via "metrics" pings
    • 0.41% for Glean via "baseline" pings
  • If not clones, this points to a failure in the data collection system when it comes to sequence numbers.
    • Further analysis might be able to account for this failure by examining other aspects of the "clone candidates" to see if more certainty can be gleaned. For instance, if there are multiple pings with overlapping reporting intervals (according to the client's clock).

Having updated the linked doc with an additional section referencing these findings, I'll call this done.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.