Closed Bug 1844357 Opened 2 years ago Closed 2 years ago

Preliminary ad hoc System Validation of Contextual Services reinstrumentation

Categories

(Toolkit :: Telemetry, task, P1)

Product:
Toolkit
Component:
Telemetry
Type:
task

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: chutten|PTO, Assigned: chutten|PTO)

References

(Blocks 1 open bug)

Details

The reinstrumentation of Contextual Services has reached Beta so we can start taking a casual ad hoc look at the data that's being received and conduct iterative system validation.

This should include (but is of course not limited to):

  • Ensuring the descriptions are as full and complete and correct as they can be
  • Ensuring the data is of comparable shape and content to its predecessors in the contextual_services.* dataset namespace
  • Some volumetric comparisons (number of "quick-suggest" and "top-sites" pings, distinct numbers of metrics ending in _id (block_id, context_id, request_id, distributions of pings of various ping_types),
  • Identifier coverage of metrics ending in _id (e.g., how many context_id do we see in Glean-sent but not PingCentre-sent data?)

A reminder that the datasets are restricted: access is limited to members of a specific group so even if you have moz SSO, don't be surprised if some of the query links don't work for you. We might also choose to be cagey about what specifics we see during our analyses as we summarize it for report here in the bug. If you think you need access, don't be afraid to ask and we'll point you in the right direction.

Quick Suggest

Volumetric agreement of ping volumes and context_id volumes across the ping types is very close:

Identifier coverage was only tested against quicksuggest-impression for simplicity's sake, and because it's the only one (so far) with enough volume to have a chance to say anything about it.

  • context_id - Glean hears from 1.21% that PingCentre doesn't. PingCentre hears from 0.21% that Glean does not.
  • block_id - There are too few to really say much about, but there's nothing alarming in this preliminary look.
  • request_id - There are too few to really say much about, but there's nothing alarming in this preliminary look.

Overall, there are just so few of these events in either system to say much about. Hopefully after we hit release on Aug 1 we'll get a more authoritative answer (just rerun the queries, if you'd like). But for now: nothing worrying to my eyes. Glean's living up to its reputation of quick and complete data reporting.

An initial investigation can be found here: https://colab.research.google.com/drive/1Tkrzb4-ySKwBYMjUQ6lIk1TeD0PTv51I

This is restricted access in the same fashion as the prior queries.

Because of the comparative rarity of events, a sample size of 25,000 was chosen from a single day (July 19th, 2023). This covers both the TopSites and QuickSuggest event pings.

Looking at field values for Quick Suggest, we do not notice anything out of the ordinary.

We initially see Quick Suggest usage is somewhat low in this population, however, our indicator metrics of this quick_suggest_improve_suggest_experience and quick_suggest_is_clicked show the values we would expect to see. Moving on to quick_suggest_position, we see the expected mix of values we would hope for. Without considering the distribution, we note no exceptional values, such as negatives, saturation values (e.g. I32_MAX) or string values. string.quick_suggest_ping_type looks well formed too. We see only clicks and impressions from this field, with nothing unexpected. Similarly, source sees no surprise values.

TopSites similarly does not contain any surprises, and the same comments are applicable.

The only metric that gives pause initially is Quick Suggest's quick_suggest_iab_category. However, further investigation finds that this field's unexpected "single value" is a matter of the rarity of other values. In looking at the larger set of data for the day (which exceeds the in-memory capability of the collab notebook), we confirm that values adhere to their expected possibilities.

In both cases, the string values for the advertiser fields are well formed, and align with expectations in terms of what we would expect to see from the population.

Top Sites

Volumetric agreement of ping volumes and context_id volumes across the ping types is very close:

  • topsites-impression - oddly, the PingCentre-sent context_id volumes for impressions are very slightly higher than Glean-sent. We'll look into this more in identifier coverage.
  • topsites-click

We checked context_id and tile_id identifier coverage on both ping types.

I'm confused by PingCentre being able to deliver more impressions from more contexts (but fewer pings). A hypothesis we wouldn't be able to test in the data is that this is coming from ultrashort sessions (ie, the same thing driving bug 1837230). If so, the landing of the new Glean release followed by the fix in bug 1837230 ought to pull it more in line. This is something that'll need to be resolved before we can sign off, to my mind.

Also, I noticed that the quick_suggest.reporting_url is included in the "top-sites" ping and not the "quick-suggest" ping. I'll file a bug to get that taken care of pronto.

Depends on: 1845400

ni? :jsilverman to review these analyses (and coincidentally check that he has appropriate access to view these analyses) for correctness/completeness.

Severity: -- → N/A
Flags: needinfo?(jsilverman)

Release-channel Update:

Now that Fx116 has been on release for a little while, we have more data. I've rerun the queries and found:

  • quicksuggest-block is still fairly infrequent. But of what blocks there are, there's excellent agreement between Glean and PingCentre
  • I apparently overwrote the quicksuggest-impression context_id overlap query when I redid the analyses for "topsites" pings, so here's a new one that has 0.48% contexts reported in Glean not in PingCentre (net 0.41%).
  • All of the rest of quicksuggest reinforces what we previously saw: Good agreement, and what disagreement there was is due to an increase in Glean-sent data (the volume of which corresponds to the known data loss). IOW, all good.
  • As for topsites, tile_id remains too rare to say anything about, even in release. And we still receive more impressions pings from Glean but from more contexts via PingCentre. Release didn't magically make the small different go away or reverse. (topsites-click remains like all of quicksuggest-*: more pings from more contexts via Glean than via PingCentre.)

This one odd situation needs some investigation before we'll be able to be truly happy with this, but perhaps it can be taken to a follow-up?

  • :chutten walked me through his aforementioned analyses and the logic appears sound and thus I agree with the results and interpretation discussed above.
  • In short, all looks good except 1) topsites tile_id is too rare to say much about, and 2) topsites impressions is somewhat strange in that we're receiving more impressions pings from Glean but more contexts via PingCentre. The latter topsites impressions issue should be spun out into a separate bug IMHO.
  • I also reviewed :perry 's analysis and his checks of expected data types, data ranges, and sample values seems reasonable, with the caveat that I don't have a huge amount of detailed context around the expected values/ranges/etc. for those fields.
Flags: needinfo?(jsilverman)

ni?perry to have one more look over this. I'm happy, Jeff's happy, and if Perry's happy we can mark this RESOLVED > FIXED.

I'll file that follow-up momentarily.

Flags: needinfo?(pmcmanis)

I am also satisfied that this can move to FIXED given the existence of 1847950

Flags: needinfo?(pmcmanis)
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.