Closed Bug 2065466 Opened 2 months ago Closed 10 days ago

follows-collection-enabled=false pings can end up being invalid pings as per the schema

Categories

(Data Platform and Tools :: Glean: SDK, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: janerik, Assigned: janerik)

References

(Blocks 2 open bugs)

Details

Attachments

(2 external files)

There's a small bug that makes the Glean SDK assemble a ping that won't pass the ingestion schema check.

Full details:
There's a bunch of fields in client_info that are required. One of them is first_run_date.
We do set that value on the very first run and keep ahold of it. Even if upload is disabled, that value is not cleared, because it still goes into follows_collection_enabled=false pings for example.
All good so far.

Now we can have follows_collection_enabled=false pings that follow the baseline schedule (ping_schedule). They get triggered on every submission of the baseline ping too.
Now if on the very first run Glean is initialized with upload=false. we never generate a first_run_date.
Once a baseline ping would be triggered later (handle_client_active()) the baseline ping is NOT sent (because upload=false), BUT the ride-along ping is submitted.
That ping tries to fill the client_info part, which tries to put a first_run_date there, which doesn't exist, but the ping is assembled and sent and will not pass ingestion checks.

all in all: it's a bug, but a bug that's SO unlikely to be hit by any reasonable number of clients that we don't even need to care much about it
We might still fix it.

Whiteboard: [telemetry:glean-rs:m?]

On second thought (and after bug 1952363) ride-along is not a necessity here. Any custom ping with follows_collection_enabled=false might run into this.

See Also: → 1952363
Summary: ride-along follows-collection-enabled=false pings can end up being invalid pings as per the schema → follows-collection-enabled=false pings can end up being invalid pings as per the schema

There's now two ways to address this:

  1. We relax the schema server-side and allow first_run_date to be missing
    a. Advantage here is that we will capture all old clients running into this without updates, disadvantage is that other parts might rely on first_run_date being reasonably valid and now it might be missing more often.
  2. Client-side we set a first_run_date if it's missing
    a. We either come up with one based on whenever this missing one is detected, which is probably wrong in 99%[^1] of the cases (and definitely now wrong for all those clients that currently end up with schema errors)
    b. We set a non-sensical value like 1970-01-01, at least that way we know it can't be correct.

[^1] A number I made up.

Blocks: 1952363
See Also: 1952363 →
Assignee: nobody → jrediger
Status: NEW → ASSIGNED
Priority: P3 → P1

I'd like input on this, from the Glean team, but also DE.

Flags: needinfo?(tlong)
Flags: needinfo?(srose)
Flags: needinfo?(chutten)

Okay, here's my recommendation:

  1. Generate a first_run_date at init, regardless of upload state.
  • This fixes all new clients, with a valid date that would survive upload toggling, so it documents the first run of the application and not the "first run with upload toggled".
  • is_first_run is now accurate for these clients in all cases.
  • Clients that start opted out and later opt into collection will report the actual first run date
  • This doesn't help older clients, they will need some other option too (see Jan-Erik's suggestion to relax the schema, maybe we can do this only for the affected pings?)
  • For "existing" clients, we could fill a sentinel value when building the ping (instead of persisting it) and record an error to keep the schema correct without reporting a false/misleading first_run_date generated when upload is enabled.
Flags: needinfo?(tlong)

(In reply to Travis Long [:travis_] from comment #5)

  1. Generate a first_run_date at init, regardless of upload state.
  • This fixes all new clients, with a valid date that would survive upload toggling, so it documents the first run of the application and not the "first run with upload toggled".

That's easier said than done. We have some logic that determines if it's the first run based on whether that date is set.
This will take some time to untangle and might have implications for when a client toggles on upload later.
That's also to say: If a client starts with telemetry disabled and later enables it, we know suddenly know the actual first run date, which is way before telemetry was ever enabled. That seems like ... not good.

  • This doesn't help older clients, they will need some other option too (see Jan-Erik's suggestion to relax the schema, maybe we can do this only for the affected pings?)

I don't think we have good options to enable it for just a single ping. The schema is selected early in the ingestion pipeline based on the path, special-casing a single ping there is not something we have done before.

I think we should always set first_run_date to a value the first time we initialize the Glean SDK, update its description to be precise about that, and mark it optional in the schema.

Rationale:

  • There are (to my knowledge) no data stewardship implications to setting the value and reporting it from follows_collection_enabled: false pings (or not at all) until opt-in.
  • It it the easiest behaviour to understand and test.
  • Marking it optional in the schema means we get to accept pings from SDK versions affected by this behaviour while only slightly opening the door to malformed pings sneaking their way past schema checks.
  • I don't believe this will complicate our first-run handling, since we've already developed some depth here thanks to the client_id.

All this being said, I wouldn't be fussed if we decided to do any of the other ideas we discussed in the team chat or in this bug. About the only option I don't like is to let this continue to linger because we are giving our instrumentors no way to work around it to ensure the quality of their data.

Flags: needinfo?(chutten)

(In reply to Chris H-C :chutten from comment #7)

  • I don't believe this will complicate our first-run handling, since we've already developed some depth here thanks to the client_id.

It does in the sense that Glean determines is_first_run based on whether there's a first_run_date in the database or not (that said: we can figure this out and do it right)

I support :chutten's proposed approach, since first_run_date appears to just be a date + timezone offset, which isn't unique enough to be a re-identification risk when users disable telemetry and then later re-enable it.

Flags: needinfo?(srose)

I have updated the PR to set first_run_date unconditionally. Tests pass, but "first run" detection might be behaving slightly different now, we don't yet have a test for that behavior.

(In reply to Chris H-C :chutten from comment #7)

I think we should always set first_run_date to a value the first time we initialize the Glean SDK, update its description to be precise about that, and mark it optional in the schema.

[snip]

  • Marking it optional in the schema means we get to accept pings from SDK versions affected by this behaviour while only slightly opening the door to malformed pings sneaking their way past schema checks.

In doing the above I found that os was also not set, so unless we make that field also optional those old clients will still send invalid pings and get rejected.

Feedback from lifecycle marketing was: the ping we're doing this for (fx-accounts-client-info) is not yet in full use.
So fixing it for clients going forward is good and we keep the schema unchanged for now.

Status: ASSIGNED → RESOLVED
Closed: 10 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: