follows-collection-enabled=false pings can end up being invalid pings as per the schema
Categories
(Data Platform and Tools :: Glean: SDK, defect, P1)
Tracking
(Not tracked)
People
(Reporter: janerik, Assigned: janerik)
References
(Blocks 2 open bugs)
Details
Attachments
(2 external files)
There's a small bug that makes the Glean SDK assemble a ping that won't pass the ingestion schema check.
Full details:
There's a bunch of fields in client_info that are required. One of them is first_run_date.
We do set that value on the very first run and keep ahold of it. Even if upload is disabled, that value is not cleared, because it still goes into follows_collection_enabled=false pings for example.
All good so far.
Now we can have follows_collection_enabled=false pings that follow the baseline schedule (ping_schedule). They get triggered on every submission of the baseline ping too.
Now if on the very first run Glean is initialized with upload=false. we never generate a first_run_date.
Once a baseline ping would be triggered later (handle_client_active()) the baseline ping is NOT sent (because upload=false), BUT the ride-along ping is submitted.
That ping tries to fill the client_info part, which tries to put a first_run_date there, which doesn't exist, but the ping is assembled and sent and will not pass ingestion checks.
all in all: it's a bug, but a bug that's SO unlikely to be hit by any reasonable number of clients that we don't even need to care much about it
We might still fix it.
Comment 1•2 months ago
|
||
Updated•2 months ago
|
| Assignee | ||
Comment 2•18 days ago
|
||
On second thought (and after bug 1952363) ride-along is not a necessity here. Any custom ping with follows_collection_enabled=false might run into this.
| Assignee | ||
Updated•18 days ago
|
| Assignee | ||
Comment 3•18 days ago
|
||
There's now two ways to address this:
- We relax the schema server-side and allow
first_run_dateto be missing
a. Advantage here is that we will capture all old clients running into this without updates, disadvantage is that other parts might rely onfirst_run_datebeing reasonably valid and now it might be missing more often. - Client-side we set a
first_run_dateif it's missing
a. We either come up with one based on whenever this missing one is detected, which is probably wrong in 99%[^1] of the cases (and definitely now wrong for all those clients that currently end up with schema errors)
b. We set a non-sensical value like1970-01-01, at least that way we know it can't be correct.
[^1] A number I made up.
| Assignee | ||
Updated•18 days ago
|
| Assignee | ||
Updated•18 days ago
|
| Assignee | ||
Comment 4•18 days ago
|
||
I'd like input on this, from the Glean team, but also DE.
Comment 5•18 days ago
•
|
||
Okay, here's my recommendation:
- Generate a
first_run_dateat init, regardless of upload state.
- This fixes all new clients, with a valid date that would survive upload toggling, so it documents the first run of the application and not the "first run with upload toggled".
is_first_runis now accurate for these clients in all cases.- Clients that start opted out and later opt into collection will report the actual first run date
- This doesn't help older clients, they will need some other option too (see Jan-Erik's suggestion to relax the schema, maybe we can do this only for the affected pings?)
- For "existing" clients, we could fill a sentinel value when building the ping (instead of persisting it) and record an error to keep the schema correct without reporting a false/misleading
first_run_dategenerated when upload is enabled.
| Assignee | ||
Comment 6•18 days ago
|
||
(In reply to Travis Long [:travis_] from comment #5)
- Generate a
first_run_dateat init, regardless of upload state.
- This fixes all new clients, with a valid date that would survive upload toggling, so it documents the first run of the application and not the "first run with upload toggled".
That's easier said than done. We have some logic that determines if it's the first run based on whether that date is set.
This will take some time to untangle and might have implications for when a client toggles on upload later.
That's also to say: If a client starts with telemetry disabled and later enables it, we know suddenly know the actual first run date, which is way before telemetry was ever enabled. That seems like ... not good.
- This doesn't help older clients, they will need some other option too (see Jan-Erik's suggestion to relax the schema, maybe we can do this only for the affected pings?)
I don't think we have good options to enable it for just a single ping. The schema is selected early in the ingestion pipeline based on the path, special-casing a single ping there is not something we have done before.
Comment 7•17 days ago
|
||
I think we should always set first_run_date to a value the first time we initialize the Glean SDK, update its description to be precise about that, and mark it optional in the schema.
Rationale:
- There are (to my knowledge) no data stewardship implications to setting the value and reporting it from
follows_collection_enabled: falsepings (or not at all) until opt-in. - It it the easiest behaviour to understand and test.
- Marking it optional in the schema means we get to accept pings from SDK versions affected by this behaviour while only slightly opening the door to malformed pings sneaking their way past schema checks.
- I don't believe this will complicate our first-run handling, since we've already developed some depth here thanks to the
client_id.
All this being said, I wouldn't be fussed if we decided to do any of the other ideas we discussed in the team chat or in this bug. About the only option I don't like is to let this continue to linger because we are giving our instrumentors no way to work around it to ensure the quality of their data.
| Assignee | ||
Comment 8•17 days ago
•
|
||
(In reply to Chris H-C :chutten from comment #7)
- I don't believe this will complicate our first-run handling, since we've already developed some depth here thanks to the
client_id.
It does in the sense that Glean determines is_first_run based on whether there's a first_run_date in the database or not (that said: we can figure this out and do it right)
Comment 9•13 days ago
|
||
I support :chutten's proposed approach, since first_run_date appears to just be a date + timezone offset, which isn't unique enough to be a re-identification risk when users disable telemetry and then later re-enable it.
| Assignee | ||
Comment 10•12 days ago
|
||
I have updated the PR to set first_run_date unconditionally. Tests pass, but "first run" detection might be behaving slightly different now, we don't yet have a test for that behavior.
(In reply to Chris H-C :chutten from comment #7)
I think we should always set
first_run_dateto a value the first time we initialize the Glean SDK, update its description to be precise about that, and mark it optional in the schema.[snip]
- Marking it optional in the schema means we get to accept pings from SDK versions affected by this behaviour while only slightly opening the door to malformed pings sneaking their way past schema checks.
In doing the above I found that os was also not set, so unless we make that field also optional those old clients will still send invalid pings and get rejected.
| Assignee | ||
Comment 11•12 days ago
|
||
| Assignee | ||
Comment 12•10 days ago
|
||
Feedback from lifecycle marketing was: the ping we're doing this for (fx-accounts-client-info) is not yet in full use.
So fixing it for clients going forward is good and we keep the schema unchanged for now.
| Assignee | ||
Comment 13•10 days ago
|
||
badboy merged PR [mozilla/glean]: Bug 2065466 - Unconditionally set first_run_date to ensure pings are schema-conform (#3608) in 053c513.
| Assignee | ||
Updated•10 days ago
|
Description
•