1982711 - Determine why the Glean SDK thinks it has a new db when there's tens or hundreds of kb of data in db.safe.bin

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, task, P1)

Description

[Chris H-C :chutten]

In bug 1979075 we added instrumentation to Firefox Desktop that shows client_id regeneration despite there being what appear to be perfectly-normal-looking db.safe.bin files in the db/ dir.

How is that possible?

I think this'll be another instrumentation-adding bug, perhaps bug 1667815 related, because what makes it so that the db is "empty" despite it being housed in a file of many tens or hundreds of thousands of bytes of size? Perhaps the file is deleted between init being called and rkv_new? Maybe we need to instrument fn database_size? Maybe we need to check it again after rkv_open? Maybe check in between opening each of the single stores? Maybe iterate the data (or just check for any one item) real quick after opening to see if it's hundreds of thousands of bytes of emptiness?

Comment 1

[Travis Long [:travis_]]

Attachment: [mozilla/glean] Bug 1982711 - Glean Health Ping with directory info before and after init (#3221)

Comment 2

[Travis Long [:travis_]]

Attachment: data-collection-request

Request for data-collection review for new data-directory information metric, shipped via a new ping "health" that will also include existing (already reviewed and approved) telemetry health metrics already defined in the Glean SDK.

Comment 3

[Charlie Humphreys [:charlie] [:jeddai]]

Comment on attachment 9508860 [details]
data-collection-request

Data Review Form

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

This collection is documented in the Glean Dictionary at https://dictionary.telemetry.mozilla.org/

2. Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

Each specific application instrumenting Glean provides a data-collection preference to opt-out of this and all Mozilla data collection.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, [e-mail] and [e-mail]

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, technical information

5. Is the data collection request for default-on or default-off?

default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

data-review+

Comment 4

[Travis Long [:travis_]]

Untaking this as I think my current contribution to this is done, at least until this lands and we have more data to look into.

Comment 5

[Jan-Erik Rediger [:janerik]]

travis79 merged PR [mozilla/glean]: Bug 1982711 - Glean Health Ping with directory info before and after init (#3221) in d7d9d71.

For consistency: the patch landed (and is released in v65.1.0)

Comment 6

[Chris H-C :chutten]

Looking at a pair of "health" pings from a client who regenerates its client_id during that init, glean.database.size agrees with the dir info on the size of db.safe.bin meaning that in the line before rkv_new, there is a db.safe.bin file of hundreds of kbs of size that was created and last modified around 11:11 EDT (the same time the db/ directory itself was created). This is confirmation of what we determined from the FOG "temp-fog-initial-state" ping analysis.

The init is happening around 12:58 EDT (according to ping_info, and pending ping file creation timestamps from the "post_init"-reason "health" ping).

The "post-init" ping has no glean.database.size in it (as it shouldn't. We don't record it a second time between the submission of the two pings and besides, it only reports the initial value) but its dir info shows a db.safe.bin file of only tens of kbs of size that was created and last modified around 12:58 EDT. Though it is possible that ping submission could decrease the size of the db somewhat (the "pre_init"-reason "health" ping shows three pings were submitted (one each of "baseline", "events", and "metrics") by glean.validation.pings_submitted and three files in the pending_pings/ dir; the "post_init"-reason "health" ping does not have any value for glean.validation.pings_submitted (as it shouldn't, as there have been no built-in pings submitted in the milliseconds since the "pre_init"-reason "health" ping was submitted) but shows 12 files in the pending_pings/ dir (3 are the same as from the "pre_init"-reason "health" ping), and the sizes of these nine new pending pings are large enough that they could account for the difference in db size), ping submission would never recreate the database file, updating its creation timestamp to 12:58 EDT.

(( Other directory contents change (like the size of events/events which becomes 0 because an at-startup submission is triggered during init). Other directory contents do not change (like the size of events/bounce-tracking-proection).

There are no errors of any kind recorded in either of these pings.

Unfortunately this merely confirms previous suppositions and rules some things out (I/O errors, mistakes in recording rkv errors, the regeneration happening post init, etc.). It doesn't firmly point us at a specific cause. But it does narrow things down and supply a scaffold of pings we can put more instrumentation on.

Follow-up actions:

• Trace the code: what could recreate the file? Instrument, instrument, instrument.
• Add an count/identifier per init/session to the "health" ping to make it easier to pair up these pings.
• Add the legacy.telemetry.client_id to these pings to make it easier to identify client_info.client_id regeneration

I'll file some follow-up bugs.