1572744 - Add schema for Sync pings

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P2)

Description

[Arkadiusz Komarzewski [:akomar]]

Started in https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/268/files

Comment 1

[Mark Reid [:mreid]]

I've been working on this as I get time, and will continue to do so. If this becomes urgent or starts to block other work, please let me know.

Comment 2

[Arkadiusz Komarzewski [:akomar]]

We may also need a schema for v5 of Sync ping.

Sync ETL that's currently running uses both v4 and v5 of the ping.

:loines, :linacambridge - I see only v4 of the ping documented, but clearly some clients (dev?) are sending v5 and it's being used for analyses.
Are schemas the same across these versions?

Comment 3

[Lina Butler (ex-Mozilla)]

I think so—per bug 1410145, v5 was used in older versions of Android (and is unchanged from v4)...but that bug was fixed in 2017! https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/422 tracks removing the schema as well.

Comment 4

[Mark Reid [:mreid]]

:lina and :loines, what do you think is the right thing to do with this v5 data? We have the following options:

1. Keep the v5 data as-is. Data will continue to flow in from clients using the affected Fennec versions*. This will mean the next schema-incompatible sync ping version would be v6.
2. Drop the v5 data now and going forward. This will mean we could use v5 for the next schema-incompatible change on the client.

In the underlying implementation, we'll be dealing with a special case either way. Option 1 implies a special case to consider the unqualified telemetry.sync table as v4 rather than using the default of "the highest available version". Option 2 means we'll have to manually delete previously collected data and drop the corresponding tables. With Option 2, if we want to use the v5 table again in the future, we will be mixing those records with the ones from old versions.

I slightly prefer option 2, but I'm not sure about the implications for data analysis.

* Here are the per-version submission counts of v5 pings since Sept. 1st:

app_version,row_count
55.0.2,113840
57.0.1,109424
56.0,75893
57.0.4,55165
57.0,49707
55.0,6421
56.0a1,302
57.0a1,258
58.0a1,191
57.0.3,8

Comment 5

[Leif Oines [:loines]]

let me take a look at how many users that is and get back to you

Comment 6

[Lina Butler (ex-Mozilla)]

Hmmm, those versions are 2+ years old...I don’t think we’d be able to act on the data (like failure reasons and success rates) even if we kept collecting it. Leif, how do you feel about dropping it?

Comment 7

[Leif Oines [:loines]]

https://sql.telemetry.mozilla.org/queries/65209/source#166003

So it looks like we would be deleting/not logging data for 5-6k fennec users per day if we go with option 2. The number of users on these versions has been going down steadily. As of today, dropping/ignoring data from those clients would lead to drop of 0.5% of our measure of the number of sync clients on fennec per day. That is not whopping, but it is also not trivial.

I believe there will also be another change to the sync ping soon. It may be schema-incompatible see https://bugzilla.mozilla.org/show_bug.cgi?id=1582263.

Does that change the calculus here? Would it tip the balance more toward 2? If so, then I think I'm OK with it.

Comment 8

[Mark Reid [:mreid]]

Adding "devices" at the top level of the payload would be compatible as long as we don't need to prevent the "devices" field from being present inside the "syncs" field too.

If we do need to prevent that case, it would be incompatible and I'd say we should increment both desktop and mobile to v6 and move on (ie. it tips the balance towards option 1). This has the complication that changing the version on desktop away from v4 is tricky.

If not, and we can tolerate looking in two places in one table for the device info, we can stick with v4 and I continue to prefer option 2.

Comment 9

[Arkadiusz Komarzewski [:akomar]]

To sum up:

• v5 is lingering on for ~2 years
• dropping v5 is not trivial from the point of view of KPIs
• we'll likely have another schema change in near future which will probably force us to create a view unioning at least v4 and v6

I suggest, just in context of having proper schemas for the versions currently in use, that we:

• keep v5 and set it up with the same schema as v4
• change telemetry.sync to be a view unioning v4 and v5 so it can be used instead of legacy Sync datasets

Comment 10

[Mark Reid [:mreid]]

That sounds good to me. I also filed Bug 1587844 as one possible way to make it easier to version the sync ping in Fx Desktop.

Comment 11

[Arkadiusz Komarzewski [:akomar]]

Attachment: pr 439: Bug 1572744 - Add schema for v5 of Sync ping

Comment 12

[Arkadiusz Komarzewski [:akomar]]

V5 schema has been set up. telemetry.sync is now pointing to sync_v4 as that should be used by default.
For more sensitive analyses (like the one mentioned in Comment 7), unioning v5 should be considered (see example query).