1400921 - Telemetry update parket status=success ping has got really low numbers

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Rémy Hubscher (:natim)]

Looking at the update_parket for status="success" shows really low numbers.

For instance you can see:
- https://sql.telemetry.mozilla.org/queries/38580/source
or
- https://sql.telemetry.mozilla.org/queries/36807/source

We are looking for a similar number of users between ready and success but here it is much lower.

Comment 1

[Alessio Placitelli [:Dexter]]

Wesley, are we seeing any schema error spikes? (Is there any way I can check without bothering you? :D)

Comment 2

[Wesley Dawson [:whd]]

https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.moz_telemetry_doctype_monitor_update.ingestion_error.html has the ingestion error rate - distinct from the parquet error rate since the parquet schema per https://bugzilla.mozilla.org/show_bug.cgi?id=1384861#c11 is currently more restrictive than the json one.

The actual parquet output error rate is at 5% of pings, a number that currently requires bothering me to determine. Hitherto we've assumed that the json and parquet schemas are compatible, but we've filed bug #1400935 for properly monitoring the differences.

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Wesley Dawson [:whd] from comment #2)
> https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.
> moz_telemetry_doctype_monitor_update.ingestion_error.html has the ingestion
> error rate - distinct from the parquet error rate since the parquet schema
> per https://bugzilla.mozilla.org/show_bug.cgi?id=1384861#c11 is currently
> more restrictive than the json one.
> 
> The actual parquet output error rate is at 5% of pings, a number that
> currently requires bothering me to determine. Hitherto we've assumed that
> the json and parquet schemas are compatible, but we've filed bug #1400935
> for properly monitoring the differences.

Deploying the updated parquet schema helped quite a bit with this: https://sql.telemetry.mozilla.org/queries/38580/source

Comment 4

[Alessio Placitelli [:Dexter]]

The numbers look reasonable now, closing this as fixed. Thanks :whd.

Comment 5

[Rémy Hubscher (:natim)]

Yes you can see the nightly uptake here: https://mozilla.github.io/delivery-dashboard/#pollbot/firefox/57.0a1