1274975 - Validate moving the clientId loading earlier during startup

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Alessio Placitelli [:Dexter]]

In bug 1233986 we load/serialize the clientId earlier during startup if a ping is sent before Telemetry is initialized.

Once that bug lands and we have enough data to verify, we should make sure:

- to identify a set a measurements to perform a pre/post landing comparison and see if we've correctly attacked the "ping with no clientId" problem;
- to compare crash-pings and see if there was a drop in the numbers of pings with no client id.

Comment 1

[Alessio Placitelli [:Dexter]]

There seems to be a sharp drop in the crash pings with no client id from Firefox Nightly [1].

The 7 days before bug 1233986 landed (22nd-28th of June), there were 3621 crash pings, 80 of which had no client id.
The 7 days after the fix landed (29th of June - 6th of July), there were 3136 crash pings and no pings without a client id.

[1] - https://gist.github.com/Dexterp37/6cbb85fdff83c6c5903af7962fd13647

Comment 2

[Chris H-C :chutten]

\o/

Comment 3

[Alessio Placitelli [:Dexter]]

The same behaviour described in comment 1 can be observed for main-ping(s) [1]. There were few missing pings before for the reference week (5), and now there are none. Such a low amount of missing cliend ids is expected, as they should be sent after the client id has been loaded.

We are also starting to see the data coming from the TELEMETRY_PING_SUBMISSION_WAITING_CLIENTID probe [2].

Unfortunately, we don't have more than 7 days of data to validate bug 1233986. I've filed bug 1284880 to do repeat the analyses once we have more data/hit the other channels.

Georg, does [1] look sane to you? Can we close this bug?

[1] - https://gist.github.com/Dexterp37/f1e7ebc9d214b14bf4c154d0589c290c
[2] - https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-07-04&keys=__none__!__none__!__none__&max_channel_version=nightly%252F50&measure=TELEMETRY_PING_SUBMISSION_WAITING_CLIENTID&min_channel_version=null&product=Firefox&sanitize=0&sort_keys=submissions&start_date=2016-06-29&table=0&trim=1&use_submission_date=0

Comment 4

[Georg Fritzsche [:gfritzsche]]

This looks good for the missing client ids.

For the "pings per client" numbers, looking at them per day is more stable.
You are looking at different build id ranges for the two weeks and the first week here can have more submissions per client. As it includes build ids from a longer time-frame, it will be less affected by update lag.

Comment 5

[Alessio Placitelli [:Dexter]]

I've updated the notebook from comment 3 to look at pings per client, per day. It also restrict the buildids in the pre-fix week to a saner range. It still looks ok, even better than before, as there are no pings with missing client id in the week before the fix lands for that week's build ids.

Comment 6

[Georg Fritzsche [:gfritzsche]]

Cool, sounds good then.