1470908 - Investigate "main" ping payload size changes on Nightly

Status: RESOLVED FIXED

Description

[Chris H-C :chutten]

According to the "budget" dashboard[1] we saw size decreases to "main" pings on Nightly:

1) 2018-05-24 a drop of 15kb
2) 2018-06-15 a drop of 120kb
3) 2018-06-16 a drop of about 10kb (though variable)

We should investigate these changes to 1) Explain to our satisfaction why these happened and 2) To ensure we aren't losing data.

[1]: https://metrics.services.mozilla.com/telemetry-budget-dashboard/

Comment 1

[Chris H-C :chutten]

3) is probably bug 1468809. If so it should manifest in having decreases in the keyedHistograms portions of the pings.

Comment 2

[Chris H-C :chutten]

Changelogs (built by going to https://hg.mozilla.org/mozilla-central/firefoxreleases and filling in the fromchange/tochange revisions using those listed for the final build of the previous day and the first build of the affected day):

1) https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=47e81ea1ef10&tochange=043e4ab6e724
2) https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=91db0c695f02&tochange=0b5495dc100d
3) https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5b2a9b683f20&tochange=9a847d79eb8e

Comment 3

[Jan-Erik Rediger [:janerik] (Away Jun 24-28)]

2) could be bug 1458337. Removing `MAIN_THREAD_RUNNABLE_MS` and some other histograms.

Comment 4

[Alessio Placitelli [:Dexter] - Out until 01/July]

(In reply to Chris H-C :chutten from comment #2)
> 3)
> https://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=5b2a9b683f20&tochange=9a847d79eb8e

Note that this changelog contains bug 1468809 (dropping expired keyed histograms).

Comment 5

[Chris H-C :chutten]

Attachment: main_ping_size - Databricks.png

I've taken a look at mean payload segment size for Nightly "main" pings submitted since May 22: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/18120

The tl;dr is the plot you see as an attachment. It is illustrative, but is using a non-representative sample, so it's not the -best-.

But it pretty clearly shows that the following segments decreased for the events:

1) content, keyedHistograms, histograms, extension
2) content, keyedHistograms, extension, gpu
3) content, keyedHistograms, extension

This supports the theory that 2) is due to bug 1458337 and 3) is due to bug 1468809

1) is still confusing to me, though. It is across keyed and unkeyed histograms, which suggests a broad spectrum change. If it were from another point in time we could ascribe it to the expiry of measures as versions roll on, however the version didn't change anywhere near May 24 (Nightly rolled over to 62 on the 7th of May). There were no changes to Histograms.json that landed on either of the builds that day, so it would have to be a behavioural change.

However, it is not as clearly defined on this graph as I'd like. There's a little up-and-down, and it may have come from the build just before the 24 and this is a continued decline.

Either way, it is far less interesting than whatever happened June 20th to the Extension process. Wow.

Comment 6

[Chris H-C :chutten]

Attachment: "main" ping subsection sizes

I'm going to call this investigation complete. We have solid theories on the biggest changes, and I've now created a method for easy analysis of a population's ping's payload's segments' sizes over time.

(( If we're only interested in payload sizes in total, there's a dedicated dataset for that: telemetry_payload_size_parquet ))

As a final attachment, see here the final plot, optimized for legibility. I dove into the subprocess sections and used a broader palette, taking this as an excuse to remember how to use matplotlib.